Performance based guidance platform for robotic medical systems

Abstract

Performance metrics based user guidance platform for robotic medical systems is provided. A system can identify, via a data stream of a robot, an action of the robotic to perform a task of a medical procedure and identify, based on the action, a performance metric for the task. The system can generate, using the performance metric and the task input into a model trained on tasks of medical procedures associated with performance metrics, a plurality of series of tasks. The system can rank the plurality of series of tasks based on a plurality of performance metrics and determine, based on the ranking, a series of tasks to use for actions of the series of tasks.

Description

PERFORMANCE BASED GUIDANCE PLATFORMFOR ROBOTIC MEDICAL SYSTEMSCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 657,573, filed June 7, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] Medical procedures can be performed in an operating room. As the amount and variety of equipment in the operating room increases, or medical procedures become increasingly complex, it can be challenging to perform such medical procedures efficiently, reliably, or without incident.SUMMARY

[0003] The technical solutions introduced herein provide a performance metrics-driven machine learning (ML) based user guidance platform to improve surgical outcomes for robotic system medical procedures. In the course of ongoing medical procedures, robotic medical systems can gather various procedure related data, such as different types of data streams and performance metrics on various stages of the medical procedure, reflecting on opportunities for systematic improvement of the surgical outcome. However, the lack of real-time system-based insights into such opportunities can undermine their timely and intraoperative identification. This can adversely affect the surgical success of the medical procedure as it can be challenging to maximize the likelihood of a desired surgical outcome given the absence of a solution to identify and notify the surgeon of such opportunities, which can arise in a variety of situations.

[0004] For example, during a medical procedure, a robotic medical system can monitor real-time data and performance metrics of different actions and tasks that can be implemented in course of the procedure. Occasionally, the procedure can reach a stage in which a surgeon can choose between multiple series of actions or tasks to implement next, each of which can be associated with different performance metrics. As a result, the series of actions or tasks that the surgeon chooses can impact the likelihood of a desired surgical outcome, making it beneficial to inform the surgeon of the performance metrics associated with each of the series and provide a recommendation. The technical solutions overcome these challenges using real-time roboticdata along with machine learning (ML) models trained on performance metrics of various actions, tasks and procedures, to identify the series of actions or tasks with the performance metrics that improve the likelihood of a positive surgical outcome.

[0005] For example, in an ongoing medical procedure, a robotic medical system can operate using a particular set of system configurations, including configurations of particular medical instruments or tools, which may not correspond to performance metrics that maximize the likelihood of a desired surgical outcome. For instance, robotic medical system can detect that a given set of configurations can be suboptimal for a given procedural action, a task or a medical procedure. As the robotic medical system can include different performance metrics for different system configurations, the surgeon operating the robotic medical system can benefit from a timely notification to reconfigure the system according to the configurations improving the likelihood of a desired surgical outcome. The technical solutions can overcome these challenges by using real-time data and ML models trained to identify the system configurations with maximized performance metrics for the given medical procedure or a task.

[0006] For example, when a patient with a particular medical characteristic desires a particular type of a procedure implemented in a particular way, it can be beneficial to utilize robotic medical system performance metrics data for various surgeon profiles to identify the most suitable surgeon for this particular procedure. For instance, it can be beneficial to match the unique characteristics of the patient’s medical procedure with a unique signature of performance metrics of a particular surgeon matching the desired characteristics. Identifying such a match can match the performance metrics of the surgeon with the preferences of the particular medical procedure, improving the likelihood of a positive surgical outcome. The technical solutions can achieve such objectives by utilizing ML modeling to match the patients with characteristic medical procedures with signature performance metrics of surgeons that most closely match the characteristics of the performance.

[0007] For instance, when in the course of a medical procedure a surgeon desires to research an issue or answer a question informed by the performance metrics data gathered by the robotic medical system, it can be beneficial for the robotic medical system to provide the surgeon with a ML-powered chatbot functionality to address the surgeon’s queries. However, as ML models can be subject to hallucinations and drifting, it can be challenging to provide ML-based chatbots with a sufficient level of consistency and response accuracy suitable for such a purpose. The technical solutions can overcome these challenges by providing orchestrated selections of ML model powered chatbots specifically trained on particularprocedural areas, improving the reliability and reducing the likelihood of ML-based chatbot performance failures.

[0008] At least one aspect of the technical solutions is directed to a system. The system can include one or more processors, coupled with memory. The one or more processors can be configured to identify, using a data stream of a robotic medical system, an action of the robotic medical system to perform a task of a plurality of tasks of a medical procedure performed by the robotic medical system. The one or more processors can be configured to identify, based on the action, a performance metric for the task. The one or more processors can be configured to generate, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed by the robotic medical system in the medical procedure following the task. The one or more processors can be configured to rank the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks. The one or more processors can be configured to determine, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform one or more actions of the series of tasks.

[0009] The one or more processors can be configured to identify a plurality of documents for a plurality of medical procedures performed using a plurality of robotic medical systems. The one or more processors can be configured to determine the ranking for the plurality of series of tasks using the plurality of documents. The one or more processors can be configured to identify an agent for an ML model trained on a plurality of documents corresponding to the plurality of series of tasks. The one or more processors can be configured to provide, responsive to an input of a user of the robotic medical system into the agent, a text corresponding to the series of tasks. The text can be generated by the ML model based on a document of the plurality of documents.

[0010] The one or more processors can be configured to generate the plurality of series using a plurality of documents for the plurality of series of tasks. The one or more processors can be configured to validate the series of tasks using at least an agent for an ML model of the one or more ML models trained on the plurality of documents.

[0011] The one or more processors can be configured to identify a second action of the robotic medical system and determine, based on the second action, a second performancemetric for the task. The one or more processors can be configured to update the ranking of the plurality of series of tasks based on the second performance metric. The one or more processors can be configured to determine, based at least on the updated ranking, a second series of tasks of the plurality of series of tasks to use for a remainder of the medical procedure.

[0012] The one or more processors can be configured to rank the plurality of series of tasks based on a performance metric of the plurality of performance metrics determined based on data on a recovery of a patient following a prior medical procedure of the plurality of medical procedures in which the series of tasks was applied. The one or more processors can be configured to rank the plurality of series of tasks based on a performance metric of the plurality of performance metrics determined based on an occurrence of a medical emergency during performance of a prior medical procedure of the plurality of medical procedures.

[0013] The one or more processors can be configured to identify a plurality of documents including at least one of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on a plurality of patients. The one or more processors can be configured to use the plurality of documents to generate the plurality of series of tasks.

[0014] The one or more processors can be configured to identify, using the one or more ML models trained on a plurality of documents for the plurality of medical procedures, one or more tasks of the medical procedure using a document of the plurality of documents. The one or more processors can be configured to map the series of tasks with the one or more tasks using the performance metric.

[0015] The one or more processors can be configured to generate a plurality of confidence scores for a plurality of mappings between the plurality of series of tasks and the plurality of performance metrics. The one or more processors can be configured to rank the plurality of series of tasks based at least on the plurality of confidence scores. The one or more processors can be configured to determine, using the one or more ML models, a performance metric for the series of tasks. The one or more processors can be configured to generate a report for the series of tasks using the performance metric, the report indicating the ranking of the plurality of series of tasks and one or more documents in support for the ranking.

[0016] An aspect of the technical solution is directed to a method. The method can include one or more processors coupled with memory identifying, using a data stream of a robotic medical system, an action to perform a task of a plurality of tasks of a medical procedure. Themethod can include the one or more processors identifying, based on the action, a performance metric for the task. The method can include the one or more processors generating, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed by the robotic medical system in the medical procedure following the task. The method can include the one or more processors ranking, the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks. The method can include the one or more processors determining, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform a second action of the series of tasks.

[0017] The method can include identifying, by the one or more processors, a plurality of documents for a plurality of medical procedures performed using a plurality of robotic medical systems. The method can include determining, by the one or more processors, the ranking for the plurality of series of tasks using the plurality of documents.

[0018] The method can include identifying, by the one or more processors, an agent for an ML model trained on a plurality of documents corresponding to the plurality of series of tasks. The method can include providing, by the one or more processors, responsive to an input of a user of the robotic medical system into the agent, a text corresponding to the series of tasks. Text can be generated by the ML model based on a document of the plurality of documents.

[0019] The method can include generating, by the one or more processors, the plurality of series using a plurality of documents for the plurality of series of tasks. The method can include validating, by the one or more processors, the series of tasks using at least an agent for an ML model of the one or more ML models trained on the plurality of documents.

[0020] The method can include identifying, by the one or more processors, a second action of the robotic medical system. The method can include determining, by the one or more processors, based on the second action, a second performance metric for the task. The method can include updating, by the one or more processors, the ranking of the plurality of series of tasks based on the second performance metric. The method can include determining, by the one or more processors, based at least on the updated ranking, a second series of tasks of the plurality of series of tasks to use for a remainder of the medical procedure.

[0021] The method can include identifying, by the one or more processors, a plurality of documents including at least one of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on a pluralityof patients. The method can include using, by the one or more processors, the plurality of documents to generate the plurality of series of tasks.

[0022] The method can include identifying, by the one or more processors, using the one or more ML models trained on a plurality of documents for the plurality of medical procedures, one or more tasks of the medical procedure using a document of the plurality of documents. The method can include mapping, by the one or more processors, the series of tasks with the one or more tasks using the performance metric.

[0023] At least one aspect of the technical solutions related to a non-transitory computer- readable medium storing processor executable instructions. The instructions can be such that, when executed by one or more processors, cause the one or more processors to identify, using a data stream of a robotic medical system, an action of the robotic medical system to perform a task of a medical procedure. The instructions can be such that, when executed by one or more processors, cause the one or more processors to identify, based on the action, a performance metric for the task. The instructions can be such that, when executed by one or more processors, cause the one or more processors to generate, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed following the task. The instructions can be such that, when executed by one or more processors, cause the one or more processors to rank the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks. The instructions can be such that, when executed by one or more processors, cause the one or more processors to determine, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform a second action of the series of tasks.

[0024] An aspect of the technical solutions relates to a system. The system can include one or more processors, coupled with memory. The one or more processors can be configured to identify a surgical outcome for a medical procedure performed using a robotic medical system. The one or more processors can be configured to determine, based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure. The one or more processors can be configured to identify one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks. The one or more processors can be configured to select, using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupledto the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome. The one or more processors can be configured to provide the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

[0025] The one or more processors can be configured to determine, based at least on the data stream, a series of actions comprising the action, the series of actions indicative of a series of tasks of the medical procedure comprising the task. The one or more processors can be configured to identify the task based on the series of actions.

[0026] The one or more processors can be configured to provide a recommendation of the configuration for display on a user interface of a client device. The one or more processors can be configured to receive, via the user interface, a selection of the configuration. The one or more processors can be configured to configure, responsive to the selection, the robotic medical system according to the configuration.

[0027] The one or more processors can be configured to identify, based on the surgical outcome, the plurality of configurations for the robotic medical instrument for the surgical outcome. The one or more processors can be configured to determine the performance metrics of the plurality of tasks based on the plurality of configurations. The one or more processors can be configured to select, from the plurality of configurations, the configuration based at least on a performance metric of the configuration exceeding performance metrics of another configuration of the plurality of configurations.

[0028] The one or more processors can be configured to identify, using the one or more ML models, the performance metrics for the plurality of medical procedures completed using the robotic medical system. The one or more processors can be configured to identify the surgical outcome for the medical procedure based at least on the performance metrics. The one or more ML models can include an ML model trained using a plurality of documents on the plurality of tasks. The plurality of documents can include at least one of a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on one or more patients that underwent the medical procedure.

[0029] The one or more processors can be configured to identify, using the one or more ML models, the plurality of configurations for the plurality of medical procedures completed using the robotic medical system. The one or more processors can be configured to select, from the plurality of configurations, the configuration based at least on the surgical outcome and the performance metrics.

[0030] The one or more processors can be configured to detect a phase of the medicalprocedure based at least on an order of a plurality of tasks detected using the plurality of metrics over a time interval of the medical procedure. The one or more processors can be configured to generate a plurality of recommendations for a plurality of tasks remaining in the phase, the plurality of recommendations according to an order of the plurality of tasks remaining in the phase. The one or more processors can be configured to generate a report for an account of associated with a user, the report including one or more citations to one or more documents corresponding to the medical procedure.

[0031] The configuration can include at least one of: a selection of a medical instrument to use for the medical procedure, an arrangement of one or more medical instruments on the robotic medical system, or a setting of a medical instrument. The one or more processors can be configured to generate, using one or more ML models, one or more sections of a report including a recommendation for the one or more remaining tasks. The one or more sections can include an introductory section and a conclusion generated using a document selected for the report based on the recommendation from one or more documents on the medical procedure.

[0032] An aspect of the technical solutions is directed to a method. The method can include the one or more processors coupled with memory identifying a surgical outcome for a medical procedure performed using a robotic medical system. The method can include the one or more processors determining based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure. The method can include the one or more processors identifying one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks. The method can include the one or more processors selecting using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome. The method can include the one or more processors providing, by the one or more processors, the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

[0033] The method can include the one or more processors determining based at least on the data stream, a series of actions comprising the action, the series of actions indicative of a series of tasks of the medical procedure comprising the task. The method can include the one or more processors identifying the task based on the series of actions. The method can include the one or more processors providing a recommendation of the configuration for display on a user interface of a client device. The method can include the one or more processors receiving viathe user interface, a selection of the configuration. The method can include the one or more processors configuring responsive to the selection, the robotic medical system according to the configuration.

[0034] The method can include the one or more processors identifying, based on the surgical outcome, the plurality of configurations for the robotic medical instrument for the surgical outcome. The method can include the one or more processors determining the performance metrics of the plurality of tasks based on the plurality of configurations. The method can include the one or more processors selecting, from the plurality of configurations, the configuration based at least on a performance metric of the configuration exceeding performance metrics of another configuration of the plurality of configurations.

[0035] The method can include the one or more processors identifying, using the one or more ML models, the performance metrics for the plurality of medical procedures completed using the robotic medical system. The method can include the one or more processors identifying the surgical outcome for the medical procedure based at least on the performance metrics. The method can include identifying, by the one or more processors, using the one or more ML models, the plurality of configurations for the plurality of medical procedures completed using the robotic medical system. The method can include selecting, by the one or more processors, from the plurality of configurations, the configuration based at least on the surgical outcome and the performance metrics.

[0036] The method can include detecting, by the one or more processors, a phase of the medical procedure based at least on an order of a plurality of tasks detected using the plurality of metrics over a time interval of the medical procedure. The method can include generating, by the one or more processors, a plurality of recommendations for a plurality of tasks remaining in the phase, the plurality of recommendations according to an order of the plurality of tasks remaining in the phase.

[0037] An aspect of the technical solutions is directed to a non-transitory computer- readable medium storing processor executable instructions. The instructions, when executed by one or more processors, can cause the one or more processors to identify a surgical outcome for a medical procedure performed using a robotic medical system. The instructions, when executed by one or more processors, can cause the one or more processors to determine, based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure. The instructions, when executed by one or more processors, can cause the one or more processors to identify one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve aplurality of surgical outcomes according to performance metrics of the plurality of tasks. The instructions, when executed by one or more processors, can cause the one or more processors to select, using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome. The instructions, when executed by one or more processors, can cause the one or more processors to provide the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

[0038] An aspect of the technical solutions can be directed to a system. The system can include the one or more processors, coupled with memory. The one or more processors can be configured to receive a characteristic of a medical procedure to be performed on a patient using a robotic medical system. The one or more processors can be configured to identify one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems. The one or more processors can be configured to select, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold. The one or more processors can be configured to provide the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

[0039] The one or more processors can be configured to identify a plurality of profiles of surgeons trained to perform the medical procedure. The one or more processors can be configured to select, from the plurality of profiles, the surgeon based at least on availability of the surgeon. The one or more processors can be configured to identify a set of characteristics of the medical procedure comprising the characteristic. The one or more processors can be configured to compare the set of characteristics with a plurality of signatures of performance metrics of a plurality of profiles of surgeons. The one or more processors can be configured to select the identifier of the profile, based on the comparison.

[0040] The threshold can correspond to a maximum performance metric of a plurality of performance metrics of a plurality of profiles of surgeons, the plurality of profiles comprising the profile of the surgeon. The one or more processors can be configured to map the plurality of medical procedures performed on a plurality of patients with a plurality of signatures of performance metrics corresponding to the plurality of medical procedures. The one or moreprocessors can be configured to identify the signature performance metric based at least on the mapping.

[0041] The plurality of signatures of performance metrics can include data corresponding to the plurality of surgeons. The data can include at least one of a number of surgeries performed by the surgeon, one or more types of surgeries performed by the surgeon, training completed by the surgeon, performance metrics for surgeries performed by the surgeon, performance metrics for a type of a task included in the medical procedure.

[0042] The characteristic of the medical procedure can include at least one of a type of a medical procedure, a medical condition of the patient, a preference for series of tasks of a plurality of series of tasks to be performed during the medical procedure, or a health risk associated with a performance metric of the plurality of signatures of performance metrics. The one or more processors can be configured to identify a performance metric corresponding to the characteristic of the medical procedure. The one or more processors can be configured to identify the identifier of the profile based at least on a match between the performance metric and one or more performance metrics of the signature of performance metrics of the profile of the surgeon.

[0043] The one or more processors can be configured to identify one or more tasks associated with the characteristic of the medical procedure. The one or more processors can be configured to identify the identifier of the surgeon based at least on a performance metric for a task of the profile of the surgeon corresponding to the one or more tasks associated with the characteristic. The one or more processors can be configured to identify the identifier of the surgeon based at least on a performance metric of the profile of the surgeon associated with the characteristic of the medical procedure corresponding to a type of the medical procedure.

[0044] The one or more processors can be configured to select a plurality of identifiers of a plurality of profiles of surgeons, each profile of the plurality of profiles having a respective signature of performance metrics. The one or more processors can be configured to determine that the profile of the surgeon includes the signature of performance metrics corresponding to the characteristic of the medical procedure. The one or more processors can be configured to select the profile responsive to the determination.

[0045] The one or more processors can be configured to identify a plurality of characteristics of the medical procedure comprising the characteristic. The one or more processors can be configured to determine that the signature of performance metrics of the profile of the surgeon corresponds to the plurality of characteristics of the medical procedure. The one or more processors can be configured to identify the identifier of the profile based onthe determination.

[0046] The one or more processors can be configured to identify a plurality of characteristics of the medical procedure, the plurality of characteristics corresponding to a plurality of performance metrics for a plurality of tasks the medical procedure. The one or more processors can be configured to rank the plurality of characteristics of the medical procedure according to the plurality of performance metrics. The one or more processors can be configured to identify the identifier of the profile based at least on the ranked characteristic of the plurality of characteristics. The one or more processors can be configured to identify a desired surgical outcome based on the characteristic; and select the identifier of the profile based at least on the desired the desired surgical outcome.

[0047] An aspect of the technical solutions can be directed to a method. The method can include the one or more processors coupled with memory. The method can include the one or more processors receiving a characteristic of a medical procedure to be performed on a patient using a robotic medical system. The method can include the one or more processors identifying one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems. The method can include the one or more processors selecting, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold. The method can include the one or more processors providing the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

[0048] The method can include identifying, by the one or more processors, a plurality of profiles of surgeons trained to perform the medical procedure. The method can include selecting, by the one or more processors, from the plurality of profiles, the surgeon based at least on availability of the surgeon. The threshold can correspond to a maximum value of a performance metric of a plurality of values of the plurality of performance metrics of a plurality of profiles of surgeons. The plurality of profiles can include the profile of the surgeon.

[0049] The method can include identifying, by the one or more processors, a set of characteristics of the medical procedure comprising the characteristic. The method can include comparing, by the one or more processors, the set of characteristics with a plurality of signatures of performance metrics of a plurality of profiles of surgeons. The method can include selecting, by the one or more processors, the identifier of the profile, based on thecomparison.

[0050] The method can include mapping, by the one or more processors, the plurality of medical procedures performed on a plurality of patients with a plurality of signatures of performance metrics corresponding to the plurality of medical procedures. The method can include identifying, by the one or more processors, the signature performance metric based at least on the mapping. The method can include identifying, by the one or more processors, a performance metric corresponding to the characteristic of the medical procedure. The method can include identifying, by the one or more processors, the identifier of the profile based at least on a match between the performance metric and one or more performance metrics of the signature of performance metrics of the profile of the surgeon.

[0051] An aspect of the technical solutions can be directed to a non-transitory computer- readable medium storing processor executable instructions. The instructions, when executed by one or more processors, can cause the one or more processors to receive a characteristic of a medical procedure to be performed on a patient using a robotic medical system. The instructions, when executed by one or more processors, can cause the one or more processors to identify one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems. The instructions, when executed by one or more processors, can cause the one or more processors to select, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold. The instructions, when executed by one or more processors, can cause the one or more processors to provide the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

[0052] An aspect of the technical solutions can be directed to a system. The system can include one or more processors, coupled with memory. The one or more processors can be configured to receive, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system. The one or more processors can be configured to determine, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system. The one or more processors can be configured to identify a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality ofperformance metrics for a plurality of configurations of medical instrument and a plurality of tasks of a plurality of medical procedures. The one or more processors can be configured to generate, using the state information and the chatbot, a response to the query. The one or more processors can be configured to provide, via the user interface, the response to the query.

[0053] The one or more processors can be configured to receive a plurality of data streams indicative of one or more tasks of the medical procedures completed using the robotic medical system. The one or more processors can be configured to determine, based on one or more portions of the plurality of data streams input into the one or more ML models, the state information.

[0054] The state information can include at least one of: a task of a plurality of tasks of the medical procedure currently performed, one or more tasks of a plurality of tasks of the medical procedure completed prior to a task currently performed, one or more tasks of a plurality of tasks of the medical procedure to be performed following a task currently performed, a phase of the medical procedure performed by the robotic medical system, one or more phases of the medical procedure completed, one or more phases of the medical procedure to be performed following a phase currently performed, information on one or more medical instruments used during the medical procedure, a configuration for a medical instrument, a medical history of a patient, or a profile of a surgeon performing the medical procedure.

[0055] The portion of the medical procedure can include at least one of: a phase of a plurality of phases of the medical procedure, a task of a plurality of tasks of the phase of the medical procedure or an action of a plurality of actions of a task of the medical procedure.

[0056] The one or more processors can be configured to identify that the query includes a text requesting data on a configuration of a medical instrument of the robotic medical system. The one or more processors can be configured to generate, responsive to a portion of the text input into the chatbot, the response including the data on the configuration of the medical instrument.

[0057] The one or more processors can be configured to identify that the query includes a text requesting information on a task of the medical procedure performed using the robotic medical system. The one or more processors can be configured to generate, responsive to a portion of the text input into the chatbot, the response including the information on the task. The one or more processors can be configured to parse the query into a plurality of portions. The one or more processors can be configured to identify, from the plurality of portions of the query, a portion corresponding to a task to be performed using the robotic medical system.

[0058] The one or more processors can be configured to input the query into the chatbot,the query requesting information on one or more actions of a task of a plurality of tasks of the medical procedure to be performed using the robotic medical system. The one or more processors can be configured to generate, based at least on a performance metric of the one or more actions, the response comprising a recommendation corresponding to the one or more actions to be performed.

[0059] The one or more processors can be configured to identify a performance metric of a completed task of a prior medical procedure completed by a user associated with an account corresponding to the query. The one or more processors can be configured to generate, based at least on the performance metric of the completed task, the response comprising a recommendation on a task to be performed during a remaining portion of the medical procedure.

[0060] The one or more processors can be configured to receive the query responsive to a codeword detected by a device configured to record sound. The one or more processors can be configured to identify a portion of the query corresponding to a task of the medical procedure. The one or more processors can be configured to generate the response based at least on the portion of the query input into the one or more ML models.

[0061] The one or more processors can be configured to determine the state information comprising a first one or more tasks of the medical procedure completed using the robotic medical system. The one or more processors can be configured to generate the response comprising a second one or more tasks of the medical procedure based at least on the first one or more tasks.

[0062] The one or more processors can be configured to identify, based on the data stream input into the one or more ML models, one or more tasks of a phase of the medical procedure performed by the robotic medical system. The one or more processors can be configured to determine the state information based at least one the phase of the medical procedure. The one or more processors can be configured to generate the response to the query using the one or more tasks of the phase.

[0063] The one or more processors can be configured to identify that the query corresponds to a task of the medical procedure. The one or more processors can be configured to identify, using a portion of the query input into the one or more ML models, a document corresponding to the task. The one or more processors can be configured to provide, for display via the user interface, the response comprising a content of the document. The one or more processors can be configured to identify that the query corresponds to an action for a task of the medical procedure. The one or more processors can be configured to generate, usinga portion of the query input into the one or more ML models, the response.

[0064] An aspect of the technical solutions can be directed to a method. The method can include the one or more processors coupled with memory. The method can include the one or more processors receiving, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system. The method can include the one or more processors determining, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system. The method can include the one or more processors identifying a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality of performance metrics for a plurality of configurations of medical instruments and a plurality of tasks of a plurality of medical procedures. The method can include the one or more processors generating, using the state information and the chatbot, a response to the query. The method can include the one or more processors providing, via the user interface, the response to the query.

[0065] The method can include receiving, by the one or more processors, a plurality of data streams indicative of one or more tasks of the medical procedures completed using the robotic medical system. The method can include determining, by the one or more processors, based on one or more portions of the plurality of data streams input into the one or more ML models, the state information. The method can include identifying, by the one or more processors, that the query includes a text requesting data on a configuration of a medical instrument of the robotic medical system. The method can include generating, by the one or more processors, responsive to a portion of the text input into the chatbot, the response including the data on the configuration of the medical instrument.

[0066] The method can include identifying, by the one or more processors, that the query includes a text requesting information on a task of the medical procedure performed using the robotic medical system. The method can include generating, by the one or more processors, responsive to a portion of the text input into the chatbot, the response including the information on the task. The method can include inputting, by the one or more processors, the query into the chatbot, the query requesting information on one or more actions of a task of a plurality of tasks of the medical procedure to be performed using the robotic medical system. The method can include generating, by the one or more processors, based at least on a performance metric of the one or more actions, the response comprising a recommendation corresponding to the one or more actions to be performed.

[0067] An aspect of the technical solutions can be directed to a non-transitory computer- readable medium storing processor executable instructions. The instructions, when executed by one or more processors, can cause the one or more processors to receive, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system. The instructions, when executed by one or more processors, can cause the one or more processors to determine, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system. The instructions, when executed by one or more processors, can cause the one or more processors to identify a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality of performance metrics for a plurality of configurations of medical instruments and a plurality of tasks of a plurality of medical procedures. The instructions, when executed by one or more processors, can cause the one or more processors to generate, using the state information and the chatbot, a response to the query. The instructions, when executed by one or more processors, can cause the one or more processors to provide, via the user interface, the response to the query.BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings:

[0069] FIG. 1 depicts an example block diagram of a system for providing a performance metrics and machine learning based user guidance platform for robotic medical systems.

[0070] FIG. 2 illustrates an example of a surgical system, in accordance with some aspects of the technical solutions.

[0071] FIG. 3 illustrates an example block diagram of an example computer system is shown, in accordance with some aspects of the technical solutions.

[0072] FIG 4. illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems identifying task series to utilize for remainder of an ongoing medical procedure.

[0073] FIG. 5 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems to identify a most suitable configuration for a remaining portion of the ongoing medical procedure.

[0074] FIG. 6 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems to identify a most suitable surgeon for particular medical procedure.

[0075] FIG. 7. illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems providing response to user queries using a chatbot.

[0076] FIGS. 8-23 illustrate examples of graphical user interface of the guidance platform for robotic medical systems.DETAILED DESCRIPTION

[0077] Following below are more detailed descriptions of various concepts related to, and implementations of, systems, methods, apparatuses for performance metrics based user guidance platform for robotic medical systems. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways.

[0078] Although the present disclosure is discussed in the context of a surgical procedure, in various aspects, the technical solutions of this disclosure can be applicable to other medical or non-medical applications, treatments, sessions, environments or activities, in which performance metrics based user guidance for robotic systems can be sought. For instance, technical solutions can be applied in any environment, application or industry in which activities, operations, processes or acts by robots or robotic tools involve performance metrics that can be used to provide a platform for user guidance while utilizing robotic systems.

[0079] The technical solutions provide a performance metrics based user guidance platform in a robotic medical system. When a user (e.g., a surgeon) of a robotic medical systems performs a medical operation, various real-time robotic medical system data can be gathered and processed by the system. This data can be indicative of various actions, tasks and phases of the operation taking place and which can be associated with various performance metrics (e.g., objective performance indicators or OPIs). These performance metrics (e.g., OPIs) associated with the given medical procedure actions, tasks or procedures can affect a desired surgical outcome. A surgeon performing medical operations using the robotic medical system can findit advantageous to utilize various performance metrics to better inform the surgeon of the actions to be taken.

[0080] For instance, actions taken using the medical instrumentation of the robotic system can produce real-time data on movements and gestures indicative of medical procedure tasks and phases and their associated performance metrics. This data can be used to provide a deeper insight into clinical and operative factors influencing the surgical outcome, which the users can utilize to share findings and use best practices across various operational situations. The technical solutions can utilize performance metrics gathered from various data streams of the robotic medical system and provide machine learning (ML) based user guidance. The technical solutions can include ML models trained to utilize performance metrics of different operational activities (e.g., medical procedure actions, tasks or phases) to generate clinical and operative predictors for determining preferred courses of action for a variety of situations during the course of a medical procedure.

[0081] For instance, during a medical procedure, a robotic medical system can continuously monitor real-time data and performance metrics across various actions and tasks. At certain stages, the surgeon can face multiple options for upcoming series or actions or tasks, each of which can be associated with different performance metrics, potentially affecting the surgical outcome. To help inform the surgeon about the metrics and provide recommendations to improve surgeon’s decision making process, the solutions can utilize real-time robotic data and machine learning (ML) models trained on performance metrics to identify the actions or tasks series that improve the likelihood of a positive surgical outcome. For instance, the solutions can utilize the real-time data and ML models to detect suboptimal configurations of the robotic medical system during an ongoing procedure, identify the configurations with improved performance metrics and notify the surgeon to adjust accordingly for better surgical outcome. For example, the solutions can use the ML models trained on surgeon profile performance metrics to match patients with specific medical characteristics to surgeons whose signature of performance metrics align with such characteristics, improving the likelihood of success. For example, the technical solutions can provide ML-powered chatbot orchestration functionality to address surgeon queries based on performance metrics and other medical data to improve ML performance, mitigating consistency and accuracy issues associated with ML- based chatbots.

[0082] FIG. 1 depicts an example system 100 for providing a performance metrics based and ML-powered user guidance platform for robotic medical systems. System 100 can includea medical environment 102 comprising one or more sensors 104, objects 106, data capture devices 110, medical instruments 112, visualization tools 114, displays 116 and robotic medical systems 120. Robotic medical system (RMS) 120 can include one or more configurations 118 for setting, arranging or configuring system operation, such as movements and positioning of the features of the medical environment 102 (e.g., medical instruments 112). RMS 120 can communicate via one or more data processing systems (DPSs) 130 and one or more head mounted devices (HMDs) 122 via one or more networks 101.

[0083] Data processing system (DPS) 130 can include one or more machine learning (ML) frameworks 132, surgeon profiles 144, procedure data 150, data repositories 168, performance functions 180, state functions 186, ranking functions 166, user interfaces 190 and configuration functions 196. DPS 130 can include one or more action and task identifiers 133, series selectors 135, outcome identifiers 137, profile selectors 139 and chatbot selectors 141. ML framework 132 can include one or more ML trainers 134 trained using training data 178, as well as one or more ML models 136 and agents 138 (e.g., chatbots) that can process queries 140 and provide responses 142. Surgeon profile 144 can include metrics signature 146 and surgeon data 148. Procedure data 150 can include data or information on one or more phases 152, tasks 154, actions 156, target outcomes 198, action series 148, task series 160 and procedure characteristics 162 with patient data 164. Data repository 168 can include one or more procedure data 150, training data 178 and data streams 170, which can include kinematics data 172, sensor data 174 and events data 176. Performance function 180 can include performance metrics 182 and thresholds 184. State functions 186 can include state information 188. User interfaces 190 can include or provide recommendations 192 and reports 194. Configuration function 196 can include or provide configurations 196. Head mounted device (HMD) 122 can include one or more displays 116, eye trackers 124, hand trackers 126, voice controllers 128 and sensors 104.

[0084] System 100 can include a surgical robotic system for performing tasks using medical instruments, such as a robotic medical system 120 used by a surgeon to perform a surgery on a patient. Robotic medical system 120, also referred to as an RMS 120, can be deployed in a medical environment 102. Medical environment 102 can include any space or facility for performing medical procedures, such as a surgical facility, or an operating room. Medical environment 102 can include medical instruments 112 that the RMS 120 can use for performing surgical patient procedures, whether invasive, non-invasive, in-patient, or outpatient procedures. Medical environment 102 can include any objects 106 for a medicalprocedure. Objects 106 can include systems or tools, such as medical operating tables, various medical instruments separate from those used by the RMS 120, surgical lights, medical equipment carts, imaging equipment or other systems or tools. Objects 106 can include or carry fluids or patient monitoring equipment.

[0085] The medical environment 102 can include one or more data capture devices 110 (e.g., optical devices, such as cameras or sensors, depth sensors, or other types of sensing devices or detectors) for capturing data streams 170, that can include sensor data 174 (e.g., data from sound sensors, video cameras or other sensors), events data 176 (e.g., events or occurrences involving an RMS 120) and kinematics data 172 (e.g., movements of medical instruments 112). The medical environment 102 can include one or more visualization tools 114 to gather the captured data streams 170 and process it for display to the user (e.g., a surgeon or other medical professional) at one or more displays 116. A display 116 can present data stream 170 (e.g., video frames, kinematics or sensor data) of an ongoing medical procedure (e.g., an ongoing surgery) performed using the robotic medical system 120 handling, manipulating, holding or otherwise utilizing medical instruments or tools 112 to perform surgical tasks at the surgical site.

[0086] Machine learning (ML) framework 132 can include any combination of hardware and software for providing machine learning functionalities of the DPS 130. ML framework 132 can include and utilize ML trainer 134 to use training data 178 to train one or more ML models 136. ML framework 132 can include and utilize agents 138 (e.g., chatbots) to receive user queries 140 and provide responses 142 to the queries using ML models 136. ML framework 132 can include various ML architecture or functions, such as attention mechanisms, large language models (LLMs), neural networks, transformers with encoder and decoder architecture, or any other type and form of ML architecture or functionality. ML framework 132 can be configured to facilitate effective determinations by the ML models 136 using, for example performance metrics 182 (e.g., OPIs) for various portions of medical procedures, including actions 156, tasks 154 and phases 152, or metrics signatures 146. ML framework 132 can utilize ML models 136 to make determinations, identify most suitable action series 158 to complete tasks 154, most suitable task series 160 to complete a phase 152, identify most suitable surgeon profiles 144, provide recommendations 192 or generate reports 194. ML framework 132 can include attention mechanisms which can utilize weights to improve the capacity of the ML models to discern, detect or recognize specific details within a context, improving the accuracy of determination, detection and prediction.

[0087] ML models 136 can be trained, configured or set up to implement or process any actions, recognitions, identifications, predictions, determinations or processing for, or on behalf of any functions of the DPS 130, such as: action identifier 133, series selector 135, outcome identifier 137, profile selector 139, chatbot selector 141, performance function 180, state function 186, ranking function 166, chatbots or agents 138, configuration function 196 or user interface 190. ML models 136 can be configured to sample and process incoming data streams 170 to identify opportunities to provide recommendations 192 to the surgeon in real-time. For example, ML models 136 can be temporally coordinated to operate multiple functionalities (e.g., monitoring of data streams 170, evaluating performance metrics 182 for various action series 158 and task series 160 on behalf of series selector 135, identify and evaluate performance metrics 182 for various actions 156, tasks 54 or phases 152 on behalf of action identifier 133, analyze particular target outcomes 198 on behalf of outcome identifiers 137, identify profiles 144 on behalf of profile selectors 139 and process chatbot orchestration or selection on behalf of a chatbot selector 141. These actions can be performed in real-time, simultaneously or in temporal coordination among various ML models 136, allowing for the DPS 130 to provide these determinations in real-time.

[0088] ML models 136 can be configured to generate synthetic data, such as artificial or simulated data, to mimic the real -world data, such as any procedure data 150 or any data in repository 168, including any data stream 170 data. Synthetic data can include data generated based on any raw data (e.g., data streams 170, procedure data 150, performance metrics 182 and surgeon profiles 144). ML models 136 can generate the synthetic data by replicating patterns, distributions or relationships found in the original dataset (e.g., 150, 182, 144 or 170). The synthetic data can supplement or improve existing datasets, particularly in scenarios in which available data is limited or lacks diversity.

[0089] Data repository 168 of the DPS 130 can include one or more data streams 170, of various types and from various sources. Data streams 170 can include measurements from sensors 104, which can be referred to as sensor data 174. Sensor data 174 can include data from video cameras (e.g., images or video frames), or various force, torque or biometric data, haptic feedback data, pressure or temperature data, vibration, tension or compression data, endoscopic images or data, ultrasound images or videos or communication and command data streams. Data repository 168 can include events data 176, such as installation, uninstallation, configuration, reconfiguration, setting or resetting data or information related to system files or logs or particular medical instruments 112. ML models 136 or their functionalities (e.g., MLframework 132 components) can each be partially or fully stored in a data repository 168, along with training data sets (e.g., 178) and data streams 170.

[0090] The system 100 can include one or more data capture devices 110 (e.g., video cameras, sensors or detectors) for collecting any data stream 170. Data streams 170 from data capture devices 110 can be used for machine learning based detection of objects, such as detection of medical instruments, processing of performance metrics 182 or determining action series 158, task series 160, target outcomes 198, or making any other determinations. This data can also be used for ML based determination, identification or detection of actions 156, tasks 154 or phases 152 of medical procedures. Data capture devices 110 can include cameras or other image capture devices for capturing video data (e.g., type of sensor data 174) from a particular viewpoint within the medical environment 102. The data capture devices 110 can be positioned, mounted, or otherwise located to capture content from any viewpoint that facilitates the data processing system 130 capturing various surgical tasks or actions.

[0091] Data capture devices 110 can include any of a variety of sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of imaging devices that can be suitable for the purposes described herein. Data capture devices 110 can include cameras that a surgeon can use to perform a surgery and observe manipulation components within a purview of field of view suitable for the given task performance.

[0092] Data capture devices 110 can capture, detect, or acquire sensor data, such as videos or images, including for example, still images, video images, vector images, bitmap images, other types of images, or combinations thereof. The data capture devices 110 can capture the images at any suitable predetermined capture rate or frequency. Settings, such as zoom settings or resolution, of each of the data capture devices 110 can vary as desired to capture suitable images from any viewpoint. For instance, data capture devices 110 can have fixed viewpoints, locations, positions, or orientations. The data capture devices 110 can be portable, or otherwise configured to change orientation or telescope in various directions. The data capture devices 110 can be part of a multi-sensor architecture including multiple sensors, with each sensorbeing configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, or pressure).

[0093] Data capture devices 110 can include any type and form of a sensor 104 that can be configured to measure and provide sensor data 174, including a positioning sensor, a biometric sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature sensor, a haptic or tactile sensor or any other type and form of sensor used for providing data on medical tools 112, or data capture devices (e.g., optical devices). Sensor 104 can include a depth sensor configured to determine a distance between the sensor and an object (e.g., distance to a medical instrument 112 or a patient’s anatomy). For example, a data capture device 110 can include a location sensor, a distance sensor or a positioning sensor providing coordinate locations of a medical tool 112 or a data capture device 110. Data capture device 110 can include a sensor providing information or data on a location, position or spatial orientation of an object (e.g., medical tool 112 or a lens of data capture device 110) with respect to a reference point. The reference point can include any fixed, defined location used as the starting point for measuring distances and positions in a specific direction, serving as the origin from which all other points or locations can be determined.

[0094] Display 116 can show, illustrate or play data streams 170, including video data, in which medical tools 112 at or near surgical sites are shown. For example, display 116 can display a rectangular image (e.g., a frame of a video data) of a surgical site along with at least a portion of medical instruments 112 being used to perform surgical tasks. Display 116 can provide compiled or composite images generated by the visualization tool 114 from a plurality of data capture devices 110 to provide visual feedback from one or more points of view.

[0095] Visualization tool 114 that can be configured or designed to receive any number of different data streams 170 from any number of data capture devices 110 and combine them into a single data stream displayed on a display 116. The visualization tool 114 can be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream 170. For instance, the visualization tool 114 can receive a visual sensor data from one or more medical tools 112, sensors or cameras with respect to a surgical site or an area in which a surgery is performed. The visualization tool 114 can incorporate, combine or utilize multiple types of data (e.g., positioning data of a medical tool 112 along sensor readings of pressure, temperature, vibration or any other data) to generate an output to present on a display 116. Visualization tool 114 can combine or correlate various datastreams 170 based on their respective time of generation, using for example, metadata indicative of time of each portion of data stream 170 (e.g., timestamps in the metadata) to match the data across the data streams 170 to use for determinations.

[0096] Medical instruments or tools 112 can be any type and form of tool or instrument used for surgery, medical procedures or a tool in an operating room or environment. Medical tool 112 can be imaged by, associated with or include an image capture device and can be handled using robotic manipulator arms 235 of the RMS 120. For instance, a medical tool 112 can be a tool for making incisions, a tool for suturing a wound, an endoscope for visualizing organs or tissues, an imaging device, a needle and a thread for stitching a wound, a surgical scalpel, forceps, scissors, retractors, graspers, or any other tool or instrument to be used during a surgery. Medical tools 112 can include hemostats, trocars, surgical drills, suction devices or any instruments for use during a surgery. The medical tool 112 can include other or additional types of therapeutic or diagnostic medical imaging implements. The medical tool 112 can be configured to be installed in, coupled with, or manipulated by an RMS 120, such as by manipulator arms 235 or other components for holding, using and manipulating the medical instruments 112 during procedure.

[0097] RMS 120 can be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient via or using or with the assistance of one or more robotic components or medical tools 112. RMS 120 can include configurations 118 that can provide or include various settings, configurations, adjustments, operating parameters or constraints for controlling movements, motion or actions performed using the RMS 120. RMS 120 can include any number of manipulator arms for grasping, holding or manipulating various medical tools 112 and performing computer-assisted medical tasks using medical tools 112 controlled by the manipulator arms.

[0098] Video data, including any images or videos captured by a medical tool 112 (e.g., endoscopic camera) can be sent to the visualization tool 114. The robotic medical system 120 can include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, the visualization tool 114 can be connected to the RMS 120 to receive the images from the medical instrument 112 when the medical instrument 112 is installed in the RMS 120 (e.g., on a manipulator arm of the RMS 120 that is used for moving, managing or otherwise handing medical instruments 112). The visualization tool 114 can combine the data streams 170 from the data capture devices 110 and the medical tool 112 into a single combined data stream 170 for use by the ML framework 132.

[0099] The system 100 can include a data processing system 130. The data processing system 130 can be deployed in or associated with the medical environment 102, or it can be provided by a remote server or be cloud-based. The data processing system 130 can include an interface 180 designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the robotic medical system 120. Data processing system 130 can be implemented using instructions stored in memory locations and processed by one or more processors, controllers or integrated circuitry. Data processing system 130 can include functionalities, computer codes or programs for executing or implementing any functionality of ML framework 132, including any ML models 136 along with any associated functions or features.

[0100] The ML trainer 134 can any combination of hardware and software for training ML models 136. Machine learning (ML) trainer 134 can include or generate ML models 136, each of which can be trained using training datasets that can include various data streams 170 corresponding to medical procedures using the RMS 120. ML trainer 134 can include a framework or functionality for training different types of ML models 136, such as LLMs, neural network models, spatial -temporal attention mechanism models or any other types of ML models 136. ML trainer 134 can include the functionality to utilize training data 178 for training ML models 136. ML trainer 134 can include the functionality for supervised or unsupervised learning or providing reinforcement learning algorithms for various types of ML models. ML trainer 134 can include the functionality for generating natural language processing, time series forecasting and recommendation ML systems.

[0101] Training data 178 can include any information or data used by the ML trainer 134 to train an ML model 136. Training data 178 can include documentation, RMS 120 records or logs, data streams 170, procedure data 150, hospital records, performance metrics 182 for various phases 152, tasks 154 or actions 154, metrics signatures 146 of various surgeons, procedure characteristics 164 of various patients, patient data 164, surgeon data 148, any surgical procedure related recommendations 192 or reports 194, or any other information corresponding to medical procedures using RMS 120. Training data 178 can include one or more collections of medical documents, medical journal publications, research papers, surgical procedures, medical data from various medical procedures, each of which can be organized in ontologies, including tables that can interrelate various types of data for ML framework purposes.

[0102] Action identifier 133 can include any combination of hardware and software for detecting, identifying, selecting or recognizing actions 156, tasks 154 or phases 152 of a medical procedure. Action identifier 133 can include the functionality for detecting or recognizing individual actions 156 based on monitoring or analysis of data from any data streams 170 (e.g., 172, 174 or 176). Action identifier 133 can utilize one or more ML models 136 for identification of any one or more of actions 156, tasks 154 or phases 152. Action identifier 133 can be configured to identify actions 156 of the RMS 120 to perform, or indicative of, one or more tasks 154 performed by the RMS 120. Action identifier 133 can identify any actions 156, tasks 154 or phases 152 by sampling incoming data from data streams 170 to input the data samples into one or more ML models 136 for detecting any one or more of actions 156, tasks 154 or phases 152 based on, for example kinematics data 172 on movement of medical instruments 112, video stream data (e.g., 174) showing movements of medical instruments 112 and gestures of the surgeon, or using any other data.

[0103] Agent 138, also referred to as a chatbot, can include any combination of hardware and software for providing responses 142 in response to received user queries 140. Agent 138 can include the functionality or an interface to provide a user (e.g., a surgeon) with an access to ML functionality. Agent 138 can include ML-powered interface facilitating interaction between users and DPS 130 using LLM and NLP based ML models 136. For instance, agent 138 can receive queries 140 via a user interface 190, including textual description of user questions for the ML framework 132. Agent 138 can include a parser function to parse and preprocess the textual input. Agent 138 can process the text of the query 140 using one or more selected ML models 136 suitable for a given query 140. ML models 136 can process the query 140 within its context and provide response 142. For instance, agent 138 can utilize ML models 136 to extract from the query 140, a portion of the query 140 that can be input into one or more ML models 136 to generate a response 142 for the query 140. Agent 138 can function as an intermediary, delivering these responses 142 back to the user, via a user interface 190 and allowing the user to enter new queries 140 for additional responses.

[0104] Query 140 can include any request for information or action to retrieve specific data or perform a particular task within a system or database. Query 140 can include of one or more keywords or phrases that can project user's intent or question. Queries 140 can be used by ML models 136 to narrow down and retrieve relevant information from a large pool of data. Query 140 can include requests of varying complexity, triggering any number of ML models, such asML models 136 for refining a query 140 and determining its meaning, ML models 136 for determining, detecting or evaluating actions 156, tasks 154 or phases 154.

[0105] Responses 142 can include any output (e.g., an answer) to a query 140. Responses 142 can be generated by ML models 136 responsive to queries 140 input into one or more ML models 136. Agent 138 can provide responses 142 to a user interface 190 for display to the user. Depending on configuration, responses 142 can be output in various forms, including text, audio, or visual feedback, depending on the nature of the interaction. Responses 142 can include, for example, one sentence answers, paragraphs of description, portions of documents, reports 194 and recommendations.

[0106] Chatbot selector 141, also referred to as an agent selector 141, can include any combination of hardware and software for detecting, identifying, selecting or orchestrating chatbots 138 (e.g., agents 138) to use or apply for given queries 140. Chatbot selector 141 can include the functionality for detecting or recognizing individual agents 138 or chatbots based ML-based analysis of queries 140. For instance, chatbot selector 141 can select an individual chatbot 138, from a plurality of chatbots 138, each one of which is configured for a ML model 136 trained to provide responses 142 to queries 140 from particular areas or regions of focus. The areas or regions of focus on which each of the ML models 136 for each of the chatbots 138 can be trained can include particular types of procedures, particular anatomic group, particular medical or surgical field or any medical field or region. Chatbot selector 141 can include the functionality to process queries 140 and identify which of the particular areas or regions of focus the query 140 relates to and can orchestrate or select a particular chatbot 138 (e.g., a particular ML model 136) for the given area pertaining to the incoming query 140.

[0107] Chatbot selector 141 can include the functionality to implement conversational analysis on the incoming queries 140. The conversational analysis can be utilized to more accurately understand the incoming queries 140 and more accurately orchestrate or select the chatbot 138 for the particular query 140. In doing so, the chatbot selector 141 can reduce or minimize ML-based drifting or hallucinations as more focused ML models 136 can reduce the chance of these ML performance issues.

[0108] The chatbot selector 141 can utilize one or more ML models 136 to determine confidence scores for various responses 142. The confidence scores can be determined by the ML models 136 on behalf of the chatbot selector 141 or performance function 180 to provide a value corresponding to the level of confidence or likelihood of the response 142 being correct.The chatbot selector 141 can identify a particular chatbot 138 in response to a confidence score exceeding a threshold for a determination that a level of certainty that an incoming query 140 corresponds to a particular ML model 136 trained in a particular field. In response to the score for matching such a query 140 to the particular ML model 136, the chatbot 141 can select or assign the chatbot 138 to the query 140. The chatbot selector 141 can determine confidence scores to evaluate whether a response 142 for a given query 140 exceeds a confidence threshold. In response to both the confidence score that an incoming query 140 corresponds to a particular ML model 136 trained in a given field out of a plurality of ML models 136 trained across various fields and the confidence score that a response 142 to that query is correct, the chatbot selector 141 can provide the response 142 to the user interface 190 (e.g., along with the confidence scores). By providing the responses 142 only if the responses 142 (e.g., and their query assignment to correct chatbots 138) exceed the threshold, the chatbot selector 141 can take measures to reduce hallucinations, drift or other ML-based performance issues.

[0109] The chatbot selector 141 can include one or more ML models 136 trained to analyze the conversations (e.g., queries 140) and determine which chatbot 138 to select (e.g., from a plurality of chatbots 138 with ML models 136 trained on a plurality of fields within the robotic surgery). Chatbot selector 141 can include the functionality to parse incoming queries 140, input portions of the queries into ML models 136 for determining the context, intent, the message or the meaning of the query 140, allowing the DPS 130 to more accurately identify the chatbot 138 as well as more accurately utilize the ML models 136 for generating the responses 142. Chatbot selector 141 can provide incoming queries 140 to multiple chatbots 138 to receive multiple responses and compare the responses 142 across the chatbots 138 to determine a confidence score. The confidence scores for the queries 140 or responses 142 can be provided to users via the user interface to indicate to the user the level of confidence or probability that the response 142 is correct.

[0110] Procedure data 150 can include any information or data on medical procedures, including any actions 156, tasks 154 or phases 152. Procedure data 150 can include information on actions, such as actions 156 taken in the course of a single task 154, including action series 158. Procedure data can include information on tasks 154 making up a phase 152 of a medical procedure, including task series 160. Procedure data 150 can include information or data on various medical procedure phases 152, as well as surgical or target outcomes 198.

[0111] Procedure type 150 can include any information on type of the medical procedure performed, such as a name or designation of a medical procedure (e.g., open heart surgery,appendectomy, cholecystectomy, knee arthroscopy or a coronary artery bypass graft). Different procedure types can include certain phases 152 of the procedure, such as general portions of a medical procedure that can include multiple tasks 154. Phase 152 can include, for example, a pre-operative phase involving preparations for a surgery, an incision phase in which incisions are made to access an area on which to perform an action or an intervention, a surgical intervention phase in which medical interventions are implemented, a closure phase (e.g., suturing and hemostasis tasks) and a post-operative phase.

[0112] Task 154 of a medical procedure can include any set or arrangement of actions 156 to complete a particular portion of a phase 152. Tasks 154 can include activities or actions taken in a particular order (e.g., one after another) to complete a particular phase 152 of the procedure. Tasks 154 can include, for example, preparation of surgical instruments, making an incision, placing a retractor or other medical instrument 112 at a particular position to maintain a clear access to a tissue, perform a particular tissue manipulation or excision, administering of an anesthetic, suturing a wound, or any other tasks 154 of a medical procedure. Task 154 can include multiple actions 156 implemented in a particular order to complete the task 154. A phase 152 can include multiple tasks 154 implemented in a particular order to complete the phase 152.

[0113] Tasks 154 can be arranged or organized in a particular order, or an arrangement (e.g., a task series 160). The arrangement or order of tasks 154 can be tailored to implement or complete a particular phase 152. For example, a particular order or arrangement of several tasks 154 of particular type can be unique to or indicative of a particular phase 152. ML models 136 can be trained to detect tasks 154 based at least one multiple specific actions 156 arranged in a particular order.

[0114] Task series 160 can include multiple specific tasks 154 arranged in particular order that is indicative of a phase 152. Task series 160 can include, for example, multiple tasks 154 ordered one after another to complete a particular phase 152 By training ML models 136 to recognize individual tasks 154 and the order or arrangement of these tasks 154, the ML model 136 can determine or identify a particular phase 152 of a plurality of phases that is being completed or that has been completed.

[0115] Actions 156 of a medical procedure can include any actions or movements performed during the course of completing a task 154. For instance, an action 156 can include a movement or gesture by one or more medical instruments 112 during the implementation of atask 154. Actions 156 can include a movement or a maneuver by a particular surgical instrument 112, such as from one location to another, an act of grabbing a tissue, an act of piercing a tissue, an act of placing a thread to suture a wound any one of which can be done in a particular order of movements or motions (e.g., actions 156) in order to complete a task 154.

[0116] Actions 156 can be arranged or organized in action series 158. Action series 158 can include multiple actions 156 that are specifically selected from a larger pool of actions and arranged in a particular order in order to complete a particular task 154. For example, a particular order or arrangement of several actions 156 of a particular type can be unique to or indicative of a particular task 154 that they accomplish. By training ML models 136 to recognize individual actions 156 and the order or arrangement of such individual actions 156, the ML model 136 can determine or identify the particular task 154 that is being completed or accomplished.

[0117] Series selector 135 can include any combination of hardware and software for detecting, identifying, selecting or recognizing action series 158 or task series 160. Series selector 135 can include the functionality for detecting or recognizing individual action series 160 or task series 160 based on monitoring or analysis of data from any data streams 170 (e.g., 172, 174 or 176). Series selector 135 can utilize one or more ML models 136 for identification of any one or more of action series 158 or task series 160. Series selector 135 can be configured to identify action series 158 of the RMS 120 to perform, or indicative of, one or more tasks 154 performed by the RMS 120. Series selector 135 can be configured to identify task series 160 of the RMS 120 to perform, or indicative of, one or more phases 152 performed by the RMS 120. Series selector 135 can identify any action series 158 or task series 160 by sampling incoming data from data streams 170 to input the data samples into one or more ML models 136 for detecting any one or more of action series 158 and task series 160 based on, for example kinematics data 172 on movement of medical instruments 112, video stream data (e.g., 174) showing movements of medical instruments 112 and gestures of the surgeon, or using any other data.

[0118] Target outcomes 198, also referred to as surgical outcomes 198, can include any desired outcome or goal of a particular medical procedure. Target outcome 198 can include a particular medical procedure that can be adjusted or modified to accommodate a particular procedure characteristic 162. For instance, a target outcome 198 can be a desired surgical outcome for a medical procedure that is modified or adjusted to accommodate particular medical characteristic of a patient, based on patient data 164. For example, patient data 164 caninclude that a patient has a particular medical condition or an issue for which a medical procedure should adjust some of the actions 156 or tasks 154 or perform tasks 154 in a particular (e.g., adjusted) way. For instance, a target outcome 198 can be adjusted as to complete the medical procedure, for a patient that easily bleeds or has a hard time stopping the bleeding, a target outcome 198 can include rearranging or adjusting the actions 156, tasks 154 and phases 152 to minimize the number of cuts or bleeding. Depending on patient data 164 procedure characteristics 162 can be identified to modify the target outcomes 198 to conform to the preferences of the patient. ML models 136 can be trained to identify, from a pool of surgeon profiles 144, a particular surgeon profile 144 whose history of chosen tasks 154 and their corresponding performance metrics 182 exceeds that of other surgeons, thereby making this surgeon most suitable for a particular medical procedure.

[0119] Outcome identifier 137 can include any combination of hardware and software for detecting, identifying, selecting or recognizing action target outcomes 198. Outcome identifier 137 can include the functionality for detecting or recognizing individual target outcomes 198 based on monitoring or analysis of data from any data streams 170 (e.g., 172, 174 or 176). Outcome identifier 137 can utilize one or more ML models 136 for identification of any one or more of target outcomes 198. Outcome identifier 137 can be configured to identify target outcomes 198 for one or more actions 156, tasks 154, phases 152, action series 158 or task series 160 to be selected or performed by the RMS 120. Outcome identifier 137 can be configured to identify target outcomes 198 for any particular determination or selection of action series 158, or task series 160, configuration 118, any performance metrics 182 or surgeon profile 144 (e.g., based on procedure characteristics 162). Outcome identifier 137 can make the determinations by sampling incoming data from data streams 170 to input the data samples into one or more ML models 136.

[0120] Procedure characteristic 162 can include any feature or attribute for a medical procedure. Procedure characteristic 162 can include a feature corresponding to a patient data 164, such as a medical history or medical condition of the patient. Procedure characteristic 162 can include certain medical conditions identified in the patient data 164 causing the medical procedure to be adjusted or changed. As a result, certain actions 156 or tasks 154 performed can be modified, removed, rearranged or otherwise adjusted from a standard medical procedure of that type.

[0121] Surgeon profile 144 can include any collection of information about a surgeon. Surgeon profile 144 can include a unique identifier of the surgeon, and various surgeon data148, including a history of surgeon’s prior medical procedures, along with any phases 152, tasks 154 and actions 156 and their corresponding performance metrics 182. Surgeon profile 144 can include, or be related to, a metrics signature 146 (e.g., a fingerprint of all the performance metrics 182) of the surgeon. Surgeon profile 144 can include or enumerate training, specialized expertise, and a track record of performance with respect to any type of a procedure, phase 152, task 154 or action 156. Surgeon profile 144 can include information on medical education and training with respect to particular medical tasks or actions. Surgeon profile 144 can include a history of medical procedures performed by the surgeon, including any performance metrics 182 associated with such procedures.

[0122] Profile selector 139 can include any combination of hardware and software for detecting, identifying, selecting or recognizing action surgeon profiles 144. Profile selector 139 can include the functionality for detecting or recognizing individual surgeon profiles 144 based on monitoring or analysis of performance metrics 182, such as performance metrics of metrics signatures 146. Profile selector 139 can utilize one or more ML models 136 for identification of any one or more of surgeon profiles 144. Profile selector 139 can be configured to identify surgeon profiles 144 based on one or more target outcomes 198 in view of metrics signature 146. Profile selector 139 can be configured to identify surgeon profile 144 for any particular determination or selection of performance metric 182 pertaining to any action 156, task 154, phase 152, action series 158, or task series 160, configuration 118 or procedure characteristics 162. Profile selector 139 can make the determinations by identifying performance metrics 182 from metrics signature 146 and procedure characteristics 162 (e.g., patient data 164) into one or more ML models 136 trained to identify matching signature performance metrics 182 for a given patient data 164.

[0123] Metrics signature 146 (e.g., surgeon’s OPI fingerprint) can include any collection of performance metrics 182 for the surgeon. Metrics signature 146 can include compilation of success rates, complication rates, patient outcomes, and surgical proficiency scores. Metrics signature 146 can include a unique set of performance indicators and metrics distinguishing each surgeon based on surgeon’s historical data, particular proficiency with respect to certain phases 152, tasks 154 or actions 156. Metrics signature 146 can be used to identify medical procedures with particular procedure characteristics 162 (e.g., based on patient data 164) for which the surgeon is most suitable. Metrics signature 146 can include OPIs indicative of a surgeon's personalized approach, technique, and individual capabilities. Performance metrics 182 for the metrics signature 146 can include data on precision, speed, accuracy, complicationrates, patient outcomes, types of phases 152 in which the surgeon exceeds a threshold 184, types of tasks 154 in which the surgeon exceeds the threshold level, or any other information.

[0124] Surgeon data 148 can include any information on a surgeon. Surgeon data 148 can include educational background, training history, certification status, and professional affiliations. Surgeon data 148 can include a surgeon identifier, name, years of experience, or professional or personal data. Surgeon data 148 can include performance metrics 182 of the surgeon gathered across any number of medical procedures. Surgeon data 148 can include information on types of surgeries performed, success rates, complication rates, patient outcomes, and proficiency scores. Surgeon data 148 an include metrics signature 146 (e.g., fingerprint OPI) and availability data (e.g., schedule of surgeon’s work time and available slots for new medical procedures).

[0125] The data repository 168 can include one or more data files, data structures, arrays, values, or other information that facilitates operation of the data processing system 130. The data repository 168 can include one or more local or distributed databases and can include a database management system. The data repository 168 can include, maintain, or manage a data stream 170. The data stream 170 can include or be formed from one or more of a video stream, image stream, stream of sensor measurements, event stream, or kinematics stream. The data stream 170 can include data collected by one or more data capture devices 110, such as a set of 3D sensors from a variety of angles or vantage points with respect to the procedure activity (e.g., point or area of surgery).

[0126] Data stream 170 can include video data, which can include a series of video frames formed or organized into video fragments, such as video fragments of about 1, 2, 3, 4, 5, 10 or 15 seconds of a video. Each second of the video can include, for example, 30, 45, 60, 90 or 120 video frames 308 per second. Data stream 170 can include a stream of events data 176 which can include a stream of event data or information, such as packets, which identify or convey a state of the robotic medical system 120 or an event that occurred in association with the robotic medical system 120. Events data 176 can include information on a state of the RMS 120 indicating whether a medical instrument 112 is calibrated, adjusted or includes a manipulator arm installed on an RMS 120. Event data 176 can include data on whether an RMS 120 is fully functional (e.g., without errors) during the procedure. For example, when a medical instrument 112 is installed on a manipulator arm of the RMS 120, a signal or data packet(s) can be generated indicating that the medical instrument 112 has been installed on the manipulator arm of the RMS 120.

[0127] Data stream 170 can include a stream of kinematics data 172, which can refer to or include data associated with one or more of the manipulator arms or medical tools 112 (e.g., instruments) attached to the manipulator arms, such as arm movements, locations or positioning. Data corresponding to medical tools 112 can be captured or detected by one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. The kinematics data 172 can include sensor data along with time stamps and an indication of the medical tool 112 or type of medical tool 112 associated with the data stream 170.

[0128] Performance function 180 can include any combination of hardware and software for generating, using or managing performance metrics 182. Performance function 180 can generate performance metrics using ML models 136, such as performance metrics 182 or objective performance indicators (OPIs) of actions 156, tasks 154, phases 152, action series 158, task series 160 or their relation with target outcomes 198. Performance function 180 can utilize any performance metrics 182 to identify the most efficient or most suitable action series 158 (e.g., selection and order or arrangement of individual actions 156) or task series 160 (e.g., selection and order or arrangement of individual tasks 154) to achieve a particular target outcome 198. Performance function 180 can manage performance metrics 182 for making determinations using multiple ML models 136. Performance function 180 can include or be implemented using one or more ML models 136. Performance function 180 can include or utilize OPIs corresponding to particular surgeon profiles 144 (e.g., metrics signature 146) to determine most suitable surgeons for particular patients, based on patient data 164.Performance function 180 can include a scoring function for determining or implementing confidence score determinations with respect to any determinations. For instance, performance function 180 can determine confidence scores on behalf of chatbots 138 in terms of the confidence of the quality of their responses 142 for various queries 140. Performance function 180 can utilize ML models 136 for the confidence score determination.

[0129] Performance metrics 182 (e.g., OPIs) can include any values, indicators or metrics for any aspect or portion of medical procedures performed using RMS 120, such as any phase 152, task 154 or action 156. Performance metrics 182 can include any values, indicators or metrics indicative of a surgeon’s ability to perform particular aspects of a medical procedure, such as any phase 152, task 154 or action 156. Performance metrics 182 can include values indicative of aspects of surgeon’s productivity, quality of care, timeliness, or specialized skills. Performance metrics 182 level of consistency with which particular tasks related to one ormore medical procedures, patients or surgeons. Performance metrics 182 can be indicative of a surgeon’s productivity, quality, timeliness, customer satisfaction, specialized skills or abilities, success rates with respect to particular medical procedures, phases 152 of medical procedures, particular tasks 154 or actions 156.

[0130] Performance metrics 182 can include values or series of characters indicative of a value, quality, preference or performance of a series or combination of factors, including actions 156, tasks 154, phases 152, or medical procedures of one or more types. Performance metrics 182 can indicate a performance of a series or arrangement of tasks 154 or phases 152. For instance, performance metric 182 can be indicative of a performance or OPI of an action series 158 or of a performance or OPI of a task series 160. Performance metric 182 for action series 158 or task series 160 can be determined in connection with a particular configuration 118, a particular procedure characteristic 162 for a certain procedure for a patient with a special medical condition. Performance metrics 182 can include any metrics or OPIs corresponding to a particular selection or mapping of a surgeon (e.g., surgeon profile 144 based on surgeon’s metrics signature 146) with respective procedure characteristic 162 or patient data 164.

[0131] Performance metrics 182 can include any type and form of OPIs. For example, performance metric 182 can include an OPI of a duration, which can be expressed in the units of minutes and can correspond to a total time spent to perform a particular case, phase or a step. Performance metric 182 can include an OPI of a maximum force, which can be expressed in the units of Newtons (N) and correspond to the maximum detected force for a medical instrument. Performance metric 182 can include an OPI of an average force, which can be expressed in N and correspond to the average detected force for a medical instrument. Performance metric 182 can include an OPI of a time above a threshold N of force (e.g., time above 6.5N), which can be expressed in the units of % and correspond to the percentage of time with force applied above the threshold force amount. Performance metric 182 can include an OPI of an endoscope clutch count, which can be expressed in the units of numbers of a count, and which can correspond to the number of endoscope clutches performed on the console.Performance metric 182 can include an OPI of a hand controller clutch count, which can be expressed in numbers of a count, and which can correspond to the number of finger clutches performed on either hand controller on this console. Performance metric 182 can include an OPI of an energy pedal count, which can be expressed in a number of a count and correspond to the number of energy pedal presses initiated on the console. Performance metric 182 can include an OPI of a total instrument path length, which can be expressed in meters andcorrespond to the path length traveled by all instrument tips on all manipulator arms of the RMS 120. Performance metric 182 can include an OPI of a total instrument angular path length, which can be expressed in radians and correspond to the total angular path length traveled by all instruments on all arms. Performance metric 182 can include an OPI of a hand controller movement percentage, which can be expressed in % and correspond to the proportion of time this hand controller was in motion, relative to the total time either hand controller was in motion on the given console. Performance metric 182 can include an OPI of a console movement percentage, which can be expressed in % and correspond to the proportion of time either hand controller was in motion on this console, relative to the total duration.Performance metric 182 can include an OPI of an instrument movement duration, which can be expressed in minutes and correspond to the total time the tip of the particular instrument type was in motion on any manipulator arm of the RMS 120. Performance metric 182 can include an OPI of a hand controller movement duration, which can be expressed in minutes and correspond to the total time this hand controller was in motion on this console. Performance metric 182 can include an OPI of a console movement duration, which can be expressed in minutes and correspond to the total time either hand controller was in motion on this console. Performance metric 182 can include an OPI of an arm swap count, which can be expressed in swaps and correspond to the number of arm swaps performed on the given console.Performance metric 182 can include an OPI of a head out count, which can be expressed in the number or count of events and which can correspond to the number of head out events on this console. Performance metric 182 can include an OPI of a head out rate, which can be expressed in a count over a time period (e.g., 1 / hr) and which can correspond to the rate of head out events on this console.

[0132] For example, performance metric 182 can include an OPI of an instrument path length, which can be expressed in meters and correspond to the path length traveled by the tip of this instrument type on any arm. Performance metric 182 can include an OPI of a hand controller path length, which can be expressed in meters and correspond to the path length traveled by this hand controller. Performance metric 182 can include an OPI of an average hand controller speed, which can be expressed in meters per second and correspond to the average speed of this hand controller (while it was in motion). Performance metric 182 can include an OPI of a hand controller workspace volume, which can be expressed in the units of area (e.g., square centimeters) and correspond to the volume of a spheroid that encompasses 75% of this hand controller's positions. Performance metric 182 can include an OPI of aconsole workspace volume, which can be expressed in the units of area (e.g., square centimeters) and correspond to the volume of a spheroid that encompasses 75% of hand controller positions on this console. Performance metric 182 can include an OPI of an instrument install duration, which can be expressed in minutes and correspond to the total time this instrument type was installed on any arm. Performance metric 182 can include an OPI of an average energy pedal duration, which can be expressed in seconds and correspond to the average time an energy pedal was pressed on this console (per press). Performance metric 182 can include an OPI of a total energy pedal duration, which can be expressed in minutes and correspond to the total time an energy pedal was pressed on this console. Performance metric 182 can include an OPI of an endoscope clutch rate, which can be expressed in in a rate over time (e.g., 1 / min) and which can correspond to the rate of endoscope clutches (while either hand controller was in motion) performed on this console. Performance metric 182 can include an OPI of a hand controller clutch rate, which can be expressed the rate over time (e.g., in 1 / min) and which can correspond to the rate of hand controller clutches (while either hand controller was in motion) performed on this console. Performance metric 182 can include an OPI of an arm swap rate, which can be expressed in the rate over time (e.g., 1 / min) and which can correspond to the rate of arm swaps (while either hand controller was in motion) performed on this console. Performance metric 182 can include an OPI of an energy pedal rate, which can be expressed in the rate over time (e.g., 1 / min) and correspond to the rate of energy pedal presses initiated (while either hand controller was in motion) on this console. Performance metric 182 can include an OPI of an instrument angular path length, which can be expressed in radians and correspond to the angular path length traveled (e.g., of endowrist) by the given instrument type on any arm. Performance metric 182 can include an OPI of an instrument speed peak count, which can be expressed in peaks and correspond to the number of changes between acceleration and deceleration with this instrument type on any arm. Performance metric 182 can include an OPI of an average instrument speed, which can be expressed in meters per second and correspond to the average speed of the tip of this instrument type on any arm (e.g., while the instrument was in motion). Performance metric 182 can include an OPI of an instrument smoothness (e.g., an abrupt motion or a jerk), which can be a measurement of smoothness for this instrument type on any arm. For instance, the instrument smoothness OPI can sum up the jerk over time, normalize it for a duration of time and maximum speed, and then transform it by taking the negative log, such that a larger value corresponds to a smoother movement.

[0133] Threshold 184 can include any value for comparing another value or establishing a condition. Threshold 184 can include a value for selecting a particular performance level, such as a value against which performance metrics 182 are compared. Threshold 184 can include a maximum performance metric 182 out of a plurality of performance metrics 182. Threshold 184 can include a particular level of a metrics signature 146 for identifying a most suitable (e.g., most closely matching) surgical profile 144 to a procedure characteristic 162 or patient data 164. For instance, a threshold 184 can correspond to a particular metrics signature 146 medical condition of a patient to address with a particular surgeon who has particular skill level in certain tasks 154 or phases 152 of a procedure that is most important or most sensitive to the particular patient.

[0134] State function 186 can include any combination of hardware and software for determining a state of a medical procedure (e.g., a state information 188). State function 186 can include functionality (e.g., ML model 136) to utilize data and identify certain actions 156, tasks 154 and phases 152 and identify the state of the medical operation (e.g., at what state is the medical operation currently). State function 186 can determine or identify the state information 188, such as a current state of a medical procedure using any number of ML models 136. State function 186 can utilize ML models 136 to identify tasks 154, phases 152 and actions 156 to predict the medical procedure and the specific stage (e.g., phase 152, task 154 and action 156) within the procedure to identify the state. For instance, state function 186 can include the functionality (e.g., ML model 136) to identify a particular task 154 within a particular task series 160, or a particular phase 152 of a medical procedure. State function can include the functionality (e.g., ML model 136) to identify a particular action 156 within a particular action series 158.

[0135] Ranking function 166 can include any combination of hardware and software for ranking data. Ranking function 166 can include the functionality to rank any features or parameters, such as actions 156, tasks 154, phases 152, action series 158, task series 160, procedure characteristics 162, surgeon profiles 144, metrics signatures 146, surgeon data 148 or state information 188. Ranking function 166 can include ranking of performance metrics 182 for particular task series 160 or action series 158. Ranking function 166 can utilize one or more ML models 136 to implement the ranking based on documentation (e.g., procedure data 150).

[0136] User interfaces 190 can include any combination of hardware and software to provide interface to a user (e.g., surgeon performing a medical procedure via an RMS 120). User interface 190 can include a web application or a computer (e.g., smartphone) applicationproviding access or functionality of the DPS 130. For instance, user interface 190 can provide a web application (e.g., application provided via a server device or a cloud-based system) to allow users to select and access various datasets of (e.g., dataset selection of procedure data 150), upload of datasets and implement user-defined featurization of the user interfaces 190. User interface 190 can include user-guided or user controlled interactive functions for providing analyses per user requests (e.g., queries 140) and can include a report generating interface for providing reports 194.

[0137] User interface 190 can include backend functionality for uploading and processing data provided by users (e.g., data ingestion). For instance, users can use the user interface 190 (e.g., a web-based application) to upload, access or process clinical, financial, operative and surgical data from hospitals, medical institutions or third parties. User interface 190 can be designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the RMS 120 or another device, such as a client’s personal computer. User interface 190 can include a graphical user interface for a touch screen or a computer. The graphical user interface can include, for example, a window for displaying video data, or indications (e.g., recommendations 192 or reports). User interface 190 can be a part of an application executed on a DPS 130 and made accessible to a client via one or more agents 138. User interface 190 can provide data for presentation via a display, such as a display 116, and can depict, illustrate, render, present, or otherwise provide indications indicating determinations (e.g., outputs) of the ML models 136.

[0138] User interface 190 can include the functionality for providing access to, filtering, transforming and visualizing various OPI data, including providing access to, filtering, transforming, and visualizing OPI data computed from segments within surgical cases identifying specific clinically relevant activities. For instance, a user can preview the case data in a table format that interactively updates with the user selection of specific filters that subselect specific cases from the dataset. For example, a user can sub-select data by date, surgical procedure type, surgeon identifier, and by values within specific columns in the data. For example, visualizations of data can be provided, including tables to summarize important features of the dataset that allow the user to understand what data has been selected based on their interactions.

[0139] User interface 190 can include various clinical or operative data association and management functionalities. For instance, user interface 190 can allow multiple data sources to associate clinical and operative data to the robotic OPI data for each individual medicalprocedure. Data sources provided can include documentation for clinical intelligence specialists who review the case, users, hospitals where the surgery is to be performed or third party vendors with access to clinical data related to that surgical case. When the data is uploaded by the user directly, the user can identify which columns in their data include clinical or operative predictors and which columns can include surgical outcomes that should be targeted for analysis.

[0140] User interface 190 (e.g., web-based application) can include default settings based on column names or other methods. When the data source is a third party, the DPS 130 can perform entity resolution and match the case record from the external data source to the case record maintained by the system 100. For example, while accessing data, the user can have the option to attach other relevant clinical and operative factors by uploading a dataset to join with the existing data at the case level. For instance, the user can select an existing dataset they had uploaded previously to join at the case-level to the currently selected data. For instance, the user can also edit previously uploaded data with new values and columns or update existing values and columns.

[0141] For example, the user interface 190 can allow the user to upload a data set, while the web-based application (e.g., the application operating with the DPS 130) can run an automated data quality check to check if data types from a single column match, generate warnings for missing data, generate visualizations of values for each uploaded column, and indicate how many cases with OPI data now also have data uploaded by the user. For example, when the user uploads a dataset, the user can define whether specific columns in the dataset are predictors, surgical outcomes (targets), or both. This information can be used later during the automated analysis.

[0142] For example, user interface 190 can include the features in which some clinical and operative data may be associated with the case as they can be derived from manual annotations by clinical intelligence specialists who reviewed the case data. The user can be made aware of this through the data preview table and the visualization features of the user interface 190. For example, clinical and operative data input by the user through the RMS 120 or other applications can be integrated with the web-based application executed on the DPS 130 and associated with procedure data 150 (e.g., the OPI data). The user can be made aware of this functionality through the data preview table and the visualizations features of the user interface 190.

[0143] For example, user interface 190 can receiver and process (e.g., ingest) various clinical, operative, and financial data from the hospital / s where the cases were performed. This data can be shared by the hospitals through manual or automated secure file transfers completed on a regular basis. During ingestion, the web-based application can solve entity resolution by using surgeon identifiers, procedure date, procedure time, and procedure name to match case data across two different records using a fuzzy matching technique. This technique can include breaking each case record into subsets by surgeon identifier and solving multiple linear sum assignment problems where the cost is defined as the difference in procedure start time across all cases. The matching results of these optimizations are further filtered by procedure name to remove spurious assignments of cases across records.

[0144] User interface 190 can include a data analysis functionality. For instance, when the user selects a specific surgical outcome, the web based application can generate analyses of how OPIs from different segments of surgical activity along with other clinical and operative predictors correlate with that outcome. The user interface 190 can include report generation functionality by which any analysis can generate a report containing both text and figures relevant to the analysis. Reports 194 generated by the user interface 190 can include the data used to run the analysis with appropriate tables and figures, the methods of the analysis and the results of the analysis summarized in text and figure with appropriate captions.

[0145] For example, a surgeon can request the user interface 190 to generate a summary report, such as: information about the dataset selected for analysis and the surgical outcome of interest (e.g., procedure type, number of cases, and a table with variables in analysis with relevant descriptions of the values in the columns), plots about the distribution of values for the surgical outcome and predictors, each bar plot from the automated analysis with captions that explain the content of the plots, additional plots for each top- 10 OPI, clinical, and operative predictor that have a statistically significant relationship or text explaining the analysis methods (e.g., automated model building and statistical tests) and text summarizing the relevant results including the step / phase, OPIs, clinical, operative, and financial predictors most relevant to the surgical outcome, including relevant sources and reasoning.

[0146] User interface 190 can allow the user can select a specific surgical outcome of interest and level of ontological hierarchy in the case segmentation at which the analysis should be run, including the entire case, phase, or step, including in an increasing order of granularity. The web based application of the user interface can consolidate the OPI data from theappropriate ontological level along with other clinical, financial, or operative predictors for the analysis.

[0147] The user interface 190 can automatically compare all predictors against the surgical outcome of interest and automatically exclude any predictors that are redundant with the surgical outcome. The DPS 130 and the user interface 190 can internally handle missing data by excluding all rows where the surgical outcome of interest selected by the user is missing, excluding all predictor columns that have missing data above a specific threshold that would nullify analysis results, or imputing missing data in predictor columns with missingness below the above-mentioned threshold.

[0148] The user interface 190 can subset the data appropriately to build separate models for each step or phase, depending on the user selection. For instance, for each step or phase of a procedure, the user interface can use the surgical outcome column to determine the appropriate model-type (e.g., regression or classification), separate the data into entries (e.g., at least three or up to 5 folds for but up to N folds for model cross-validation). The user interface 190 can be used to utilize ML framework 132 to train a model (e.g., 136) to predict the surgical outcome from all predictors in the folds and test the model performance on the last fold (e.g., repeat this N times). The DPS 130 can accumulate predictor importance from each modeling fold above, save the average model performance and top most important predictors (e.g., top 10 or top N), including OPIs, clinical, financial, and operative factors.

[0149] For instance, the DPS 130 can resend the steps or phases ordered by average model (e.g., 136) performance to the user in a bar chart or a plot. For instance, the user can select one phase or step to populate a new bar plot that contains the top-10 predictors including OPIs ordered by directional influence of the predictor on the surgical outcome and determine whether this configuration increases or decreases the value or probability of occurrence. When a user hovers over each predictor in the user interface 190, a human-readable explanation of the predictor importance can be presented, including a text explaining a how one standard deviation or unit change in the predictor changes the value of the surgical outcome. From this predictor importance output or a plot, the user can select a single predictor which generates a new plot that plots the values of this predictor only (y-axis) along with the surgical outcome of interest (x-axis).

[0150] For instance, a user interface 190 can provide an example output or a plot that can be determined based on the parameters. These parameters which can include predictorparameters that can be mapped against surgical outcomes, such as contingency table heatmap, vertical boxplot, horizontal boxplot and scatter plot. The predictor parameters and the surgical outcomes can each include categorical and numerical features, as presented in Table 1 :Table 1 : Example table depicting predictors and surgical outcomes.

[0151] In an example, a user interface 190 can perform a statistical test to determine whether the relationship between the selected predictor and the selected surgical outcome is statistically significant. The appropriate statistical test can be output or selected based on the following predictor and surgical outcome parameters: Chi-square, Ranksum, Anova and Pearson’s R, each of which can include a binary, categorical or a numerical feature for mapping, such as shown in Table 2:Table 2: Example table depicting binary categorical and numerical predictors and surgical outcomes.

[0152] Recommendations 192 can include any suggestions DPS 130 can provide to a user. For instance, agent 138 can utilize ML models 136 to process input data and provide a suggestion or a recommendation to perform particular tasks 154 or actions 156 in a particular way or using a particular medical instrument 112. Recommendation 192 can include real-time guidance on procedural steps, based on established best practices and clinical guidelines. Recommendation 192 can include an indication to display on a user interface 190 in order to assist a surgeon in performing particular actions 156, tasks 154 or phases 152 of the medical procedure in order to achieve a target outcome 198.

[0153] Reports 194 can include any compilation of analysis to present to a user. Report 194 can include a text or content generated based on documentation (e.g., procedure data 150) or training data 178. Report 194 can include a ML model generated extracted patterns of data or trends. Reports 194 can include sections focusing on particular performance metrics 182, tasks 154, phases 152, actions 156 or medical procedures overall. Report 194 can be interactive,allowing the user to select the sources on which the generated content is based and further analyze the reasoning behind the content.

[0154] Configuration function 196 can include any combination of hardware and software for generating configurations 118 for RMS 120. Configuration function 196 can include the functionality to identify, select or generate settings, configurations, operating parameters or constraints (e.g., configurations 118) for RMS 120. Configuration function 196 can generate configurations 118 based on historical settings and preferences, as well as performance metrics 182 of the surgeon associated with such settings and configurations 118. Configuration function 196 can include, or function based on, one or more ML models 136 trained to identify most suitable configurations 118, such as settings or arrangements of medical instruments 112, objects 106, sensors 104, data capture devices 110, visualization tools 114 and displays 116, HMD 122 or any other feature or functionality.

[0155] Configurations 118 can include any combination of settings, operating parameters, system arrangements or adjustments to establish, correct, control or otherwise manage operation or performance of the RMS 120, or any other feature of the medical environment 102. Configurations 118 can include control settings for HMD 122, data capture devices 110, sensors 104, objects 106, visualization tools 114, displays 116 or medical instruments 112. Configuration 118 can include, for example, adjustments, settings or operating parameters for medical instruments 112. Configuration 118 can include arrangement of medical instruments 112 on an RMS 120, their installation, reconfiguration or any type and form of adjustment.

[0156] Head-Mounted Display (HMD) 122 can include any combination of hardware and software that can be worn on a user’s head to provide a user with a virtual reality, augmented reality, live-view or a mixed reality functionality. HMD 122 can include a system or a device, which can include a pair of glasses or a helmet, which a user can wear over the user’s eyes. HMD 122 can include one or more displays 116 that can be positioned in front of, adjacent to, or proximate to the user’s eyes, providing the user with a visual interface that replaces or augments the user’s perception of the surrounding environment.

[0157] HMD 122 can include any number of sensors 104 to track head movements or eye trackers 124, to facilitate the virtual or augmented content to respond dynamically to the user's perspective as well as receive user inputs or selections. Eye trackers 124 can include, for example, cameras or image capture devices tracking the user’s pupils, iris or other portions of user’s eye. HMD 122 can be utilized by users, such as medical professionals, to participate inmedical procedures remotely. HMD 122 can be worn on a user’s head or can be fixed to a specific location, providing stability, and display extended reality (XR) that includes augmented, virtual, live-view or mixed realities, depicting various features of the medical environment 102. HMD 122 can include sensors 104 providing information about the user's location, orientation, and gaze direction, allowing the HMD 122 to generate corresponding views and images.

[0158] HMD 122 can include sensors 104, a communication interface, and a visualization tool 114 to detect its location, orientation, and the user's gaze direction. Using this information, the HMD 122 can render images representing the current state of the medical environment 102 and integrate real-time data from different capture devices 110. HMD 122 can include the functionality to deliver extended or mixed reality content, including by rendering images, videos, or audios based on user interactions. HMD 122 can be included in a medical environment 102 and can be considered an object to be simulated, rendered or displayed as an AR object, VR object or a VR simulation.

[0159] Eye tracker 124 can include any combination of hardware and software for measuring, detecting, monitors and analyzing a user's gaze, including eye movements and locations. Eye tracker 124 can include the functionality to use the user’s gaze (e.g., movements and locations of the eyes) to determine various user commands or actions, such as selections of features displayed. Eye tracker 124 can utilize sensor data (e.g., camera images) or infrared sensors to track the user's eye movements in real-time, or gestures indicated by eye movements (e.g., eye tracking) or movement of any other part of the user’s body, allowing for accurate detection of where the user is looking with respect to the features displayed in the extended reality on the display 116. Eye tracker 124 can interpret the user’s visual focus and trigger actions such as selecting options, navigating through menus, activating controls, or even initiating specific commands without the need for physical input devices.

[0160] Hand tracker 126 can include any combination of hardware and software that uses data from various sensors 104 to monitor and analyze the movements of any part of a user’s body, such as users’ hands, arms, head movement or any other portion of the user’s body. Hand tracker 126 can include or utilize computer vision techniques (e.g., objects generated by ML model 136) to use images or videos of user’s hands to detect or discern various gestures that the user makes (e.g., hand tracking). Hand tracker 126 can track movements of user’s hands, arms, head, torso, legs or any other portion of body of the user. Body movements can be tracked individually (e.g., movements of hands, arms, eyes) or collectively (e.g., body of a plurality ofbody parts moving together). Hand tracker 126 can operate in real time and allow the user to seamlessly interact with DPS 130. For instance, the user can use the hand tracker 126 to (e.g., via sensors 104) to control medical tools 112 in the medical environment 102. Hand tracker 126 can detect gestures, such as finger movements for scrolling or selecting items, hand waving for navigation or moving of objects, pinching or expanding fingers for zooming in or out, and making specific hand movements or shapes to trigger specific commands or actions.

[0161] Voice controller 128 can include any combination of hardware and software that uses voice for controlling devices. Voice controller 128 can include a microphone to detect a user’s verbal command or an instruction. Voice controller 128 can detect statements, such as code words, voice commands which can be used to generate queries for chatbots, or to trigger a command for the user to control RMS 120 via HMD 122. For example, a voice controller 128 can facilitate the user to provide a verbal instruction to an HMD 122 to implement an action, such as a switch between different modes of operation.

[0162] The data processing system 130 can interface with, communicate with, or otherwise receive or provide information with one or more component of system 100 via network 101, including, for example, the RMS 120. The data processing system 130, RMS 120 and devices in the medical environment 102 can each include at least one logic device such as a computing device having a processor to communicate via the network 101. The DPS 130, any portion of the ML framework 132, the RMS 120 or a client device that can be communicatively coupled with the DPS or the RMS 120 via the network 101, can each include at least one computation resource, server, processor or memory for processing data. For example, the data processing system 130 can include a plurality of computation resources or processors coupled with memory.

[0163] The data processing system 130, as well as any of its components (e.g., ML framework) can each be a part of or include a cloud computing environment functionality or features. The data processing system 130 can include multiple, logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can also be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform.

[0164] The data processing system 130, or components thereof can include a physical or virtual computer system operatively coupled, or associated with, the medical environment 102. In some embodiments, the data processing system 130, or components thereof can be coupled, or associated with, the medical environment 102 via a network 101, either directly or directly through an intermediate computing device or system. The network 101 can be any type or form of network. The geographical scope of the network can vary widely and can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 101 can assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network 101 can utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network 101 can be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0165] The data processing system 130, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 102 or remotely therefrom. Elements of the data processing system 130, or components thereof can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 130, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The data processing system 130, or components thereof, can include, or be associated with, one or more components or functionality of a computing including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the DPS 130 discussed herein.

[0166] FIG. 2 depicts a surgical system 200, in accordance with some embodiments. The surgical system 200 may be an example of the medical environment 102. The surgical system 200 may include a robotic medical system 205 (e.g., the robotic medical system 120), a user control system 210, and an auxiliary system 215 communicatively coupled one to another. A visualization tool 220 (e.g., the visualization tool 114) may be connected to the auxiliary system 215, which in turn may be connected to the robotic medical system 205. Thus, whenthe visualization tool 220 is connected to the auxiliary system 215 and this auxiliary system is connected to the robotic medical system 205, the visualization tool may be considered connected to the robotic medical system. In some embodiments, the visualization tool 220 may additionally or alternatively be directly connected to the robotic medical system 205.

[0167] The surgical system 200 may be used to perform a computer-assisted medical procedure on a patient 225. In some embodiments, surgical team may include a surgeon 230 A and additional medical personnel 230B-230D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who may assist with the surgical procedure or medical session. The medical session may include the surgical procedure being performed on the patient 225, as well as any pre-operative (e.g., which may include setup of the surgical system 200, including preparation of the patient 225 for the procedure), and post-operative (e.g., which may include clean up or post care of the patient), or other processes during the medical session. Although described in the context of a surgical procedure, the surgical system 200 may be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that may benefit from the accuracy and convenience of the surgical system.

[0168] The robotic medical system 205 can include a plurality of manipulator arms 23 SA- 235D to which a plurality of medical tools (e.g., the medical tool 112) can be coupled or installed. Each medical tool can be any suitable surgical tool (e.g., a tool having tissueinteraction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that can be used for a computer-assisted surgical procedure on the patient 225 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer- assisted surgical procedure on the patient). Although the robotic medical system 205 is shown as including four manipulator arms (e.g., the manipulator arms 235A-235D), in other embodiments, the robotic medical system can include greater than or fewer than four manipulator arms. Further, not all manipulator arms can have a medical tool installed thereto at all times of the medical session. Moreover, in some embodiments, a medical tool installed on a manipulator arm can be replaced with another medical tool as suitable.

[0169] One or more of the manipulator arms 235A-235D and / or the medical tools attached to manipulator arms can include one or more displacement transducers, orientational sensors, positional sensors, and / or other types of sensors and devices to measure parameters and / or generate kinematics information. One or more components of the surgical system 200 can be configured to use the measured parameters and / or the kinematics information to track (e.g.,determine poses of) and / or control the medical tools, as well as anything connected to the medical tools and / or the manipulator arms 235A-235D.

[0170] The user control system 210 can be used by the surgeon 230A to control (e.g., move) one or more of the manipulator arms 235A-235D and / or the medical tools connected to the manipulator arms. To facilitate control of the manipulator arms 235A-235D and track progression of the medical session, the user control system 210 can include a display (e.g., the display 116) that can provide the surgeon 230A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 225 as captured by a medical tool (e.g., the medical tool 112, which can be an endoscope) installed to one of the manipulator arms 235A-235D. The user control system 210 can include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 225 and generated by a stereoscopic imaging system can be viewed by the surgeon 230 A. In some embodiments, the user control system 210 can also receive images from the auxiliary system 215 and the visualization tool 220.

[0171] The surgeon 230A can use the imagery displayed by the user control system 210 to perform one or more procedures with one or more medical tools attached to the manipulator arms 235A-235D. To facilitate control of the manipulator arms 235A-235D and / or the medical tools installed thereto, the user control system 210 can include a set of controls. These controls can be manipulated by the surgeon 230 A to control movement of the manipulator arms 23 SA- 235D and / or the medical tools installed thereto. The controls can be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 230A to allow the surgeon to intuitively perform a procedure on the patient 225 using one or more medical tools installed to the manipulator arms 235A-235D.

[0172] The auxiliary system 215 can include one or more computing devices configured to perform processing operations within the surgical system 200. For example, the one or more computing devices can control and / or coordinate operations performed by various other components (e.g., the robotic medical system 205, the user control system 210) of the surgical system 200. A computing device included in the user control system 210 can transmit instructions to the robotic medical system 205 by way of the one or more computing devices of the auxiliary system 215. The auxiliary system 215 can receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical tools) attached to the robotic medical system 205, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices (e.g., the image capturedevices 110) can be located within the surgical system 200. These image capture devices can capture images from various viewpoints within the surgical system 200. These images (e.g., video streams) can be transmitted to the visualization tool 220, which can then passthrough those images to the auxiliary system 215 as a single combined data stream. The auxiliary system 215 can then transmit the single video stream (including any data stream received from the medical tool(s) of the robotic medical system 205) to present on a display (e.g., the display 116) of the user control system 210.

[0173] In some embodiments, the auxiliary system 215 can be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medical personnel 230B-230D) who might not have access to the user control system 210. Thus, the auxiliary system 215 can include a display 240 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 225 and / or the surgical procedure, and / or any other visual content (e.g., the single combined data stream). In some embodiments, display 240 can be a touchscreen display and / or include other features to allow the medical personnel 230A-230D to interact with the auxiliary system 215.

[0174] The robotic medical system 205, the user control system 210, and the auxiliary system 215 can be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 205, the user control system 210, and the auxiliary system 215 can be communicatively coupled by way of control lines 245, which can represent any wired or wireless communication link that can serve a particular implementation. Thus, the robotic medical system 205, the user control system 210, and the auxiliary system 215 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc. It is to be understood that the surgical system 200 can include other or additional components or elements that can be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0175] FIG. 3 depicts an example block diagram of an example computer system 300 is shown, in accordance with some embodiments. The computer system 300 can be any computing device used herein and can include or be used to implement a data processing system or its components. The computer system 300 includes at least one bus 305 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 310 or processing circuit coupled to the bus 305 for processing information. The computersystem 300 also includes at least one main memory 315, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 305 for storing information, and instructions to be executed by the processor 310. The main memory 315 can be used for storing information during execution of instructions by the processor 310. The computer system 300 can further include at least one read only memory (ROM) 320 or other static storage device coupled to the bus 305 for storing static information and instructions for the processor 310. A storage device 325, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus 305 to persistently store information and instructions.

[0176] The computer system 300 can be coupled via the bus 305 to a display 330, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 335, such as a keyboard or voice interface can be coupled to the bus 305 for communicating information and commands to the processor 310. The input device 335 can include a touch screen display (e.g., the display 330). The input device 335 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 310 and for controlling cursor movement on the display 330.

[0177] The processes, systems and methods described herein can be implemented by the computer system 300 in response to the processor 310 executing an arrangement of instructions contained in the main memory 315. Such instructions can be read into the main memory 315 from another computer-readable medium, such as the storage device 325. Execution of the arrangement of instructions contained in the main memory 315 can cause the processor 310 or the computer system 300 as a whole to perform the illustrative functionalities or processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 315. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0178] For example, technical solutions can include a system 100 that can have or utilize one or more processors 310 coupled with memory 315. The processors 310 can be configured using commands or instructions stored in any combination of memory 315, ROM 320 or storage 325. The processors 310 can be configured to access and read the stored instructions, which can cause the one or more processors 310 to implement various functionalities of the technical solutions.

[0179] For example, one or more processors 310 of system 100 be configured to identify, using a data stream 170 of a robotic medical system 120, an action 156 of the robotic medical system 120 to perform a task 154 of a plurality of tasks 154 of a medical procedure. The medical procedure can be performed using or by the robotic medical system 120. The RMS 120 can be operated by a user (e.g., a surgeon) associated with a surgeon profile 144. The identified action 156 can include a movement, a motion, an operation or an activity performed by a medical instrument 112, such as an act of touching, grabbing or moving a tissue of a patient.

[0180] The one or more processors 310 can be configured to identify, based on the action 156, a performance metric 182 for the task 154. The one or more processors 310 can be configured to generate, using the performance metric 182 and the task 154 input into one or more machine learning (ML) models 136, a plurality of series of tasks (e.g., task series 160) of the medical procedure. The one or more ML models 136 can be trained by a ML trainer 134 on a plurality of tasks 154 of a plurality of medical procedures associated with a plurality of performance metrics 182. The series of tasks 160 can be performed by the robotic medical system 120 in the medical procedure following the task 154.

[0181] The one or more processors 310 can be configured to rank the plurality of series of tasks 154 (e.g., 160) based on a plurality of performance metrics 182. The plurality of performance metrics 182 can corresponding to the plurality of series of tasks 154 (e.g., 160). The one or more processors 310 can be configured to determine, based on the ranking, a series of tasks 154 (e.g., 160) of the plurality of series of tasks 154 (e.g., 160) to cause the robotic medical system 120 to perform one or more actions of the series of tasks 154 (e.g., 160).

[0182] For instance, performance function 180 or one or more ML models 136 can identify a plurality of task series 160 that can accomplish a particular target outcome 198. The target outcome 198 can include completion of a particular phase 152 of a plurality of tasks 154, which can be accomplished using any one of a plurality of task series 160. The ranking function 166 can rank different task series 160 that are identified as options to achieve the given target outcome 198. The ranking function 166 can determine, based on the performance metrics 182 of the individual tasks 154 in each of the series of tasks 160, which of the series of tasks 154 has highest performance metrics 182. In response to identifying the series of tasks 160 that has highest performance metrics 182 overall, the DPS 130 can identify the given task series 160 and provide it in a recommendation 192 on a user interface 190.

[0183] The one or more processors 310 can be configured to identify a plurality of documents (e.g., training data 178) for a plurality of medical procedures performed using a plurality of robotic medical systems 120. The one or more processors 310 can be configured to determine the ranking for the plurality of series of tasks 154 (e.g., 160) using the plurality of documents. For example, one or more ML models 136 can be trained by an ML trainer 134 to utilize training data 178 (e.g., surgical documents, medical research data and hospital records) to determine the ranking of the plurality of task series 160 and identify the most suitable task series 150 of the plurality.

[0184] The one or more processors 310 can be configured to identify an agent for an ML model 136 trained on a plurality of documents corresponding to the plurality of series of tasks 154 (e.g., 160). The one or more processors 310 can be configured to provide, responsive to an input of a user of the robotic medical system 120 into the agent, a text corresponding to the series of tasks 154 (e.g., 160). The text can be generated by the ML model 136 based on a document of the plurality of documents. The text can include information on the series of tasks 160 selected or chosen.

[0185] In an example, a performance function 180 or one or more ML models 136 can identify a plurality of action series 158 that can accomplish a particular target outcome 198. The target outcome 198 can include completion of a particular phase 152 or a task 154, which can be accomplished using any one of a plurality of action series 158 to choose from. The ranking function 166 can rank different action series 158 that are identified as options to achieve the given target outcome 198. The ranking function 166 can determine, based on the performance metrics 182 of the individual actions 156 in each of the series of actions 158, which of the series of actions 156 (e.g., action series 158) includes highest performance metrics 182. In response to identifying the action series 158 with the highest performance metrics 182 overall, the DPS 130 can identify the given action series 158 and provide it in a recommendation 192 on a user interface 190.

[0186] The one or more processors 310 can be configured to generate the plurality of series using a plurality of documents for the plurality of series of tasks 154 (e.g., 160). The one or more processors 310 can be configured to validate the series of tasks 154 (e.g., 160) using at least an agent 138 for an ML model 136 of the one or more ML models 136 trained on the plurality of documents. For instance, one or more ML models 136 can be used to validate task series 160, responsive to a query 140 140 to verify that a given task series 160 results in a superior set of performance metrics 182 of all performance metrics. For instance, DPS 130 canprovide or determine an average or a mean of performance metrics 182 of all tasks 154 in the task series 160, or a performance metric 182 for a combination of all tasks 154 within a task series 160.

[0187] The one or more processors 310 can be configured to identify a second action of the robotic medical system 120 and determine, based on the second action, a second performance metric 182 for the task. The one or more processors 310 can be configured to update the ranking of the plurality of series of tasks 154 (e.g., 160) based on the second performance metric. The one or more processors 310 can be configured to determine, based at least on the updated ranking, a second series of tasks 154 (e.g., 160) of the plurality of series of tasks 154 (e.g., 160) to use for a remainder of the medical procedure.

[0188] The one or more processors 310 can be configured to rank the plurality of series of tasks 154 (e.g., task series 160) based on a performance metric 182 of the plurality of performance metrics 182 determined based on data on a recovery of a patient (e.g., patient data 164) following a prior medical procedure of the plurality of medical procedures in which the series of tasks 154 (e.g., 160) was applied. For example, patient data 164 can correspond to a plurality of patients that previously underwent a particular medical procedure with a particular task series 154. Performance metrics 182 of the given task series 160 or given action series 158 of those medical procedures from such patient data 164 can be used to determine whether to proceed with that particular task series 160 or an action series 158.

[0189] The one or more processors 310 can be configured to rank the plurality of series of tasks 154 (e.g., 160) based on a performance metric 182 of the plurality of performance metrics 182 determined based on an occurrence of a medical emergency during performance of a prior medical procedure of the plurality of medical procedures. The one or more processors 310 can be configured to identify a plurality of documents (e.g., training data 178, patient data 164 or surgeon data 148). These can include at least one of, or a combination of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system 120, or a data on a plurality of patients. The one or more processors 310 can be configured to use the plurality of documents to generate the plurality of series of tasks 154 (e.g., task series 160).

[0190] The one or more processors 310 can be configured to identify, using the one or more ML models 136 trained on a plurality of documents for the plurality of medical procedures, one or more tasks 154 of the medical procedure using a document of the pluralityof documents. The ML model 136 can identify a task series 160 based on the documentation on historical medical procedures in which same or similar task series 160 were utilized. The ML model 136 can make determination by comparing performance metrics 182 of the task series 160 with the performance metrics 182 of other task series 160 that can be used instead or as an alternative in the medical procedure. The one or more processors 310 can be configured to map the series of tasks 154 (e.g., 160) with the one or more tasks 154 using the performance metric 182 and select the task series 160 based on the mapping.

[0191] The one or more processors 310 can be configured to identify, using the one or more ML models 136 trained on a plurality of documents for the plurality of medical procedures, one or more actions 156 to use, based on the plurality of documents. The ML model 136 can identify an action series 158 based on the documentation on historical medical procedures in which same or similar action series 158 were utilized. The ML model 136 can make the determination by comparing performance metrics 182 of the action series 158 with the performance metrics 182 of other action series 158 that can be used instead or as an alternative in the medical procedure. The one or more processors 310 can be configured to map the series of actions 156 (e.g., action series 148) with the one or more actions 156 using the performance metric 182 and select the action series 158 based on the mapping.

[0192] The one or more processors 310 can be configured to generate a plurality of confidence scores for a plurality of mappings between the plurality of series of tasks 154 (e.g., 160) and the plurality of performance metrics 182. For instance, performance function 180 can generate confidence scores for determinations of performance metrics 182. Confidence scores can be generated to provide a level of confidence or a probability that a response 142 to a query 140 input into a chatbot 138 is correct. Confidence scores can be generated based on statistical analysis (e.g., mean, median, variance) of the performance metrics 182 of the task series 160 to choose from, or based on statistical analysis of the performance metrics 182 of the action series 158 to choose from. The one or more processors 310 can be configured to rank the plurality of series of tasks 154 (e.g., 160) based at least on the plurality of confidence scores. The one or more processors 310 can be configured to determine, using the one or more ML models 136, a performance metric 182 for the series of tasks 154 (e.g., 160). The one or more processors 310 can be configured to generate a report for the series of tasks 154 (e.g., 160) using the performance metric, the report indicating the ranking of the plurality of series of tasks 154 (e.g., 160) and one or more documents in support for the ranking.

[0193] For example, the system 100 can include one or more processors 310 configured to identify a surgical outcome (e.g., target outcome 198) for a medical procedure performed using a robotic medical system 120. The one or more processors 310 can be configured to determine, based at least on a data stream 170 of the robotic medical system 120, an action indicative of a task 154 of one or more tasks 154 of the medical procedure.

[0194] The one or more processors 310 can be configured to identify one or more machine learning (ML) models 136 trained on a plurality of tasks 154 of a plurality of medical procedures to achieve a plurality of surgical outcomes (e.g., 198) according to performance metrics 182 of the plurality of tasks. The one or more processors 310 can be configured to select, using the one or more ML models 136 from a plurality of configurations 118, a configuration 118 of one or more instruments 112 coupled to the robotic medical system 120 for one or more remaining tasks 154 of the medical procedure according to the surgical outcome (e.g., target outcome 198). The one or more processors 310 can be configured to provide the configuration 118 to operate the robotic medical system 120 for the one or more remaining tasks 154 of the medical procedure.

[0195] The one or more processors 310 can be configured to determine, based at least on the data stream 170, a series of actions (e.g., action series 158) comprising the action 156, the series of actions (e.g., 158) indicative of a series of tasks 154 (e.g., 160) of the medical procedure comprising the task 154. The one or more processors 310 can be configured to identify the task 154 based on the series of actions (e.g., task series 160). For instance, ML model 136 can determine, detect or recognize a task 154 by identifying individual actions 156 in an action series 158 and determining the order or arrangement of the individual actions 156 within the series. The task 154 can be determined based on a unique selection of actions 156 the arrangement or order of such actions 156.

[0196] The one or more processors 310 can be configured to provide or generate a recommendation 192 of the configuration 118 for display on a user interface 190 of a client device. The client device can include any computing device that a user (e.g., surgeon) can use to connect to the DPS 130. The one or more processors 310 can be configured to receive, via the user interface 190, a selection of the configuration 118. The one or more processors 310 can be configured to setup or configure, responsive to the selection, the robotic medical system 120 according to the configuration 118.

[0197] The one or more processors 310 can be configured to identify, based on the surgical outcome 198, the plurality of configurations 118 for the medical instrument 112. The one ormore processors 310 can be configured to determine the performance metrics 182 of the plurality of tasks 154 based on the plurality of configurations 118. The one or more processors 310 can be configured to select, from the plurality of configurations 118, the configuration 118 based at least on a performance metric 182 of the configuration 118 exceeding performance metrics 182 of another configuration 118 of the plurality of configurations 118. For instance, ML model 136 can determine, out of a plurality of configurations 118 for a medical procedure, its phase 152, tasks 154 or action, a single configuration 118 whose performance metrics 182 is the highest out of all different performance metrics 182 for all different configurations 118. In response to determining that the procedure characteristic 162 (e.g., medical condition of a patient) is such that it triggers maximizing performance characteristic for the given portion of medical procedure, the DPS 130 can initiate the new selected configuration 118.

[0198] The one or more processors 310 can be configured to identify, using the one or more ML models 136, the performance metrics 182 for the plurality of medical procedures completed using the robotic medical system 120. The one or more processors 310 can be configured to identify the surgical outcome 198 for the medical procedure based at least on the performance metrics 182. The one or more ML models 136 can include an ML model 136 trained using a plurality of documents (e.g., training data 178) on the plurality of tasks. The plurality of documents can include at least one of a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system 120, or a data on one or more patients that underwent the medical procedure. This can be determined based at least on patient data 164 for one or more patients.

[0199] The one or more processors 310 can be configured to identify, using the one or more ML models 136, the plurality of configuration 118s for the plurality of medical procedures completed using the robotic medical system 120. The one or more processors 310 can be configured to select, from the plurality of configuration 118s, the configuration 118 based at least on the surgical outcome 198 and the performance metrics 182.

[0200] The one or more processors 310 can be configured to detect a phase 152 of the medical procedure based at least on an order of a plurality of tasks 154 detected using the plurality of metrics (e.g., 182) over a time interval of the medical procedure. The time interval can include one or more actions 156, tasks 154 or phases 152. The one or more processors 310 can be configured to generate a plurality of recommendations for a plurality of tasks 154 remaining in the phase 152, the plurality of recommendations according to an order of the plurality of tasks 154 remaining in the phase 152. The one or more processors 310 can be configured to generate a report 194 for an account of associated with a user (e.g., surgeonprofile 144). The report 194 can include one or more citations to one or more documents corresponding to the medical procedure.

[0201] The configuration 118 can include at least one of a selection of a medical instrument 112 to use for the medical procedure, an arrangement of one or more medical instrument 112s on the robotic medical system 120, or a setting of a medical instrument 112. The one or more processors 310 can be configured to generate, using one or more ML models 136, one or more sections of a report including a recommendation 192 for the one or more remaining tasks 154. The one or more sections of the report 194 can include an introductory section and a conclusion generated using a document (e.g., training data 178) selected for the report 194 based on the recommendation 192 from one or more documents on the medical procedure.

[0202] An aspect of the technical solutions can be directed to a system. The system can include the one or more processors 310, coupled with memory 315. The one or more processors 310 can be configured to receive a characteristic (e.g., 162) of a medical procedure to be performed on a patient using a robotic medical system 120. The procedure characteristic 162 can include a feature or attribute prompting an adjustment to a medical procedure to address or satisfy a condition or a characteristic (e.g., medical condition) of the patient (e.g., based on patient data 164).

[0203] The one or more processors 310 can be configured to identify one or more machine learning (ML) models 136 that can be trained on a plurality of characteristics 162 of a plurality of medical procedures. The procedure characteristics 162 can include medical procedures with particular medical issue to address (e.g., perform a particular action 156, task 154 or phase 152 differently so as to it is more suitable to the patient’s condition) the procedure characteristic 162 by matching a surgeon with a suitable metrics signature 146. For instance, ML model 136 can be trained on a plurality of medical procedures with procedure characteristics 162. ML model 136 can identify which performance metrics 182 are most suitable, important or useful in improving the desired target outcome 198 for the surgery. ML model 136 can identify, out of a plurality of surgeon profiles 144, each one having its own metrics signature 146, a particular surgeon profile 144 whose metrics signature 146 include performance metrics 182 that are most suitable for the given procedure characteristic 162.

[0204] The one or more processors 310 can be configured to select, using the characteristic of the medical procedure (e.g., 162) input into the one or more ML models 136, an identifier of a profile of a surgeon (e.g., 144) having a signature of performance metrics 182 (e.g., metrics signature 146) that satisfies a threshold 184. The threshold 184 can include the most closelymatching selection of performance metrics 182 to the selected features of the procedure characteristics 162 to be satisfied during the course of the medical procedure. The one or more processors 310 can be configured to provide the identifier of the selected surgeon profile 144 to cause configuration 118 of the robotic medical system 120 according to the profile of the surgeon. The one or more processors 310 can output the identifier of the surgeon profile 144, or any other information of the surgeon profile 144, to schedule the surgeon associated with the surgeon profile 144 with the medical procedure of the patient with the procedure characteristic 162.

[0205] The one or more processors 310 can be configured to identify a plurality of profiles of surgeons (e.g., surgeon profile 144) trained to perform the medical procedure. The one or more processors 310 can be configured to select, from the plurality of profiles (e.g., 146), the surgeon based at least on availability of the surgeon. The one or more processors 310 can be configured to identify a set of characteristics of the medical procedure comprising the characteristic. The one or more processors 310 can be configured to compare the set of characteristics with a plurality of signatures of performance metrics 182 of a plurality of profiles of surgeons (e.g., surgeon profile 144). The one or more processors 310 can be configured to select the identifier of the profile, based on the comparison. For example, procedure characteristics 162 can include features or parameters corresponding to particular actions 156 or tasks 154 during the course of a medical procedure which can match with particular performance metrics 182 of the particular metrics signatures 146 of a particular surgeon profile 144 more than with performance metrics 182 of other surgeon profiles 144. The matching can be implemented using an ML 136 and utilizing any similarity function, such as a cosine similarity function, which can be implemented in the ML framework 132.

[0206] The threshold 184 can correspond to a maximum performance metric 182 of a plurality of performance metrics 182 of a plurality of profiles of surgeons (e.g., surgeon profile 144). The plurality of profiles 144 can include the profile of the surgeon 144. The one or more processors 310 can be configured to map the plurality of medical procedures performed on a plurality of patients with a plurality of signatures of performance metrics 182 (e.g., metrics signatures 146) which can correspond (e.g., match with, via cosine similarity function) to the particular phases 152, tasks 154 or actions 156 of one or more medical procedures. The one or more processors 310 can be configured to identify the signature performance metric 182 (e.g., metrics signature 146) based at least on the mapping (e.g., similarity search function).

[0207] The plurality of signatures of performance metrics 182 (e.g., plurality of metrics signatures 146) can include data corresponding to the plurality of surgeons (e.g., plurality ofsurgeon profiles 144). The data can include any surgeon data 148, such as any one or more of: a number of surgeries performed by the surgeon, one or more types of surgeries performed by the surgeon, training completed by the surgeon, performance metrics 182 for surgeries performed by the surgeon, performance metrics 182 for a type of a task 154 included in the medical procedure.

[0208] The characteristic of the medical procedure (e.g., procedure characteristic 162) can include at least one of a type of a medical procedure, a medical condition of the patient, a preference for series of tasks 154 (e.g., 160) of a plurality of series of tasks 154 (e.g., 160) to be performed during the medical procedure, or a health risk associated with a performance metric 182 of the plurality of signatures of performance metrics 182 (e.g., metrics signature 146). The one or more processors 310 can be configured to identify a performance metric 182 corresponding to the characteristic of the medical procedure. For instance, an ML model 136 can identify a performance metric 182 corresponding to a particular procedure characteristic 162 that is sensitive with the given patient. The one or more processors 310 can be configured to identify the identifier of the profile (e.g., 144) based at least on a match between the performance metric 182 and one or more performance metrics 182 of the signature of performance metrics (e.g., metrics signature 146) of the profile 144 of the surgeon.

[0209] The one or more processors 310 can be configured to identify one or more tasks 154 associated with the characteristic of the medical procedure. The one or more processors 310 can be configured to identify the identifier of the surgeon profile 144 based at least on a performance metric 182 for a task 154 of the profile of the surgeon (e.g., 144) that can correspond to the one or more tasks 154 associated with the characteristic (e.g., 162). The one or more processors 310 can be configured to identify the identifier of the surgeon based at least on a performance metric 182 of the profile of the surgeon 144 associated with the characteristic 162 of the medical procedure corresponding to a type of the medical procedure.

[0210] The one or more processors 310 can be configured to select a plurality of identifiers of a plurality of profiles of surgeons (e.g., surgeon profile 144). Each profile of the plurality of profiles 144 can have a respective metrics signature 146 having multiple performance metrics 182. The one or more processors 310 can be configured to determine that the profile of the surgeon 144 includes the signature of performance metrics 182 corresponding to the characteristic 162 of the medical procedure. The determination can be made, for example, using one or more ML models 136. The one or more processors 310 can be configured to select the profile responsive to the determination.

[0211] The one or more processors 310 can be configured to identify a plurality ofcharacteristics 162 of the medical procedure comprising the characteristic 162. The one or more processors 310 can be configured to determine that the metrics signature 146 of performance metrics 182 of the profile of the surgeon 144 corresponds to the plurality of characteristics 162 of the medical procedure. The one or more processors 310 can be configured to identify the identifier of the profile 144 based on the determination.

[0212] The one or more processors 310 can be configured to identify a plurality of characteristics of the medical procedure, the plurality of characteristics corresponding to a plurality of performance metrics 182 for a plurality of tasks 154 the medical procedure. The one or more processors 310 can be configured to rank the plurality of characteristics 162 of the medical procedure according to the plurality of performance metrics 182. The one or more processors 310 can be configured to identify the identifier of the profile based at least on the ranked characteristic 162 of the plurality of characteristics 162. The one or more processors 310 can be configured to identify a desired surgical outcome 198 based on the characteristic 162. The one or more processors can select the identifier of the profile 144 based at least on the desired the desired surgical outcome 198.

[0213] An aspect of the technical solutions can be directed to a system. The system can include one or more processors 310, coupled with memory and configured to receive, via a user interface 190 for a robotic medical system 120, a query 140 on a portion of a medical procedure to be performed using the robotic medical system 120. The query 140 can be received during a medical procedure performed using the robotic medical system 120. The one or more processors 310 can be configured to determine, based on a data stream 170 of the medical procedure received from the robotic medical system 120, state information 188 of the medical procedure being performed by the robotic medical system 120.

[0214] The one or more processors 310 can be configured to identify a chatbot (e.g., agent 138) configured to utilize one or more machine learning (ML) models 136 trained on a plurality of performance metrics 182 for a plurality of configuration 118s of medical instrument 112 and a plurality of tasks 154 of a plurality of medical procedures. The one or more processors 310 can be configured to generate, using the state information 188 and the chatbot (e.g., agent 138), a response 142 to the query 140. The one or more processors 310 can be configured to provide, via the user interface 190, the response 142 to the query 140.

[0215] The one or more processors 310 can be configured to receive a plurality of data streams 170 indicative of one or more tasks 154 of the medical procedures completed using the robotic medical system 120. The one or more processors 310 can be configured to determine, based on one or more portions of the plurality of data streams 170 input into the one or moreML models 136, the state information 188.

[0216] The state information 188 can include at least one of: a task 154 of a plurality of tasks 154 of the medical procedure currently performed, one or more tasks 154 of a plurality of tasks 154 of the medical procedure completed prior to a task 154 currently performed, one or more tasks 154 of a plurality of tasks 154 of the medical procedure to be performed following a task 154 currently performed, a phase 152 of the medical procedure performed by the robotic medical system 120, one or more phases 152 of the medical procedure completed, one or more phases 152 of the medical procedure to be performed following a phase 152 currently performed, information on one or more medical instrument 112s used during the medical procedure, a configuration 118 for a medical instrument 112, a medical history of a patient, or a profile of a surgeon (e.g., 144) performing the medical procedure.

[0217] The portion of the medical procedure can include at least one of: a phase 152 of a plurality of phases 152 of the medical procedure, a task 154 of a plurality of tasks 154 of the phase 152 of the medical procedure or an action 156 of a plurality of actions 156 of a task 154 of the medical procedure. The one or more processors 310 can be configured to identify that the query 140 includes a text requesting data on a configuration 118 of a medical instrument 112 of the robotic medical system 120. The one or more processors 310 can be configured to generate, responsive to a portion of the text input into the chatbot (e.g., agent 138), the response 142 including the data on the configuration 118 of the medical instrument 112.

[0218] The one or more processors 310 can be configured to identify that the query 140 includes a text requesting information on a task 154 of the medical procedure performed using the robotic medical system 120. The one or more processors 310 can be configured to generate, responsive to a portion of the text input into the chatbot (e.g., agent 138), the response 142 including the information on the task. The one or more processors 310 can be configured to parse the query 140 into a plurality of portions. The one or more processors 310 can be configured to identify, from the plurality of portions of the query 140, a portion corresponding to a task 154 to be performed using the robotic medical system 120.

[0219] The one or more processors 310 can be configured to input the query 140 into the chatbot (e.g., agent 138), the query 140 requesting information on one or more actions of a task 154 of a plurality of tasks 154 of the medical procedure to be performed using the robotic medical system 120. The one or more processors 310 can be configured to generate, based at least on a performance metric 182 of the one or more actions, the response 142 comprising a recommendation corresponding to the one or more actions to be performed.

[0220] The one or more processors 310 can be configured to identify a performance metric182 of a completed task 154 of a prior medical procedure completed by a user associated with an account corresponding to the query 140. The one or more processors 310 can be configured to generate, based at least on the performance metric 182 of the completed task, the response 142 comprising a recommendation on a task 154 to be performed during a remaining portion of the medical procedure.

[0221] The one or more processors 310 can be configured to receive the query 140 responsive to a codeword detected by a device configured to record sound. The one or more processors 310 can be configured to identify a portion of the query 140 corresponding to a task 154 of the medical procedure. The one or more processors 310 can be configured to generate the response 142 based at least on the portion of the query 140 input into the one or more ML models 136. The one or more processors 310 can be configured to determine the state information 188 comprising a first one or more tasks 154 of the medical procedure completed using the robotic medical system 120. The one or more processors 310 can be configured to generate the response 142 comprising a second one or more tasks 154 of the medical procedure based at least on the first one or more tasks.

[0222] The one or more processors 310 can be configured to identify, based on the data stream 170 input into the one or more ML models 136, one or more tasks 154 of a phase 152 of the medical procedure performed by the robotic medical system 120. The one or more processors 310 can be configured to determine the state information 188 based at least one the phase 152 of the medical procedure. The one or more processors 310 can be configured to generate the response 142 to the query 140 using the one or more tasks 154 of the phase.

[0223] The one or more processors 310 can be configured to identify that the query 140 corresponds to a task 154 of the medical procedure. The one or more processors 310 can be configured to identify, using a portion of the query 140 input into the one or more ML models 136, a document corresponding to the task. The one or more processors 310 can be configured to provide, for display via the user interface 190, the response 142 comprising a content of the document. The one or more processors 310 can be configured to identify that the query 140 corresponds to an action for a task 154 of the medical procedure. The one or more processors 310 can be configured to generate, using a portion of the query 140 input into the one or more ML models 136, the response 142.

[0224] Turning now to FIGS. 4-8, example flow diagrams of a methods 400-700 for providing a performance metric based user guidance for medical procedures performed via robotic medical systems, are illustrated. The methods 400-700, can each be performed by a system having one or more processors configured to perform operations of the system 100 byexecuting computer-readable instructions stored on a memory. For instance, methods 400-700 can be implemented using a non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to implement operations of the methods. The methods 400-700 can be performed, for example, in accordance with any features or techniques discussed in connection with FIGS. 1-3. For instance, the methods 400-700 can be implemented by one or more processors 310 of a computing system 300 executing non-transitory computer-readable instructions stored on a memory (e.g., the memory 315, 320 or 325) and using the functionality of a DPS 130 of system 100.

[0225] FIG. 4 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems identifying task series to utilize for remainder of an ongoing medical procedure. The method 400 can include operations 405-425. At operation 405, the method can identify an action of a task of a medical procedure. At operation 410, the method can identify a performance metric of the task. At operation 415, the method can generate multiple task series. At operation 420, the method can determine if the selected task series is the most suitable to use. At operation 425, the method can perform remaining tasks according to the selected task series.

[0226] At operation 405, the method can identify an action of a task of a medical procedure. The method can include one or more processors coupled with memory identifying an action to perform a task of a plurality of tasks of a medical procedure. The action can be identified using a data stream of a robotic medical system. The action can be identified by a ML model trained on a plurality of actions in a plurality of medical procedures. The action can be determined or detected by the ML model based at least on a sensor reading from one or more of a video data stream, sensor data stream, kinematics stream or events data stream. The action can be identified in view of, or responsive to, detection or recognition of preceding actions (e.g., actions preceding the identified current action), such as when an ML model detects an action series in order to detect the particular task.

[0227] The method can include identifying any action. For instance, the method can identify the action that includes a motion, movement or act of a robotic medical system, or the medical instruments attached to the arms of the robotic medical systems and used for performing actions, tasks and phases of the medical procedure. Identified action can include a motion, a gesture or an act (e.g., grabbing, holding or activating) by a medical instrument. The action can include inserting a surgical instrument into an operating field or a wound, positioning an instrument within an incision, moving a scalpel to dissect a tissue, ligating a vessel, or manipulating an organ or a tissue. The method can include determining a taskresponsive to detecting or identifying the action or a series of actions preceding the action. The method can include identifying the current task and a task preceding the current task based on prior detected actions.

[0228] At operation 410, the method can identify a performance metric of the task. The method can include the one or more processors identifying a performance metric for the task. The method can identify the performance metric for the task based on the action. For instance, the method can identify the performance metric of an action, or of a series of actions that include the current action. The series of actions, along with their arrangement or order of the actions can define or indicate a particular task.

[0229] The method can include one or more ML modes determining a performance metric of an action based on a training of the ML model to compare the current action with prior actions that are associated with performance metrics. The prior actions can include same or similar actions of the same tasks in the same or similar type of medical procedure previously completed. The method can include one or more ML modes determining a performance metric of the task of the action based on a training of the ML model to compare the current task with prior tasks that are associated with their own performance metrics. The prior tasks can include same or similar tasks in the same or similar type of medical procedure previously completed.

[0230] At operation 415, the method can generate multiple task series. The method can generate multiple action series. The method can include the one or more processors generating a plurality of series of tasks of the medical procedure to be performed by the robotic medical system in the medical procedure following the task. The plurality of series of tasks can be generated using the performance metric and the task input into one or more machine learning (ML) models. The one or more ML models can be trained on a plurality tasks of a plurality of medical procedures associated with a plurality of performance metrics. For example, the ML model can determine the current state information (e.g., the current state of the medical procedure). Based on the current state, the ML model can identify one or more series of actions or one or more series of tasks that can be pursued from the current state of the medical procedure to complete the medical procedure.

[0231] The method can include the one or more processors generating the plurality of series of actions or the plurality of series of tasks. The one or more ML models can use a plurality of documents for the plurality of series of tasks to generate the plurality of series of actions or tasks. The one or more ML models can validate the series of tasks using at least an agent for an ML model of the one or more ML models that can be trained on the plurality of documents. For instance, the agent can be trained to utilize additional data (e.g., data streams) to confirmwhether the determination of the task or the action is correct.

[0232] The method can include the one or more processors identifying a plurality of documents. The plurality of documents can be used to train the one or more ML models to generate the multiple task series or multiple action series. The documents can include at least one of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on a plurality of patients; and use the plurality of documents to generate the plurality of series of tasks.

[0233] At operation 420, the method can determine if the selected task series is the most suitable task series to utilize. The method can make this determination using for example a user interface to present to the user a recommendation with a selected task series to use. Responsive to a user selection on the user interface, the one or more processors can determine if the selected task series is the task series to utilize going forward. For example, the one or more processors can determine that the selected task series is the task series to use for the remainder of the task or phase, based on the selection on the user interface in response to the recommendation prompt. For example, the one or more processors can determine that the selected task series is not the task series to use for the remainder of the task or phase, based on the selection on the user interface in response to the recommendation prompt.

[0234] The method can include the one or more processors ranking the plurality of series of tasks or the plurality of series of actions. For example, the method can include ranking the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks. The method can include the one or more processors ranking the plurality of series of tasks based on a performance metric of the plurality of performance metrics. The performance metric can be identified or determined based on data on a recovery of a patient following a prior medical procedure of the plurality of medical procedures in which the series of tasks was applied. The method can include ranking of the plurality of series of tasks based on a performance metric of the plurality of performance metrics determined based on an occurrence of a medical emergency during performance of a prior medical procedure of the plurality of medical procedures.

[0235] The method can identify a plurality of documents for a plurality of medical procedures performed using a plurality of robotic medical systems and determine the ranking for the plurality of series of tasks using the plurality of documents. The method can identify an agent for an ML model trained on a plurality of documents that correspond to the plurality of series of tasks. The method can provide, responsive to an input of a user of the robotic medicalsystem into the agent, a text corresponding to the series of tasks, text generated by the ML model based on a document of the plurality of documents.

[0236] The method can include the one or more processors identifying one or more tasks of the medical procedure using a document of the plurality of documents. The one or more tasks can be identified using the one or more ML models trained on a plurality of documents for the plurality of medical procedures. The method can include mapping the series of tasks with the one or more tasks using the performance metric.

[0237] The method can include the one or more processors generating a plurality of confidence scores for a plurality of mappings between the plurality of series of tasks and the plurality of performance metrics. The method can include ranking the plurality of series of tasks based at least on the plurality of confidence scores. The method can include the one or more processors determining determine, using the one or more ML models, a performance metric for the series of tasks. The method can include generating a report for the series of tasks using the performance metric. The report can indicate the ranking of the plurality of series of tasks and one or more documents in support for the ranking.

[0238] For example, if the determination of whether the selected task series (e.g., or action series) is the most suitable one with which to proceed is in the affirmative, then the flow can move to operation 425. For example, if the determination of whether the selected task series (e.g., or the action series) is the most suitable one with which to proceed is negative, then the operation can move back to act 405 to identify the next action, which can provide new information that can lead to updating of the task series with which to proceed.

[0239] At operation 425, the method can perform remaining tasks according to the selected task series. For example, responsive to a determination at 425 that the selected task series (e.g., or action series) is the series with which to proceed, the one or more processors can take action to perform remaining tasks according to the selected task series (e.g., or action series). For example, the method can include the one or more actions to adjust, set or configure the RMS to make adjustments to proceed in accordance with the selected series. For example, the method can send a notification to the user (e.g., surgeon) via user interface indicating the next action or the next task or providing a recommendation to the user on how to proceed with the next action or the next task.

[0240] The method can include the one or more processors determining, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform a second action of the series of tasks. The method can include the one or more processors identifying a second action of the robotic medical system. The method candetermine, based on the second action, a second performance metric for the task and update the ranking of the plurality of series of tasks based on the second performance metric. The method can include determining, based at least on the updated ranking, a second series of tasks of the plurality of series of tasks to use for a remainder of the medical procedure.

[0241] FIG. 5 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems to identify a most suitable configuration for a remaining portion of the ongoing medical procedure. The method 500 can include operations 505-525. At operation 505, the method can identify a surgical outcome for a medical procedure on a robotic medical system (RMS). At operation 510, the method can determine an action indicative of a task of the medical procedure. At operation 515, the method can utilize ML models to identify configurations for RMS. At operation 520, the method can determine if the selected configuration is most suitable for the surgical outcome. At operation 525, the method can use selected configuration to operate the RMS.

[0242] At operation 505, the method can identify a surgical outcome for a medical procedure on a robotic medical system (RMS). The surgical outcome can include a desired goal or target to accomplish in a surgery. The surgical outcome can be determined based on a patient’s medical history or a procedure characteristic that affects or modifies the medical procedure for the patient in order to accommodate a particular health issue or a health risk of the patient. Surgical outcome can be based on a medical condition of a patient, patient’s age, patients state of health, patients’ allergies, personal or cultural preference, or any other consideration.

[0243] The method can include the one or more processors identifying a surgical outcome for a medical procedure performed using a robotic medical system. The surgical outcome can be identified based on the surgical outcome being entered into the patient’s medical history or a file or based on detected information or data. The method can include the one or more processors identifying, using the one or more ML models, the performance metrics for the plurality of medical procedures completed using the robotic medical system. The method can include the one or more processors identifying the surgical outcome for the medical procedure based at least on the performance metrics detected, measured, received or acquired.

[0244] At operation 510, the method can determine an action indicative of a task of the medical procedure. The method can include the one or more processors determining based at least on a data stream of the robotic medical system, an action indicative of a task of one ormore tasks of the medical procedure. For example, a ML model can be trained to determine one or more actions or tasks. The ML model can determine, detect or identify an action based at least on one or more data from one or more data streams, such as a kinematics data, sensor data, video data, or events data.

[0245] The method can include the one or more processors determining, based at least on the data stream, a series of actions comprising the action. The series of actions can be indicative of one or more tasks. For example, a series of actions (e.g., an arrangement or order of certain actions) can be indicative of a series of tasks of the medical procedure comprising the task, allowing the ML model to identify the task based on the series of actions. The method can include the one or more processors detecting a phase of the medical procedure based at least on an order of a plurality of tasks detected using the plurality of metrics over a time interval of the medical procedure.

[0246] At operation 515, the method can utilize ML models to identify configurations for RMS. The method can include the one or more processors identifying, by the one or more processors, one or more machine learning (ML) models. The one or more ML models can be trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks. For example, an ML model can determine the configuration of the RMS, or one or more medical instruments of the RMS based at least on one or more of events data (e.g., installation or configuration logs), kinematics data or sensor data readings.

[0247] The configuration can include at least one of: a selection of a medical instrument to use for the medical procedure, an arrangement of one or more medical instruments on the robotic medical system, or a setting of a medical instrument. The configuration can include installing one or more instruments into particular manipulator arms of the RMS. The configuration can include setting parameters, constraints or settings to any portion or feature of the RMS or any device within the medical environment.

[0248] The one or more ML models can include an ML model trained using a plurality of documents on the plurality of tasks or a plurality of actions defining or indicating tasks of the plurality of tasks. The plurality of documents can include at least one of a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on one or more patients that underwent the medical procedure.

[0249] The method can include the one or more processors generating, using one or moreML models, one or more sections of a report. The one or more sections can include a recommendation for the one or more remaining tasks. The one or more sections can include an introductory section and a conclusion generated using a document selected for the report based on the recommendation from one or more documents on the medical procedure. The report can cite to specific texts or documentation sections based on which the content of the report is generated.

[0250] At operation 520, the method can determine if the selected configuration is most suitable for the surgical outcome. The method can include the one or more processors selecting a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome. The configuration can be selected from a plurality of configurations using the one or more ML models. For instance, an ML model can be trained to identify which configurations are most suitable for which series of tasks (e.g., based on prior data). For instance, an ML model can be trained to identify which configurations are most suitable for which user (e.g., based on surgeon data and information on prior procedures of the surgeon). The method can include the one or more processors identifying, based on the surgical outcome, the plurality of configurations for the robotic medical instrument for the surgical outcome.

[0251] The method can include the one or more processors determining the performance metrics of the plurality of tasks based on the plurality of configurations. The method can include the one or more processors selecting, from the plurality of configurations, the configuration based at least on a performance metric of the configuration. The performance metric can exceed the performance metrics of another configuration of the plurality of configurations. For instance, the ML model can seek a configuration for which the performance metric for the series of tasks about to be implemented by the RMS is the highest (e.g., greater than all other performance metrics of all other configurations).

[0252] The method can include the one or more processors identifying, using the one or more ML models, the plurality of configurations for the plurality of medical procedures completed using the robotic medical system. The method can include the one or more processors selecting, from the plurality of configurations, the configuration based at least on the surgical outcome and the performance metrics.

[0253] For example, if the determination of whether the selected configuration is suitable for the surgical outcome is in the affirmative, then the flow can move to operation 525. For example, if this same determination is in the negative, then the operation can move back to act510 to identify or determine the next action, which can provide new information that can lead to updating or adjusting the configuration.

[0254] At operation 525, the method can use selected configuration to operate the RMS.13. The method can include the one or more providing the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure. The method can include the RMS implementing the selected configuration to reconfigure the medical instruments or any portion of the RMS to complete the upcoming portions of the medical procedure with the selected configuration.

[0255] The method can include the one or more processors providing a recommendation of the configuration for display on a user interface of a client device. The method can include the one or more processors receiving, via the user interface, a selection of the configuration; and configure, responsive to the selection, the robotic medical system according to the configuration.

[0256] The method can include the one or more processors generating a plurality of recommendations for a plurality of tasks remaining in the phase. The plurality of recommendations can be according to an order of the plurality of tasks remaining in the phase. The method can include the one or more processors generating a report for an account of associated with a user. The report can include one or more citations to one or more documents corresponding to the medical procedure.

[0257] FIG. 6 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems to identify a most suitable surgeon for particular medical procedure. The method 600 can include operations 605-620. At operation 605, the method can receive a medical procedure characteristic. At operation 610, the method can identify a surgeon profile with matching performance metrics signature. At operation 615, the method determines if the selected surgeon profile is suitable the medical procedure. At operation 620, the method can provide surgeon profile identifier.

[0258] At operation 605, the method can receive a medical procedure characteristic. The method can include the one or more processors receiving a characteristic of a medical procedure to be performed on a patient using a robotic medical system. The characteristic of the medical procedure can include at least one of: a type of a medical procedure, a medical condition of the patient, a preference for series of tasks of a plurality of series of tasks to be performed during the medical procedure, or a health risk associated with a performance metricof the plurality of signatures of performance metrics.

[0259] The plurality of signatures of performance metrics can include data corresponding to the plurality of surgeons. The data can include at least one of: a number of surgeries performed by the surgeon, one or more types of surgeries performed by the surgeon, training completed by the surgeon, performance metrics for surgeries performed by the surgeon, performance metrics for a type of a task included in the medical procedure.

[0260] The method can include the one or more processors identifying a plurality of characteristics of the medical procedure comprising the characteristic. The method can include the one or more processors identifying a plurality of characteristics of the medical procedure. The plurality of characteristics can correspond to a plurality of performance metrics for a plurality of tasks the medical procedure. The characteristics of the medical procedure can correspond to or more closely match with a first action than with a section action, or with a first task rather than a second task. For example, a characteristic can include a particular style of incisions and suturing, which may be relevant or preferred in some medical procedures of patients for which bleeding can carry an increased risk. In such instances, surgeon profiles with highest performance metrics for actions or tasks associated with reduced bleeding can be beneficial to accommodate the procedure characteristic.

[0261] At operation 610, the method can identify a surgeon profile with matching performance metrics signature. The method can include the one or more processors identifying one or more machine learning (ML) models. The one or more ML models can be trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems.

[0262] The method can include the one or more processors identifying a plurality of profiles of surgeons trained to perform the medical procedure. The method can include the one or more processors selecting, from the plurality of profiles, the surgeon based at least on availability of the surgeon. The method can include the one or more processors identifying a set of characteristics of the medical procedure comprising the characteristic. For instance, the method can include identifying a surgeon profile for whom performance metrics of the tasks that are preferred or suitable to address the characteristic of the medical procedure are higher than the performance metrics of in other surgical profiles.

[0263] At operation 615, the method determines if the selected surgeon profile is suitable the medical procedure. The method can include the one or more processors selecting, using thecharacteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon. The profile of the surgeon can include a signature of performance metrics that satisfies a threshold. The threshold can correspond to a maximum performance metric of a plurality of performance metrics of a plurality of profiles of surgeons. The plurality of profiles can include the profile of the surgeon.

[0264] The method can include the one or more processors comparing the set of characteristics with a plurality of signatures of performance metrics of a plurality of profiles of surgeons. The method can include the one or more processors selecting the identifier of the profile, based on the comparison. The method can include the one or more processors mapping the plurality of medical procedures performed on a plurality of patients with a plurality of signatures of performance metrics corresponding to the plurality of medical procedures. The method can include the one or more processors identifying the signature performance metric based at least on the mapping.

[0265] The method can include the one or more processors identifying a performance metric corresponding to the characteristic of the medical procedure. The method can include the one or more processors identifying the identifier of the profile based at least on a match between the performance metric and one or more performance metrics of the signature of performance metrics of the profile of the surgeon. The method can include the one or more processors identifying one or more tasks associated with the characteristic of the medical procedure. The method can include the one or more processors identifying the identifier of the surgeon based at least on a performance metric for a task of the profile of the surgeon corresponding to the one or more tasks associated with the characteristic.

[0266] The method can include the one or more processors identifying the identifier of the surgeon based at least on a performance metric of the profile of the surgeon associated with the characteristic of the medical procedure corresponding to a type of the medical procedure. The method can include the one or more processors selecting a plurality of identifiers of a plurality of profiles of surgeons. Each profile of the plurality of profiles can have a respective signature of performance metrics. The method can include the one or more processors determining that the profile of the surgeon includes the signature of performance metrics corresponding to the characteristic of the medical procedure. The method can include the one or more processors selecting the profile responsive to the determination.

[0267] The method can include the one or more processors determining that the signature of performance metrics of the profile of the surgeon corresponds to the plurality of characteristics of the medical procedure. The method can include the one or more processorsidentify the identifier of the profile based on the determination.

[0268] The method can include the one or more processors ranking the plurality of characteristics of the medical procedure according to the plurality of performance metrics. The method can include the one or more processors identifying the identifier of the profile based at least on the ranked characteristic of the plurality of characteristics. The method can include the one or more processors identifying a desired surgical outcome based on the characteristic and selecting the identifier of the profile based at least on the desired the desired surgical outcome.

[0269] For example, if the determination at act 615 is in the affirmative, then the flow can move to operation 620. For example, if this same determination is in the negative, then the operation can move back to act 610 to identify or determine the next surgeon profile to consider and repeat the actions.

[0270] At operation 620, the method can provide surgeon profile identifier. The method can include the one or more processors providing the identifier to cause configuration of the robotic medical system according to the profile of the surgeon. For example, the method can include identifying a configuration of the surgeon from the profile of the surgeon and configure the RSM, responsive to the identification. For example, the method can send a message to a server of the medical institution to request the surgeon to be scheduled for the medical procedure, responsive to the match between the signature of the performance metrics (e.g., metrics signature) of the surgeon and the procedure characteristics of the patient’s procedure.

[0271] FIG. 7 illustrates an example flow diagram of a performance based user guidance platform for robotic medical systems providing response to user queries using a chatbot. The method 700 can include operations 705-725. At operation 705, the method can receive a chatbot query on a portion of a medical procedure to be perform. At operation 710, the method can determine state information of the medical procedure. At operation 715, the method can utilize the query and the state information to generate a response. At operation 720, the method can determine if the selected response is suitable based on performance metrics. At operation 725, the method can provide the response via the chatbot.

[0272] At operation 705, the method can receive a chatbot query on a portion of a medical procedure to be perform. The method can include the one or more processors receiving, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system. The portion of the medical procedure can include at least one of: a phase of a plurality of phases of the medical procedure, a task of a plurality of tasks of thephase of the medical procedure or an action of a plurality of actions of a task of the medical procedure.

[0273] The method can include the one or more processors identifying that the query includes a text requesting data on a configuration of a medical instrument of the robotic medical system. The method can include the one or more processors identify that the query includes a text requesting information on a task of the medical procedure performed using the robotic medical system. The method can include the one or more processors parsing the query into a plurality of portions. The method can include the one or more processors identifying, from the plurality of portions of the query, a portion corresponding to a task to be performed using the robotic medical system.

[0274] The method can include the one or more processors inputting the query into the chatbot. The query can request the information on one or more actions of a task of a plurality of tasks of the medical procedure to be performed using the robotic medical system. The method can include the one or more processors receiving the query responsive to a codeword detected by a device configured to record sound. The method can include the one or more processors identifying a portion of the query corresponding to a task of the medical procedure. The method can include the one or more processors identifying that the query corresponds to an action for a task of the medical procedure.

[0275] At operation 710, the method can determine state information of the medical procedure. The method can include the one or more processors determining, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system. The method can include the one or more processors receiving a plurality of data streams indicative of one or more tasks of the medical procedures completed using the robotic medical system. The method can include the one or more processors determine, based on one or more portions of the plurality of data streams input into the one or more ML models, the state information.

[0276] The state information can include at least one of: a task of a plurality of tasks of the medical procedure currently performed, one or more tasks of a plurality of tasks of the medical procedure completed prior to a task currently performed, one or more tasks of a plurality of tasks of the medical procedure to be performed following a task currently performed, a phase of the medical procedure performed by the robotic medical system, one or more phases of the medical procedure completed, one or more phases of the medical procedure to be performed following a phase currently performed, information on one or more medical instruments used during the medical procedure, a configuration for a medical instrument, a medical history of apatient, or a profile of a surgeon performing the medical procedure.

[0277] The method can include the one or more processors determining the state information comprising a first one or more tasks of the medical procedure completed using the robotic medical system. The method can include the one or more processors identifying, based on the data stream input into the one or more ML models, one or more tasks of a phase of the medical procedure performed by the robotic medical system. The method can include the one or more processors determining the state information based at least one the phase of the medical procedure.

[0278] At operation 715, the method can utilize the query and the state information to generate a response. The method can include the one or more processors identifying, by the one or more processors, a chatbot configured to utilize one or more machine learning (ML) models. The one or more ML models can be trained on a plurality of performance metrics for a plurality of configurations of medical instruments and a plurality of tasks of a plurality of medical procedures.

[0279] The method can include the one or more processors identifying a performance metric of a completed task of a prior medical procedure completed by a user associated with an account corresponding to the query. The method can include the one or more processors generating, responsive to a portion of the text input into the chatbot, the response including the data on the configuration of the medical instrument. The method can include the one or more processors generating, responsive to a portion of the text input into the chatbot, the response including the information on the task.

[0280] The method can include the one or more processors generating, based at least on a performance metric of the one or more actions, the response comprising a recommendation corresponding to the one or more actions to be performed. The method can include the one or more processors generating, based at least on the performance metric of the completed task, the response comprising a recommendation on a task to be performed during a remaining portion of the medical procedure.

[0281] The method can include the one or more processors generating the response based at least on the portion of the query input into the one or more ML models. The method can include the one or more processors generating the response comprising a second one or more tasks of the medical procedure based at least on the first one or more tasks. The method can include the one or more processors generating the response to the query using the one or more tasks of the phase. The method can include the one or more processors generating, using a portion of the query input into the one or more ML models.

[0282] At operation 720, the method can determine if the selected response is suitable based on performance metrics. The selection can be made, based on, for example a determination that the performance metrics of a current one or more actions or tasks being performed are greater than the performance metrics of any alternative tasks or actions. The selection can be made based on, for example, determining that the confidence score in the response is below a threshold. For example, the response can be generated by the ML model. The ML model can generate a confidence score for the response in view of the reliability of the data identified in the documentation which the ML model utilized to generate the response.

[0283] The method can include the one or more processors generating, by the one or more processors, using the state information and the chatbot, a response to the query. The method can include the one or more processors identifying that the query corresponds to a task of the medical procedure. The method can include the one or more processors identifying, using a portion of the query input into the one or more ML models, a document corresponding to the task; and provide, for display via the user interface, the response comprising a content of the document.

[0284] For example, if the determination at act 720 is in the affirmative, then the flow can move to operation 725. For example, if this same determination is in the negative, then the operation can move back to act 715 to generate another response to consider and repeat the actions at 720.

[0285] At operation 725, the method can provide the response via the chatbot. The method can include providing, by the one or more processors, via the user interface, the response to the query. The response can be provided as a prompt on a user interface. The response can include textual data, graphical data, multimedia (e.g., video or image data). The response can include information requested in the query. The response can be provided via an email, a message, a text message or a voice mail. The response can be provided on an HMD of the user.

[0286] Although an example computing system has been described in FIG. 7, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0287] FIGS. 8-17 illustrate various examples of screenshots of a graphical user interface 190 for implementing the functionalities of the performance metrics based user guidance platform. Graphical user interface 190 can be implemented for an application executed via aDPS 130 and providing various functionalities of an action identifier 133, series selector 135, outcome identifier 137, profile selector 139, chatbot selector 141, performance function 1980, state function 186, ranking function 166 and configuration function 196, any of which can be implemented using any ML framework 132 functionalities, any surgeon profiles 144 or procedure data 150.

[0288] FIG. 8 presents an example 800 of a user interface 190 of an application (e.g., insights engine) that can executed on the DPS 130. The user interface can display various alerts or indications, such as recommendations 192. A recommendation 192 can prompt the user to gain insight about relationships in surgical data (e.g., surgeon data 148 or metric signature 146), including for example, performance metrics 182 about the economy of motion matter for salient bleeding.

[0289] FIG. 9 presents an example 900 of a user interface 190 displaying various procedure data 150 and surgeon data 148. For example, user interface 190 can display various links, prompts or other information about various procedures of different surgeons (e.g., information from surgeon profiles 144). User interface 190 can display different data on tasks 154 of different medical procedures performed by specific surgeons.

[0290] FIG. 10 presents an example 1000 of a user interface 190 displaying different study tags 1005 for different procedure data 150 and surgeon data 148. The study tags 1005 of the user interface 190 can include data pertaining to different procedures, tasks 154, phases 152 or surgeons (e.g., surgeon data 148) which the user can upload as datasets for further study or analysis. Study tags 1005 can include performance metrics 182 for different procedures or tests that the application of the user interface 190 can ingest, process and provide for access for future users.

[0291] FIG. 11 presents an example 1100 of a user interface 190 displaying various study tags 1005 for different procedure data 150 and surgeon data 148. The study tags 1005 of the user interface 190 can include data that can be uploaded and combined with kinematics data 172 and performance metrics 182 for specific procedures. The user interface 190 can provide the prompts for uploading files with the study tags for different studies.

[0292] FIG. 12 presents an example 1200 of a user interface 190 displaying various different procedure data 150 along with study tags 1005. Study tag details can include the dates when the tags are created or updated, various procedure identifiers and information about robotic volume (e.g., robotic movement data).

[0293] FIGs. 13-16 present examples 1300-1600 of the user interface 190 in which target outcomes 198 (e.g., test hypotheses) are provided for user selection, allowing the user to select the target outcomes 198 for which to identify action series 158 or task series 160. Examples 1300, 1400, 1500 and 1600 can provide different performance metrics 182 (e.g., OPIs and weighted scores) that can apply to any sets of tasks 154 and actions 156.

[0294] FIG. 17 presents an example 1700 of a user interface 190 displaying various performance metrics 182 corresponding to actions 156, such as forceps usage speed. Performance metrics 182 can be displayed in various visual formats indicative of mean or median scores and lower and upper boundaries of values, visually presenting total average instrument speed for any one or more tasks 154 or actions 156.

[0295] FIGS. 18-19 present examples 1800-1900 of the user interface 190 providing an agent 138 (e.g., a chatbot) that can be used by a user to enter queries 140 to receive responses 142. As shown in example 1800 of FIG. 18, a user can enter a query 140 into an agent 138. The query 140 can correspond to any user request, such as requests related to reports, result explanations, any questions related to performance metrics 182 (e.g., objective performance indicators or OPIs), operation of RMS 120, or references. As shown in example 1900 of FIG.19, the user interface 190 can provide the agent 138 with a displayed query 140, which can correspond to performance metrics 182 (e.g., OPIs). Agent 138 can provide a response 142 for the query 140, which can describe various OPIs and provide references in support of the response.

[0296] FIGS. 20-23 provide examples 2000-2300 of the user interface 190 with different agents 138 providing various queries and responses 142. As shown in example 2000 of FIG.20, a user interface 190 can provide an agent 138 with a prompt window for entering queries 140 with respect to a surgeon profile 144 providing various surgeon data 148. The agent 138 can provide a set of prepared proposed queries 140 for user selection in relation to a particular type of a study or a procedure topic (e.g., cholecystectomy study) from the surgeon data 148.

[0297] FIG. 21 shows an example 2100 of a user interface 190 in which an agent 138 is used to enter queries 140 in reference to data of a surgeon profile 144. Surgeon profile 144 can include surgeon data 148 with respect to various procedures of the surgeon associated with the surgeon profile 148. Agent 138 can be used by a user to enter queries 140 about the surgeon data 148 (e.g., history and OPIs of various surgeon procedures from the surgeon data 148).

[0298] FIG. 22 shows an example 2200 of a user interface 190 in which an agent 138 is used to provide a response 142 to a query 140. The query 140 can be related to a robotic medical procedure and the response 142 can provide various insights responsive to the query 140. Shown in example 2300 of FIG. 23, user interface 190 can provide an agent 138 that can present a response 142 that includes or incorporates various performance metrics 182 and procedure data 150, including for example graphical representations of the performance metrics 182 or performance data 150.

[0299] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable or physically interacting components or wirelessly interactable or wirelessly interacting components or logically interacting or logically interactable components.

[0300] With respect to the use of plural or singular terms herein, those having skill in the art can translate from the plural to the singular or from the singular to the plural as is appropriate to the context or application. The various singular / plural permutations can be expressly set forth herein for sake of clarity.

[0301] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0302] Although the figures and description can illustrate a specific order of method steps, the order of such steps can differ from what is depicted and described, unless specified differently above. Also, two or more steps can be performed concurrently or with partial concurrence, unless specified differently above. Such variation can depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0303] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0304] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and Ctogether, or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0305] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0306] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: identify, using a data stream of a robotic medical system, an action of the robotic medical system to perform a task of a plurality of tasks of a medical procedure performed by the robotic medical system; identify, based on the action, a performance metric for the task; generate, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed by the robotic medical system in the medical procedure following the task; rank the plurality of series of tasks based on the plurality of performance metrics corresponding to the plurality of series of tasks; and determine, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform one or more actions of the series of tasks.

2. The system of claim 1, comprising the one or more processors to: identify a plurality of documents for a plurality of medical procedures performed using a plurality of robotic medical systems; and determine the ranking for the plurality of series of tasks using the plurality of documents.

3. The system of claim 1, comprising the one or more processors to: identify an agent for an ML model trained on a plurality of documents corresponding to the plurality of series of tasks; and provide, responsive to an input of a user of the robotic medical system into the agent, a text corresponding to the series of tasks, text generated by the ML model based on a document of the plurality of documents.

4. The system of claim 1, comprising the one or more processors togenerate the plurality of series using a plurality of documents for the plurality of series of tasks; and validate the series of tasks using at least an agent for an ML model of the one or more ML models trained on the plurality of documents.

5. The system of claim 1, comprising the one or more processors to: identify a second action of the robotic medical system; determine, based on the second action, a second performance metric for the task; and update the ranking of the plurality of series of tasks based on the second performance metric.

6. The system of claim 5, comprising the one or more processors to determine, based at least on the updated ranking, a second series of tasks of the plurality of series of tasks to use for a remainder of the medical procedure.

7. The system of claim 1, comprising the one or more processors to: rank the plurality of series of tasks based on a performance metric of the plurality of performance metrics determined based on data on a recovery of a patient following a prior medical procedure of the plurality of medical procedures in which the series of tasks was applied.

8. The system of claim 1, comprising the one or more processors to: rank the plurality of series of tasks based on a performance metric of the plurality of performance metrics determined based on an occurrence of a medical emergency during performance of a prior medical procedure of the plurality of medical procedures.

9. The system of claim 1, comprising the one or more processors to: identify a plurality of documents including at least one of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on a plurality of patients; and use the plurality of documents to generate the plurality of series of tasks.

10. The system of claim 1, comprising the one or more processors to: identify, using the one or more ML models trained on a plurality of documents for theplurality of medical procedures, one or more tasks of the medical procedure using a document of the plurality of documents; and map the series of tasks with the one or more tasks using the performance metric.

11. The system of claim 1, comprising the one or more processors to: generate a plurality of confidence scores for a plurality of mappings between the plurality of series of tasks and the plurality of performance metrics; and rank the plurality of series of tasks based at least on the plurality of confidence scores.

12. The system of claim 1, comprising the one or more processors to: determine, using the one or more ML models, a performance metric for the series of tasks; and generate a report for the series of tasks using the performance metric, the report indicating the ranking of the plurality of series of tasks and one or more documents in support for the ranking.

13. A method, comprising: identifying, by one or more processors coupled with memory, using a data stream of a robotic medical system, an action to perform a task of a plurality of tasks of a medical procedure; identifying, by the one or more processors, based on the action, a performance metric for the task; generating, by the one or more processors, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed by the robotic medical system in the medical procedure following the task; ranking, by the one or more processors, the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks; and determining, by the one or more processors, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform a second action of the series of tasks.

14. The method of claim 13, comprising:identifying, by the one or more processors, a plurality of documents for a plurality of medical procedures performed using a plurality of robotic medical systems; and determining, by the one or more processors, the ranking for the plurality of series of tasks using the plurality of documents.

15. The method of claim 13, comprising: identifying, by the one or more processors, an agent for an ML model trained on a plurality of documents corresponding to the plurality of series of tasks; and providing, by the one or more processors, responsive to an input of a user of the robotic medical system into the agent, a text corresponding to the series of tasks, text generated by the ML model based on a document of the plurality of documents.

16. The method of claim 13, comprising: generating, by the one or more processors, the plurality of series using a plurality of documents for the plurality of series of tasks; and validating, by the one or more processors, the series of tasks using at least an agent for an ML model of the one or more ML models trained on the plurality of documents.

17. The method of claim 13, comprising: identifying, by the one or more processors, a second action of the robotic medical system; determining, by the one or more processors, based on the second action, a second performance metric for the task; updating, by the one or more processors, the ranking of the plurality of series of tasks based on the second performance metric; and determining, by the one or more processors, based at least on the updated ranking, a second series of tasks of the plurality of series of tasks to use for a remainder of the medical procedure.

18. The method of claim 13, comprising: identifying, by the one or more processors, a plurality of documents including at least one of: a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on a plurality of patients; and using, by the one or more processors, the plurality of documents to generate theplurality of series of tasks.

19. The method of claim 13, comprising: identifying, by the one or more processors, using the one or more ML models trained on a plurality of documents for the plurality of medical procedures, one or more tasks of the medical procedure using a document of the plurality of documents; and mapping, by the one or more processors, the series of tasks with the one or more tasks using the performance metric.

20. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: identify, using a data stream of a robotic medical system, an action of the robotic medical system to perform a task of a medical procedure; identify, based on the action, a performance metric for the task; generate, using the performance metric and the task input into one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures associated with a plurality of performance metrics, a plurality of series of tasks of the medical procedure to be performed following the task; rank the plurality of series of tasks based on a plurality of performance metrics corresponding to the plurality of series of tasks; and determine, based on the ranking, a series of tasks of the plurality of series of tasks to cause the robotic medical system to perform a second action of the series of tasks.

21. A system, comprising: one or more processors, coupled with memory, to: identify a surgical outcome for a medical procedure performed using a robotic medical system; determine, based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure; identify one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks; select, using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome; and provide the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

22. The system of claim 21, comprising the one or more processors to: determine, based at least on the data stream, a series of actions comprising the action, the series of actions indicative of a series of tasks of the medical procedure comprising the task; and identify the task based on the series of actions.

23. The system of claim 21, comprising the one or more processors to: provide a recommendation of the configuration for display on a user interface of a client device; receive, via the user interface, a selection of the configuration; and configure, responsive to the selection, the robotic medical system according to the configuration.

24. The system of claim 21, comprising the one or more processors to: identify, based on the surgical outcome, the plurality of configurations for the robotic medical instrument for the surgical outcome; determine the performance metrics of the plurality of tasks based on the plurality ofconfigurations; and select, from the plurality of configurations, the configuration based at least on a performance metric of the configuration exceeding performance metrics of another configuration of the plurality of configurations.

25. The system of claim 21, comprising the one or more processors to: identify, using the one or more ML models, the performance metrics for the plurality of medical procedures completed using the robotic medical system; and identify the surgical outcome for the medical procedure based at least on the performance metrics.

26. The system of claim 21, where in the one or more ML models includes an ML model trained using a plurality of documents on the plurality of tasks, the plurality of documents including at least one of a publication on one or more medical procedures, one or more hospital records, one or more data of the robotic medical system, or a data on one or more patients that underwent the medical procedure.

27. The system of claim 21, comprising the one or more processors to: identify, using the one or more ML models, the plurality of configurations for the plurality of medical procedures completed using the robotic medical system; and select, from the plurality of configurations, the configuration based at least on the surgical outcome and the performance metrics.

28. The system of claim 21, comprising the one or more processors to detect a phase of the medical procedure based at least on an order of a plurality of tasks detected using the plurality of metrics over a time interval of the medical procedure.

29. The system of claim 21, comprising the one or more processors to generate a plurality of recommendations for a plurality of tasks remaining in the phase, the plurality of recommendations according to an order of the plurality of tasks remaining in the phase.

30. The system of claim 21, comprising the one or more processors to generate a report for an account of associated with a user, the report including one or more citations to one or more documents corresponding to the medical procedure.

31. The system of claim 21, wherein the configuration includes at least one of: a selection of a medical instrument to use for the medical procedure, an arrangement of one or more medical instruments on the robotic medical system, or a setting of the medical instrument.

32. The system of claim 21, comprising the one or more processors to: generate, using one or more ML models, one or more sections of a report including a recommendation for the one or more remaining tasks, the one or more sections including an introductory section and a conclusion generated using a document selected for the report based on the recommendation from one or more documents on the medical procedure.

33. A method, comprising: identifying, by one or more processors coupled with memory, a surgical outcome for a medical procedure performed using a robotic medical system; determining, by the one or more processors, based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure; identifying, by the one or more processors, one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks; selecting, by the one or more processors, using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome; and providing, by the one or more processors, the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

34. The method of claim 33, comprising: determining, by the one or more processors, based at least on the data stream, a series of actions comprising the action, the series of actions indicative of a series of tasks of the medical procedure comprising the task; and identifying, by the one or more processors, the task based on the series of actions.

35. The method of claim 33, comprising:providing, by the one or more processors, a recommendation of the configuration for display on a user interface of a client device; receiving, by the one or more processors, via the user interface, a selection of the configuration; and configuring, by the one or more processors, responsive to the selection, the robotic medical system according to the configuration.

36. The method of claim 33, comprising: identifying, by the one or more processors, based on the surgical outcome, the plurality of configurations for the robotic medical instrument for the surgical outcome; determining, by the one or more processors, the performance metrics of the plurality of tasks based on the plurality of configurations; and selecting, by the one or more processors, from the plurality of configurations, the configuration based at least on a performance metric of the configuration exceeding performance metrics of another configuration of the plurality of configurations.

37. The method of claim 33, comprising: identifying, by the one or more processors, using the one or more ML models, the performance metrics for the plurality of medical procedures completed using the robotic medical system; and identifying, by the one or more processors, the surgical outcome for the medical procedure based at least on the performance metrics.

38. The method of claim 33, comprising: identifying, by the one or more processors, using the one or more ML models, the plurality of configurations for the plurality of medical procedures completed using the robotic medical system; and selecting, by the one or more processors, from the plurality of configurations, the configuration based at least on the surgical outcome and the performance metrics.

39. The method of claim 33, comprising: detecting, by the one or more processors, a phase of the medical procedure based at least on an order of a plurality of tasks detected using the plurality of metrics over a time interval of the medical procedure; andgenerating, by the one or more processors, a plurality of recommendations for a plurality of tasks remaining in the phase, the plurality of recommendations according to an order of the plurality of tasks remaining in the phase.

40. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: identify a surgical outcome for a medical procedure performed using a robotic medical system; determine, based at least on a data stream of the robotic medical system, an action indicative of a task of one or more tasks of the medical procedure; identify one or more machine learning (ML) models trained on a plurality of tasks of a plurality of medical procedures to achieve a plurality of surgical outcomes according to performance metrics of the plurality of tasks; select, using the one or more ML models from a plurality of configurations, a configuration of one or more instruments coupled to the robotic medical system for one or more remaining tasks of the medical procedure according to the surgical outcome; and provide the configuration to operate the robotic medical system for the one or more remaining tasks of the medical procedure.

41. A system, comprising: one or more processors, coupled with memory, to: receive a characteristic of a medical procedure to be performed on a patient using a robotic medical system; identify one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems; select, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold; and provide the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

42. The system of claim 41, comprising the one or more processors to: identify a plurality of profiles of surgeons trained to perform the medical procedure; and select, from the plurality of profiles, the surgeon based at least on availability of the surgeon.

43. The system of claim 41, comprising the one or more processors to: identify a set of characteristics of the medical procedure comprising the characteristic; compare the set of characteristics with a plurality of signatures of performance metrics of a plurality of profiles of surgeons; and select the identifier of the profile, based on the comparison.

44. The system of claim 41, wherein the threshold corresponds to a maximum performance metric of a plurality of performance metrics of a plurality of profiles of surgeons, the plurality of profiles comprising the profile of the surgeon.

45. The system of claim 41, comprising the one or more processors to: map the plurality of medical procedures performed on the plurality of patients with a plurality of signatures of performance metrics corresponding to the plurality of medical procedures; andidentify the signature of performance metrics based at least on the mapping.

46. The system of claim 41, wherein the plurality of signatures of performance metrics includes data corresponding to the plurality of surgeons, the data including at least one of: a number of surgeries performed by the surgeon, one or more types of surgeries performed by the surgeon, training completed by the surgeon, performance metrics for surgeries performed by the surgeon, performance metrics for a type of a task included in the medical procedure.

47. The system of claim 41, wherein the characteristic of the medical procedure includes at least one of: a type of a medical procedure, a medical condition of the patient, a preference for series of tasks of a plurality of series of tasks to be performed during the medical procedure, or a health risk associated with a performance metric of the plurality of signatures of performance metrics.

48. The system of claim 41, comprising the one or more processors to: identify a performance metric corresponding to the characteristic of the medical procedure; and identify the identifier of the profile based at least on a match between the performance metric and one or more performance metrics of the signature of performance metrics of the profile of the surgeon.

49. The system of claim 41, comprising the one or more processors to: identify one or more tasks associated with the characteristic of the medical procedure; and identify the identifier of the surgeon based at least on a performance metric for a task of the profile of the surgeon corresponding to the one or more tasks associated with the characteristic.

50. The system of claim 41, comprising the one or more processors to: identify the identifier of the surgeon based at least on a performance metric of the profile of the surgeon associated with the characteristic of the medical procedure corresponding to a type of the medical procedure.

51. The system of claim 41, comprising the one or more processors to:select a plurality of identifiers of a plurality of profiles of surgeons, each profile of the plurality of profiles having a respective signature of performance metrics; determine that the profile of the surgeon includes the signature of performance metrics corresponding to the characteristic of the medical procedure; and select the profile responsive to the determination.

52. The system of claim 41, comprising the one or more processors to: identify a second plurality of characteristics of the medical procedure comprising the characteristic; determine that the signature of performance metrics of the profile of the surgeon corresponds to the second plurality of characteristics of the medical procedure; and identify the identifier of the profile based on the determination.

53. The system of claim 41, comprising the one or more processors to: identify a plurality of characteristics of the medical procedure, the plurality of characteristics corresponding to a plurality of performance metrics for a plurality of tasks the medical procedure; rank the plurality of characteristics of the medical procedure according to the plurality of performance metrics; and identify the identifier of the profile based at least on the ranked characteristic of the plurality of characteristics.

54. The system of claim 41, comprising the one or more processors to: identify a desired surgical outcome based on the characteristic; and select the identifier of the profile based at least on the desired the desired surgical outcome.

55. A method, comprising: receiving, by one or more processors coupled with memory, a characteristic of a medical procedure to be performed on a patient using a robotic medical system; identifying, by the one or more processors, one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeonsusing one or more robotic medical systems; selecting, by the one or more processors, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold; and providing, by the one or more processors, the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

56. The method of claim 55, comprising: identifying, by the one or more processors, a plurality of profiles of surgeons trained to perform the medical procedure; and selecting, by the one or more processors, from the plurality of profiles, the surgeon based at least on availability of the surgeon, wherein the threshold corresponding to a maximum performance metric of a plurality of performance metrics of the plurality of profiles of surgeons, the plurality of profiles comprising the profile of the surgeon.

57. The method of claim 55, comprising: identifying, by the one or more processors, a set of characteristics of the medical procedure comprising the characteristic; comparing, by the one or more processors, the set of characteristics with a plurality of signatures of performance metrics of a plurality of profiles of surgeons; and selecting, by the one or more processors, the identifier of the profile, based on the comparison.

58. The method of claim 55, comprising mapping, by the one or more processors, the plurality of medical procedures performed on a plurality of patients with a plurality of signatures of performance metrics corresponding to the plurality of medical procedures; and identifying, by the one or more processors, the signature of performance metrics based at least on the mapping.

59. The method of claim 55, comprising: identifying, by the one or more processors, a performance metric corresponding to the characteristic of the medical procedure; and identifying, by the one or more processors, the identifier of the profile based at least ona match between the performance metric and one or more performance metrics of the signature of performance metrics of the profile of the surgeon.

60. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: receive a characteristic of a medical procedure to be performed on a patient using a robotic medical system; identify one or more machine learning (ML) models trained on a plurality of characteristics of a plurality of medical procedures performed on a plurality of patients and a plurality of signatures of performance metrics corresponding to a plurality of medical procedures performed by a plurality of surgeons using one or more robotic medical systems; select, using the characteristic of the medical procedure input into the one or more ML models, an identifier of a profile of a surgeon having a signature of performance metrics that satisfies a threshold; and provide the identifier to cause configuration of the robotic medical system according to the profile of the surgeon.

61. A system, comprising: one or more processors, coupled with memory, to: receive, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system; determine, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system; identify a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality of performance metrics for a plurality of configurations of medical instrument and a plurality of tasks of a plurality of medical procedures; generate, using the state information and the chatbot, a response to the query; and provide, via the user interface, the response to the query.

62. The system of claim 61, comprising the one or more processors to: receive a plurality of data streams indicative of one or more tasks of the medical procedures completed using the robotic medical system; and determine, based on one or more portions of the plurality of data streams input into the one or more ML models, the state information.

63. The system of claim 61, wherein the state information includes at least one of: a task of a plurality of tasks of the medical procedure currently performed, one or more tasks of a plurality of tasks of the medical procedure completed prior to a task currently performed, one or more tasks of a plurality of tasks of the medical procedure to be performed following a task currently performed, a phase of the medical procedure performed by the robotic medical system, one or more phases of the medical procedure completed, one or more phases of the medical procedure to be performed following a phase currently performed, information on one or more medical instruments used during the medical procedure, a configuration for a medical instrument, a medical history of a patient, or a profile of a surgeon performing the medical procedure.

64. The system of claim 61, wherein the portion of the medical procedure includes at least one of: a phase of a plurality of phases of the medical procedure, a task of a plurality of tasks of the phase of the medical procedure or an action of a plurality of actions of a task of the medicalprocedure.

65. The system of claim 61, comprising the one or more processors to: identify that the query includes a text requesting data on a configuration of a medical instrument of the robotic medical system; and generate, responsive to a portion of the text input into the chatbot, the response including the data on the configuration of the medical instrument.

66. The system of claim 61, comprising the one or more processors to: identify that the query includes a text requesting information on a task of the medical procedure performed using the robotic medical system; and generate, responsive to a portion of the text input into the chatbot, the response including the information on the task.

67. The system of claim 61, comprising the one or more processors to: parse the query into a plurality of portions; and identify, from the plurality of portions of the query, a portion corresponding to a task to be performed using the robotic medical system.

68. The system of claim 61, comprising the one or more processors to: input the query into the chatbot, the query requesting information on one or more actions of a task of a plurality of tasks of the medical procedure to be performed using the robotic medical system; and generate, based at least on a performance metric of the one or more actions, the response comprising a recommendation corresponding to the one or more actions to be performed.

69. The system of claim 61, comprising the one or more processors to: identify a performance metric of a completed task of a prior medical procedure completed by a user associated with an account corresponding to the query; and generate, based at least on the performance metric of the completed task, the response comprising a recommendation on a task to be performed during a remaining portion of the medical procedure.

70. The system of claim 61, comprising the one or more processors to: receive the query responsive to a codeword detected by a device configured to record sound; identify a portion of the query corresponding to a task of the medical procedure; and generate the response based at least on the portion of the query input into the one or more ML models.

71. The system of claim 61, comprising the one or more processors to: determine the state information comprising a first one or more tasks of the medical procedure completed using the robotic medical system; and generate the response comprising a second one or more tasks of the medical procedure based at least on the first one or more tasks.

72. The system of claim 61, comprising the one or more processors to: identify, based on the data stream input into the one or more ML models, one or more tasks of a phase of the medical procedure performed by the robotic medical system; determine the state information based at least one the phase of the medical procedure; and generate the response to the query using the one or more tasks of the phase.

73. The system of claim 61, comprising the one or more processors to: identify that the query corresponds to a task of the medical procedure; identify, using a portion of the query input into the one or more ML models, a document corresponding to the task; and provide, for display via the user interface, the response comprising a content of the document.

74. The system of claim 61, comprising the one or more processors to: identify that the query corresponds to an action for a task of the medical procedure; and generate, using a portion of the query input into the one or more ML models, the response.

75. A method, comprising: receiving, by one or more processors coupled with memory, via a user interfacefor a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system; determining, by the one or more processors, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system; identifying, by the one or more processors, a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality of performance metrics for a plurality of configurations of medical instruments and a plurality of tasks of a plurality of medical procedures; generating, by the one or more processors, using the state information and the chatbot, a response to the query; and providing, by the one or more processors, via the user interface, the response to the query.

76. The method of claim 75, comprising: receiving, by the one or more processors, a plurality of data streams indicative of one or more tasks of the medical procedures completed using the robotic medical system; and determining, by the one or more processors, based on one or more portions of the plurality of data streams input into the one or more ML models, the state information.

77. The method of claim 75, comprising: identifying, by the one or more processors, that the query includes a text requesting data on a configuration of a medical instrument of the robotic medical system; and generating, by the one or more processors, responsive to a portion of the text input into the chatbot, the response including the data on the configuration of the medical instrument.

78. The method of claim 75, comprising: identifying, by the one or more processors, that the query includes a text requesting information on a task of the medical procedure performed using the robotic medical system; and generating, by the one or more processors, responsive to a portion of the text input into the chatbot, the response including the information on the task.

79. The method of claim 75, comprising: inputting, by the one or more processors, the query into the chatbot, the query requesting information on one or more actions of a task of a plurality of tasks of the medical procedure to be performed using the robotic medical system; and generating, by the one or more processors, based at least on a performance metric of the one or more actions, the response comprising a recommendation corresponding to the one or more actions to be performed.

80. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: receive, via a user interface for a robotic medical system during a medical procedure performed using the robotic medical system, a query on a portion of a medical procedure to be performed using the robotic medical system; determine, based on a data stream of the medical procedure received from the robotic medical system, state information of the medical procedure being performed by the robotic medical system; identify, using the state information, a chatbot configured to utilize one or more machine learning (ML) models trained on a plurality of performance metrics for a plurality of configurations of medical instruments and a plurality of tasks of a plurality of medical procedures; generate, using the state information and the chatbot, a response to the query; and provide, via the user interface, the response to the query.

Images