Predictive surgical event system

Abstract

Systems and methods for providing an intelligent event system during a medical session in a medical environment include determining, based on at least one data stream, a non¬ operative activity during the medical session in the medical environment, the at least one data stream captured using at least one sensor located in the medical environment, predicting, using a machine learning model and based at least on the non-operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session, and performing action responsive to predicting the time remaining to the milestone.

Description

PREDICTIVE SURGICAL EVENT SYSTEMCLAIM OF PRIORITY

[0001] This application claims priority to U.S. Patent Application No. 63 / 580,630, filed September 5, 2023, entitled “PREDICTIVE SURGICAL EVENT SYSTEM,” the full disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Surgical procedures may involve capturing imagery such as video feeds from a variety of viewpoints. For example, in some instances, at least part of the surgical procedure may be performed with a computer-assisted robotic medical system. A medical tool, such as an imaging device like an endoscope, may be used in the robotic medical system to provide imagery. Image capture devices such as cameras may also be located at various viewpoints in the medical environment to capture and provide imagery of various aspects of the surgical procedure. The captured imagery from the surgical procedure may be processed in various ways.SUMMARY

[0003] The present disclosure is directed to a system and method for a predictive surgical event system. The smart event system of the present disclosure may automatically take actions when certain conditions are satisfied. For example, the event system of the present disclosure may send alerts or communications to notify the surgeon, cleaning crew, and other medical professionals associated with a medical session when they are needed in a medical environment (e.g., a surgical facility, an operating room, and so on). The event system may take other actions such as displaying data related to the medical session, sending instructions to perform a task, etc. To take such actions, the event system gathers data from the medical environment to accurately determine, using an activity recognition model, a current state or activity in the medical session. Using the recognized activity and metadata associated with the medical session, the event system of the present disclosure predicts in real-time, using a predictive model, time remaining to a milestone (e.g., end of the surgery, end of the medical session, etc.). Based on the predicted time remaining to the milestone, the event system of the present disclosure determines when a particular medical professional (e.g., surgeon, cleaning crew, etc.) is needed in the medical environment and sends alerts accordingly and / or takes other actions.

[0004] At least one aspect is directed to a system having one or more processors coupled with memory to determine, based on at least one data stream, a non-operative activity during a medical session in a medical environment, at least one data stream captured using at least one sensor located in the medical environment, predict, using a machine learning model and based at least on the non-operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session, and take action responsive to predicting the time remaining to the milestone.

[0005] At least one aspect is directed to a non-transitory computer-readable media having computer-readable instructions stored thereon. The computer-readable instructions when executed by one or more processors cause the one or more processors to determine, based on at least one data stream, a non-operative activity during a medical session in a medical environment, the at least one data stream captured using at least one sensor located in the medical environment, predict, using a machine learning model and based at least on the non- operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session, and take action responsive to predicting the time remaining to the milestone.

[0006] At least one aspect is directed to a method including determining, based on at least one data stream, a non-operative activity during a medical session in a medical environment, the at least one data stream captured using at least one sensor located in the medical environment, predicting, using a machine learning model and based at least on the non- operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session, and taking action responsive to predicting the time remaining to the milestone.

[0007] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. The foregoing information and the following detailed description and drawings include illustrative examples and should not be considered as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings:

[0009] FIG. 1 is an example predictive surgical system.

[0010] FIG. 2 is an example graph plotting a time remaining to a milestone in the predictive surgical event system of FIG. 1.

[0011] FIG. 3 is an example flowchart outlining operations for predicting time remaining to a milestone and taking actions responsive to the prediction.

[0012] FIG. 4 is a block diagram showing an example surgical environment.

[0013] FIG. 5 is an example computer system.DETAILED DESCRIPTION

[0014] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems of detection and removal of surgically irrelevant data streams. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways.

[0015] In many applications of surgical data science, lack of effective planning and communication between entities may lead to increased turnover times and loss of efficiency and revenue. During a surgical session (also referred to herein as a medical session), a surgical procedure or surgery may be performed in a medical environment (e.g., a surgical environment, an operating room, etc.). In addition to the surgery, a medical session may include activities performed before the surgery and after the surgery. Specifically, before the surgery begins and / or after the surgery is complete, the operating room and the patient may need to be prepared. Various medical professionals may be involved with various aspects of the medical session. For example, a surgeon may perform the surgery. The surgeon may be assisted by various nurses, medical assistants, etc. The operating room may need to be sterilized, cleaned, restocked with equipment and supplies, etc. before and after the surgery. The patient may need to be cleaned, draped, undraped, intubated, etc. The surgeon may not be required in the operating room until the surgery is about to begin. The cleaning crew may not be needed in the operating room during the surgery. Certain other medical professionals may be needed for some aspects of the surgical session but not the others. However, the surgeon, the cleaning crew, and the other medical professionals may not know what is currently happening in the operating room and when they may be needed in the operating room. They may come early tothe operating room, requiring them to wait and waste time and resources. They may come late to the operating room, delaying the rest of the surgical session.

[0016] Thus, an event system is needed that automatically takes actions based on current and predicted happenings within the operating room. The present disclosure provides a smart event system to notify the surgeon, cleaning crew, and other medical professionals associated with the medical session when they are needed in the operating room, using artificial intelligence powered models that predict time remaining to surgical milestones in real-time. The present disclosure also provides a smart event system to display various types of information to keep stakeholders apprised of happenings within the operating room. The present disclosure further provides a smart event system that may take other types of actions such as sending instructions to perform tasks, etc.

[0017] The event system of the present disclosure may notify team members (whether surgeon, cleaning crew, or other medical professionals) of their next assignment with a sufficient lead time, thereby reducing idle times, improving efficiency, and minimizing lost revenue. For example, using the event system of the present disclosure, the cleaning crew may be notified to stand by to clean the operating room a few minutes (e.g., 10-15 minutes) before the medical session ends or a few minutes (e.g., 10-15 minutes) after the post-surgery activities end. The surgeon may be paged a few minutes (e.g., 10-15 minutes) before the patient is ready for surgery. Other medical professionals may similarly be alerted a few minutes before they are needed in the operating room. To provide such alerts, the event system of the present disclosure is powered by predictive models that may accurately estimate time remaining to certain desired milestones during the medical sessions.

[0018] More particularly, the event system of the present disclosure may gather data from the operating room to accurately determine, using an activity recognition model, a current state or activity in the medical session. The activities may be identified based on data collected from data sources outside of the body of the patient, for example, located or mounted in the operating room. The activity recognition model may determine the start time of the activity and how long the activity lasts. Using activity recognition and metadata associated with the medical session, the smart event system of the present disclosure may predict in real-time, using a predictive model, a time remaining to a milestone (e.g., end of the surgery, end of the medical session, etc.). Based on the predicted time remaining to the milestone, the event system of the present disclosure may determine accurately when a particular medical professional (e.g., surgeon, cleaning crew, etc.) is needed in the operating room and send alerts accordingly. The predicted time remaining to the milestone may be used for performing otheractions as well. Accurate predictions of time remaining to milestone enable accurate and timely notifications to medical professionals, minimizing turnover time, reducing loss of revenue, and increasing efficiency.

[0019] By tracking the current activity in the operating room, the event system of the present disclosure may accurately keep track of the current state of the medical session, including when certain activities complete sooner than expected or take longer to complete than expected or when unexpected events occur, and adjust its predictions. The event system of the present disclosure may also predict adverse events and preparatory tasks. In addition to sending alerts to the medical professionals, the smart event system of the present disclosure may cause medical session related details to be displayed inside and / or outside the operating room. For example, the event system may indicate which activity is currently being performed, when that activity started, and when that activity is expected to complete. The event system may cause other aspects of the medical session to be displayed as well. Further, in some situations, based on the accurate tracking of the current activity and the prediction of the time remaining to the milestone, the event system of the present disclosure may send instructions to perform one or more tasks. For example, the event system may send an instruction to schedule a follow up appointment for the patient after the end of the medical session. The event system may send a notification to the recovery room personnel to prepare the room for receiving the patient after the patient is wheeled out of the operating room. The event system may facilitate communication between different departments in a hospital, clinic, medical facility, medical environment, etc. The event system may take other actions as appropriate.

[0020] Although the present disclosure is discussed in the context of a surgical procedure, in some embodiments, the present disclosure may be applicable to other types of medical sessions or environments or activities, as well as non-medical activities where a smart event system is desired.

[0021] Referring now to FIG. 1, an example system 100 is shown. The system 100 may be associated with a medical environment 105. The medical environment 105 may include, or be associated with, a medical environment. A medical environment may be an operating room in which a surgical procedure or surgery, whether, invasive, non-invasive, in-patient, or outpatient, is performed on a patient. In other embodiments, the system 100 may be associated with other types of medical sessions or activities, or non-medical environments. The system 100 may include a sensor system 110 having a sensor 115A and a sensor 115B (collectively referred to as the sensors 115). Although two of the sensors 115 are shown in FIG. 1, in some embodiments, greater than or fewer than two sensors may be provided within the medicalenvironment 105. Each of the sensors 115 may be a time-of-flight (ToF) sensor or other type of sensor configured to capture data from a viewpoint within the medical environment 105. Thus, each of the sensors 115 may be positioned, mounted, or otherwise located based on content that is desired to be captured from a particular viewpoint. Each of the sensors 115 may collect data of multiple modalities (e.g., intensity, depth, etc.).

[0022] For example, in some embodiments, at least one of the sensors 115 may be positioned to capture data from an area where a patient is located within the medical environment 105. In some embodiments, at least one of the sensors 115 may be positioned to capture data from an area where one or more medical professionals are located within the medical environment 105. In some other embodiments, at least one of the sensors 115 may be configured to capture data from other designated areas within the medical environment 105. The sensors 115 may include any of a variety of cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of image capture devices that may be suitable for the purposes described herein. The sensors 115 may be configured to capture data that is external to the body of a patient (e.g., not captured from within the body of the patient).

[0023] The data captured by the sensors 115 may include still images, video images, vector images, bitmap images, two dimensional images, three dimensional images, infrared images, fluorescent images, and any other types or modalities of images, any other type of analog or digital data, or combinations thereof. In some embodiments, one or more of the sensors 115 may be configured to capture other parameters such as sound, motion, pressure, temperature, etc. within the medical environment 105 as well. The sensors 115 may capture data at any suitable capture rate and / or frequency. For example, in some embodiments, the sensors 115 may continuously capture data twenty-four hours a day. In other embodiments, the sensors 115 may capture data periodically during a predetermined period. Other settings, such as zoom settings, etc. of each of the sensors 115 may vary as desired to capture suitable data from a particular viewpoint. In some embodiments, one or more of the sensors 115 may have fixed locations, positions, and / or orientations. In other embodiments, one or more of the sensors 115 may be portable, or otherwise configured to change orientation or telescope in various directions. In some embodiments, one or more of the sensors 115 may be part of a multi-sensorarchitecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, pressure, etc.).

[0024] The system 100 may also include a medical tool 120 that is configured to be installed within a robotic medical system 125. The medical tool 120 may be any type and form of tool used for surgery. In some embodiments, the medical tool 120 may be an endoscope or endoscopic camera for visualizing organs, tissues, etc. within a body of the patient (e.g., inserted into the body of the patient). In some embodiments, the medical tool 120 may include other or additional types of therapeutic and / or diagnostic medical imaging implements. The medical tool 120 may be configured to be installed in the robotic medical system 125. For example, the medical tool may be connected to a manipulator arm of the robotic medical system 125 to assist with a surgery (e.g., be inserted into the body of the patient and capture images from within the body of the patient). Installing the medical tool 120 in the robotic medical system 125 may include or be associated with or be part of connecting the medical tool to the robotic medical system. The medical tool 120 may be separate from the sensor system 110. For example, the medical tool 120 may be configured to capture operative activities, while the sensor system 110 may be configured to capture non-operative activities. The images captured by the medical tool 120 may be viewed through a visualization tool (not shown in FIG. 1) connected to a port (e.g., input port) of the robotic medical system 125.

[0025] The robotic medical system 125 may be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient with the assistance of one or more robotic components and / or medical tools. As described in more detail below, the robotic medical system 125 may include one or more manipulator arms that perform one or more computer-assisted medical tasks. The medical tool 120 may be installed on a manipulator arm of the robotic medical system 125 to perform a surgical task. The robotic medical system 125 may also include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, in some embodiments, the visualization tool may be connected to the robotic medical system 125 to receive the images from the medical tool 120 when the medical tool is installed in the robotic medical system (e.g., on a manipulator arm of the robotic medical system).

[0026] The data captured by the sensor system 110 may be transmitted as one or more data streams to a data stream processing system 130 of a data processing system 135 associated with the medical environment 105. In some embodiments, data captured by the medical tool 120 and / or the robotic medical system 125 may also be transmitted as one or more data streams to the data stream processing system 130. In some embodiments, the data stream processingsystem 130 may also receive manual inputs 180 entered by a user. Each data stream may be considered any sequence of digital encoded data or analog data from a data source such as the sensors 115. The data stream processing system 130 may be configured to receive the one or more data streams through an Application Programming Interface (API) associated with the data processing system 135 and / or the sensor system 110. In other embodiments, the data stream processing system 130 may be configured to receive the one or more data streams via other suitable mechanisms. The data stream(s) may be transmitted to the data processing system 135 in real-time or substantial real-time (e.g., as the image is captured, the image is transmitted).

[0027] The data stream processing system 130 may cause the data stream(s) to be displayed on one or more displays associated with the data processing system 135 and / or one or more displays within the medical environment 105. For example, the data stream processing system 130 may cause the data stream(s) to be displayed on a display 140 of the data processing system 135. The data stream processing system 130 may be configured to prepare the data stream(s) for further processing such as segmenting the data stream(s) into frames or portions for further analysis. The data stream processing system 130 may normalize the data in the data stream(s), compress the data stream(s), encrypt the data stream(s), convert the data stream(s) into a different format, and / or perform other or additional operations on the data stream(s). The data stream processing system 130 may send the data stream(s) for storing (e.g., in the cloud) and take other actions to manage the data stream(s). The output from the data stream processing system 130 may be input into an activity recognition model 145. For ease of explanation, the discussion below states that the data stream(s) are input into the activity recognition model 145. The data stream(s) that are input into the activity recognition model 145 may be raw data stream(s) that are passed through by the data stream processing system 130 or directly to the activity recognition model or pre-processed data stream(s) on which one or more processing operations (e.g., segmentation) have been performed.

[0028] The activity recognition model 145 may be a machine learning neural network or other computer-vision model trained to receive the data steam(s) as an input and identify one or more activities from the data stream(s) during a medical session. The activity recognition model 145 may be another type of a machine learning / artificial intelligence-based model that is configured to identify activities, and particularly non-operative activities, from the data stream(s). In some embodiments, the activity recognition model 145 may receive data stream(s) in real-time. Using the real-time images, the activity recognition model 145 may determine one or more activities in real-time or substantial real-time during a medical session.For example, the activity recognition model 145 may determine an activity currently being performed in a medical environment during a medical session. A medical session is also referred to herein as a “case.” Each case may include a surgical procedure (e.g., a surgery), pre-surgery (e.g., preparation for the surgery in the medical environment), post-surgery (e.g., clean up after the surgery in the medical environment), and turnover (e.g., between two surgeries in the medical environment) activities. The activity recognition model 145 may be configured to identify these activities during the case in real-time.

[0029] Example activities may include turnover activities that start when a patient is rolled out of the medical environment and a next patient is rolled into the medical environment (e.g., turnover for case 1 happens before case 1, turnover for case 2 happens before case 2, etc.), setup activities that start after cleaning of the medical environment and end before sterile preparation, sterile preparation activities that start when a first sterile item is opened and end when a scrub technician is done preparing for the case, robot draping activities that start when a first manipulator arm of the robotic medical system 125 is draped and end when the entire robotic medical system is draped, patient-in activities that start when a gurney with the patient is rolled into the medical environment and end when the gurney without the patient is rolled out, and patient preparation activities that start when medical professional start preparing the patient for the surgical procedure and end before the roll up or first incision is made on the patient. The patient preparation activities may also include intubation activities that start when the medical professionals tilt the patient’s head back and end when the stylet is removed, patient draping activities that start when a drape is placed in the patient and end when the patient is fully draped (e.g., when a tent is formed at or near the head of the patient), and first cut activities that start when a medical professional reaches for a scalpel and end when the medical professional has made the first incision on the patient. The patient preparation activities may include other or additional activities.

[0030] The activities may further include port placement activities that start when the first incision is made on the patient and end when all ports (e.g., catheters, etc.) are placed and ready for rollup, room preparation activities that start when a first object such as a light, monitor, or other item is moved out of the way of the path of the robotic medical system 125 and end when the path of the robotic medical system is clear because all the objects are moved away from the path, rollup activities that start when a circulator begins to push the medical robotic system 125 and end when the circulator stops pushing the medical robotic system, and docking activities that start after the rollup activities and end when all manipulator arms of the robotic medical system 125 are docked and the medical professionals (e.g., the surgeon) leave the patient’sside. The activities may also include surgery activities that start after the docking activities and end before undocking of the manipulator arms of the robotic medical system 125, open surgery activities that start after the first incision on the patient and end when the first incision begins to close, and lap surgery activities that start after port placement activities and end before the patient close activities (e.g., ports are closed (e.g., catheters are removed, etc.)).

[0031] The activities may additionally include undocking activities that start after the surgical procedure is complete and end when the robotic medical system 125 starts to roll away from the patient side, rollback activities that start when the circulator starts to pull the robotic medical system 125 back and end when the circulator stops pulling the robotic medical system back, patient close activities that start when the medical professionals (e.g., surgeon) start to look at the incision during the rollback activities and end before patient undraping activities which start when the patient is begun to undrape and end when the patient is fully undraped, robot undraping activities that start when the robotic medical system 125 begins undraping and end when the robotic medical system is fully undraped, and patient-out activities that start when a gurney without a patient is rolled into the operating room and ends when the gurney with the patient is rolled out of the operating room. The activities may also include cleaning activities that start when the operating room begins to be cleaned (e.g., tablecloths are removed, etc.) and end when the operating room is fully cleaned and mopped, and idle activities that start when no activity is happening in the operating room and end when an activity is happening in the operating room.

[0032] The activity recognition model 145 may be configured to identify other or additional activities during a medical session. In some embodiments, certain activities (e.g., parent activities) may include other activities (e.g., child activities) in which case the parent activities may be considered a “phase.”

[0033] The activities described above may be considered as non-operative activities in that these activities are performed in the operating room or medical environment before, after, or independent or separate from the operation or surgical procedure itself. The non-operative activities may be performed by medical professional (non-surgeons) other than the surgeon performing the surgical procedure. The non-operative activities may be contrasted with operative activities that are performed on, within or associated with the body of the patient to conduct a medical procedure such as surgery by a surgeon. The non-operative activities may be identified based on data collected from data sources outside of the body of the patient, for example, located or mounted in the operating room. The operative activities may be associated with data collected from a medical tool operating on the body of a patient (e.g., throughendoscopic videos). For each activity, the activity recognition model 145 may estimate a start time of the activity and an end time of the activity. The activity recognition model 145 may also estimate other parameters associated with each activity. The activity recognition model 145 may recognize multiple activities happening at the same time. Therefore, the activity recognition model 145 may be considered a multi-label model. The determination of the activities by the activity recognition model 145 is discussed in more detail in PCT Publication No. WO 2022 / 104129, filed on November 12, 2021, the entirety of which is incorporated by reference herein.

[0034] The output of the activity recognition model 145 is input into an event system 150 that predicts time remaining to milestone 150. Although shown as separate machine learning models, in some embodiments, the activity recognition model 145 and the event system 150 may be configured as a single machine learning model that may perform the functions of both models. Further, although shown as part of the data processing system 135, in some embodiments, the activity recognition model 145 and / or the event system 150 may be located in a remote location (e.g., a cloud location) and accessible to the data processing system. The event system 150 may be configured to predict a time remaining to a particular milestone during a medical session. The milestone may be an end of a particular medical session. End of a medical session may be considered a point in time when the patient is rolled out of the medical environment. The milestone may be an end of a surgical procedure during a medical session. End of a surgical procedure may be considered a point in time when the surgical procedure on the patient is completed (e.g., after the incision is closed, after the patient is undraped, etc.). The milestone may be the start of a case (e.g., when the surgeon or another medical professional is needed in the medical environment). The start of a case may be associated with the start of the medical session or start of a specific activity during the medical session. The start of a case may be associated with the start of the surgical procedure (e.g., when the patient is rolled into the medical environment). The milestone may be when a medical instrument or equipment is needed in the medical environment that is generally not needed for the type of surgical procedure being performed. In addition to predicting milestones, in some embodiments, the event system 150 may predict an adverse event (e.g., bleeding), or when intra-operative test(s) or image(s) (e.g., histology tests for prostatectomy) are needed. The adverse event may be associated with the patient on which the surgical procedure is performed (e.g., excess bleeding). The adverse event may be associated with happenings in the medical environment around the patient (e.g., spillage of materials). Themilestone or adverse event may correlate with the inputs of the event system 150 in the surgical procedure or medical session.

[0035] The event system 150 may be a deep learning neural network or machine learning model including linear or non-linear models that receives the activity from the activity recognition model 145 and metadata from a metadata system 155 as inputs and predicts the time remaining to a milestone as an output. When implemented as a linear model, the event system 150 may use inputs to predict unknown values (e.g., the time remaining to milestones). In some embodiments, the event system 150 may be structured as a linear regression model to find a “best fit” relationship between the inputs (e.g., activities from the activity recognition model 145, metadata) to the event system 150 and the desired milestone. In some embodiments, the linear model may be used to determine the impact of an input on the prediction when holding other inputs (e.g., covariates) fixed. The linear models may use linear regression coefficients to describe a mathematical relationship between the various inputs. Regression analysis may be used to identify how changes in certain inputs may impact the prediction from the event system 150. The event system 150 may also implement regularization techniques such as lasso regression to sparsify inputs that are considered not as important for the prediction as others. For example, lower weights may be assigned to inputs that are less important given other inputs. The event system 150 may use other or additional linear model implementation techniques or mechanisms. The event system 150 may be another type of a machine learning or artificial learning-based model.

[0036] In some embodiments, the event system 150 may be mathematically represented as y = f(x) where y may be the time remaining to an event or milestone that is to be predicted, f(x) may be a linear or non-linear model making the prediction, and x may be a vector of inputs input into the linear or non-linear model as discussed herein.

[0037] The event system 150 may predict how much time remains from the current point in time until the milestone is reached. For example, if the milestone is end of a medical session, the event system 150 may predict the amount of time remaining for the medical session to end. For example, the event system 150 may predict that the medical session is expected to end in 15 minutes. Similarly, when predicting the end of a surgical procedure, the event system 150 may predict the amount of time remaining to the end of the surgical procedure. The event system 150 may predict the time remaining to a milestone based on the activity from the activity recognition model 145 and the metadata from the metadata system 155.

[0038] The metadata system 155 may be configured to store a variety of metadata associated with one or more medical sessions being executed in the medical environment 105.For each medical session, the metadata system 155 may store metadata relating to patient data, medical personnel data, medical environment data, or medical session data. Patient data may include data related to the identity (e.g., name) of the patient, patient age, patient demographics, patient current medical conditions such as patient health, body mass index, endoscopic video images captured from the patient, other types of images (e.g., X-ray, CT, MRI images) captured from the patient, patient’s medical history, patient’s family history, and other patient related data. Medical personnel data may include data related to surgeon skills (e.g., number of the surgical procedure that the surgeon has performed), surgeon identity (e.g., surgeon name), and other surgeon related data. Medical personnel data may also include data associated with the other medical professionals involved with a medical session. For example, the medical personnel data may include data related to the medical team size (e.g., how many medical professionals are involved in the surgery and care of the patient), medical team experience level, experience level of the anesthesiologist, experience level of the circulator and other medical professional, presence of a second scrub-technologist, and other types of information related to the medical personnel. Medical environment data may include data related to the operating room number or name where the surgical procedure is to be performed, equipment available in the medical environment, size of the medical environment, room layout of the medical environment, and any other data associated with the medical environment.Medical session data may include data related to the surgical procedure type, complexity of the surgical procedure, date (e.g., a year, month, day, weekday) when the surgical procedure is being performed, medical session number, surgical procedure code, type and identifying details (e.g., name, any identification number, etc.) of activities (e.g., non-operative activities) associated with the surgical procedure, expected time for completing each activity, event logs from the robotic medical system 125, time elapsed, motion of personnel, non-operative activities performed by one or more personnels, and any other data related to the medical session.

[0039] Certain types of metadata may have a greater impact on a time remaining to a milestone (e.g., a milestone prediction) than other types of metadata in a particular medical session. For example, in some embodiments, as experience of the surgeon increases, the expected duration of a particular activity may decrease. In other words, surgeons who have performed a particular type of surgical procedure many times may be able to perform the surgical procedure faster than surgeons who have not performed that surgical procedure as often. Thus, depending on the experience of the surgeon, the time remaining to the milestone may vary. In some embodiments, certain types of medical professionals (e.g.,anesthesiologists) may require a greater time with the patient than other medical professionals, which may impact the time remaining to the milestone. Complex surgical procedures may take longer than non-complex procedures. Thus, in some embodiments, the type of surgical procedure (e.g., the ones requiring anesthesia) and certain activities during the surgical procedure may have a greater impact on the time remaining predictions than other types of metadata. As another example, patients who present complex situations (e.g., high body mass index) may require longer times for certain activities (e.g., patient preparation) of the case than patients with non-complex situations. Therefore, certain types of metadata may have a greater impact on the time remaining to milestone prediction than other types of metadata. In addition, a particular metadata that has a greater impact in one medical session may not have as much of an impact in another medical session. For example, in some embodiments, surgeon experience may have a greater impact on a complex case than a non-complex case.

[0040] In some embodiments, certain types of metadata that have a greater impact may be assigned higher weights than the types of metadata that have a lower impact on the time remaining to milestone prediction by the event system 150. The event system 150 may adjust the weights of the metadata automatically to select optimal inputs (e.g., metadata inputs) from the input vector (e.g., the set of metadata inputs). In some embodiments, regularization may be used to sparsify the inputs and the set the weights to zero for the inputs that do not contribute to the prediction (e.g., due to those inputs having low or no impact on the prediction). In this way, the event system 150 may provide information on the important inputs for each predicted milestone.

[0041] Further, certain types of metadata may be used to convey or determine a particular type of information. For example, in some embodiments, surgeon name may be used to determine the experience level of the surgeon in performing the particular surgical procedure (e.g., surgeon name may be used to determine the surgeon’s area of expertise and access data on how many cases relating to the surgical procedure the surgeon has performed). However, if the surgeon experience data is available (e.g., number of surgical procedures performed), then the surgeon’s name may not be needed. Thus, in some embodiments, certain types of metadata may replace other types of metadata. Therefore, in each medical session, the types of metadata used for the time remaining to milestone prediction may be different. In some embodiments, the relative importance of metadata may be indicated by way of different weights that may be assigned to each metadata by the event system 150 during the prediction. For example, when both surgeon experience and surgeon name are available, surgeon experience may be assigneda higher weight than the surgeon’s name. It is to be understood that the examples used herein are only for purposes of explanation and are not intended to be limiting in any way.

[0042] In some embodiments, the metadata system 155 may be configured to receive certain types of metadata and updates to the metadata in real-time. For example, in some embodiments, the patient’s body mass index may be measured before or during the surgical procedure and input into the metadata system. In some embodiments, certain types of metadata may be pre-stored in the metadata system 155 when that information is available. For example, the year, month, day, type of surgical procedure being performed in a particular medical session, other medical session details (e.g., case number) may be known beforehand and may be programmed within the metadata system 155 in advance. The metadata may also change in real-time as the medical session progresses.

[0043] The event system 150 may receive additional types of inputs in some embodiments. For example, in some embodiments, the event system 150 may also receive time since the start of the medical session, time since the start of the surgical procedure, past non-operative activities, past and current operative activities (e.g., event logs from the robotic medical system 125, endoscopic videos, etc.), adverse events, auditory cues (e.g., from medical professionals in the medical environment), manually entered inputs (e.g., the manual inputs 180), sterility breaches, number of door openings in the medical environment, and any other inputs that may be suitable for making the predictions discussed herein. In some embodiments, the event system 150 may proactively prompt a user (e.g., a medical professional) to provide an updated or additional input in certain circumstances. For example, in some embodiments, each activity may be determined to take a threshold period of time to complete. If the activity takes longer to complete than the threshold period of time (e.g., takes greater than a predetermined time period past the threshold period of time), the event system 150 may prompt the user to provide an input. The input may indicate, for example, how much longer is the activity expected to take to complete, whether an adverse event has occurred, and any other suitable information that the event system 150 may need for an accurate and reliable prediction. The event system 150 may use the input to determine whether an update to the prediction of the time remaining to a milestone is needed and update the prediction(s) if needed. In some embodiments, the event system 150 may prompt the user to provide an input when a prediction of a time remaining to a particular milestone has been updated greater than a threshold number of times. In some embodiments, the event system 150 may prompt the user to provide an input when an initial prediction of a time remaining to a particular milestone has been updated to be pushed out longer than a predetermined threshold. For example, if an initial prediction has beenpushed out by greater than an “X” amount of time, the event system 150 may prompt the user to provide an input. In other embodiments, the event system 150 may be configured to prompt the user for an input in other circumstances.

[0044] Thus, based on the activity received from the activity recognition model 144, the metadata received from the metadata system 155, and any other received input(s), the event system 150 may predict a time remaining to one or more milestones during the medical session. The activity input received from the activity recognition model 145 may include a current activity being performed, a start time of the activity, and an actual end time of the activity. Using the activity input and the metadata, the event system 150 may determine the other activities that may still need to be performed until the milestone is achieved, expected times of completion of those activities, and based on the above determinations, predict how much time may be remaining to the milestone. The event system 150 may update the predicted time remaining to the milestone dynamically as changes to the activity and / or the metadata become known.

[0045] In some embodiments, the prediction of the time remaining to the milestone may change when the phase or activity of a case or surgical procedure changes (e.g., transitions from one phase to another). For example, when one activity (e.g., patient preparation) completes and the next activity (e.g., patient intubation) begins, the prediction of the time remaining to the milestone may be updated. The event system 150 may compare the expected time of completion of a particular activity or activity group with an actual time spent on that activity or activity group and make adjustments to the predicted time remaining to the milestone. For example, an activity may be expected to take time T1 to complete. The initial time remaining to milestone may be predicted based on the time Tl. However, if the actual time of completion of that activity ends up being time T2 different from Tl (e.g., the activity takes longer to complete or takes less time to complete), the event system 150 updates the initial time remaining to milestone prediction to an updated time remaining to milestone prediction computed based on the time T2. Thus, in some embodiments, the event system 150 may determine an actual time of completion of an activity, determine that the actual time of completion of the activity deviates from the expected time of completion of the non-operative activity, and update the time remaining to the milestone responsive to determining the deviation.

[0046] In some embodiments, the event system 150 may update the time remaining to milestone prediction when the metadata changes. For example, in some embodiments, if something about the patient, medical environment conditions, medical professional (e.g.,staffing or surgeon changes), medical procedure itself, or something else deviates from the expected or initial metadata, the event system 150 may update the initial time remaining to milestone prediction. For example, if during the surgical procedure, the surgeon discovers a patient condition that was previously unknown, the newly discovered patient condition may increase the total duration of the surgical procedure, which in turn may impact the time remaining to milestone prediction. The event system 150 may update the time remaining to milestone prediction when the milestone prediction model detects deviations from the metadata initially used to make the initial time remaining to event system 150. Thus, the event system 150 may determine that the initial metadata has changed to an updated metadata and update the time remaining to the milestone based on the updated metadata. The event system 150 may update the predictions at other stages of the medical session. Thus, the event system 150 continuously updates and adjusts the prediction as changes to the inputs are detected.

[0047] In some embodiments, the event system 150 may be configured to predict adverse events. Adverse events may be those events that are not expected to occur during a normal commencement of the medical session. Predicting adverse events may improve patient outcomes. The adverse events may be associated with the patient itself or with the surroundings of the patient (e.g., occurrences in the medical environment). For example, an adverse event may be blood loss (e.g., excessive bleeding) of the patient during the surgical procedure. The event system 150 may predict whether blood loss is expected to occur based at least on a type of the surgical procedure being performed on the patient, the body location where the surgical procedure is being performed on the patient, medical history of the patient (e.g., blood pressure, smoking, etc.), experience of the medical professional, endoscopic videos and other imagery (e.g., X-ray) captured of the patient, event logs from the robotic medical system 125, etc. Another example of an adverse event may be likelihood of infections. For example, the event system 150 may predict a likelihood that the patient may get infected based at least on sterility breaches within the medical environment, experience of the medical professional, size of the medical team, type of the surgical procedure, rate of opening / closing of the door(s) to the medical environment, modality of the surgical procedure, etc.

[0048] Another example of an adverse event may be a collapsed lung of the patient. The event system 150 may predict the likelihood that a lung of the patient may collapse based at least on the experience of the medical professionals (e.g., the anesthesiologist, surgeon, etc.), type of the surgical procedure, medical history of the patient, endoscopic videos and other imagery (e.g., X-ray) captured of the patient. Accidental or mistaken cuts / incisions of body parts (e.g., bowel, ureter, major arteries / veins, lung puncture, etc.) of the patient may also beconsidered adverse events. The event system 150 may predict such adverse events based, for example, on the experience of the medical professional, endoscopic videos and other imagery (e.g., X-ray) captured of the patient, event logs from the robotic medical system 125, etc. Bowel leakage may be yet another adverse event, which the event system 150 may predict based at least on the type of the surgical procedure, the experience of the medical professional, medical history of the patient, event logs from the robotic medical system 125, and endoscopic videos and other imagery (e.g., X-ray) captured of the patient. The event system 125 may predict other types of adverse events associated with the patient and the surroundings of the patient.

[0049] In some embodiments, adverse events may be input into the event system 150 to update the time remaining to a milestone prediction. For example, in some embodiments, adverse events may be determined or inferred by other mechanisms such as a computer vision model which may be similar to the activity recognition model 145. In some embodiments, adverse events may be input into the event system 150 manually (e.g., using auditory cues by the medical professionals). In some embodiments, the adverse events may be determined in other ways. Upon determining that an adverse event has occurred, the adverse event may be input into the event system 150 to update the time remaining to milestone prediction. In some embodiments, the event system 150 may update the time remaining to milestone prediction based on the occurrence of the adverse event, a type of the adverse event that has occurred, and / or other aspects associated with the adverse event. Thus, adverse events may both be an input into the event system 150 and / or an output from the event system. Based on the prediction of the adverse events and / or the actual occurrence of one or more adverse events during the medical session, the event system 150 may update the time remaining to the milestone prediction.

[0050] The event system 150 may be configured to predict preparatory tasks such as events that may require additional testing (e.g., blood work, histology tests, imagery, etc.), additional equipment or instruments for the surgical procedure that may generally not be required in a normal instance of the surgical procedure, etc. The event system 150 may use similar indicators as those mentioned for the adverse events to predict the instrument needs, needs for additional tests, etc. For example, the event system 150 may consume event logs from the robotic medical system 125 to predict when certain instruments may be needed during the surgical procedure. The event system 150 may also consume information (e.g., metadata) related to patient complexity (e.g., medical history), type of the surgical procedure (e.g., to determine if unusual instruments or equipment may be needed for the surgical procedure),experience of the medical professional team (e.g., to determine likelihood of complications - higher experience may indicate less chance of complication), time taken to complete certain non-operative activities (e.g., to determine likelihood of complications - greater the time taken relative to a normal amount of time for an activity, higher the chance of a complication), and endoscopic videos from the surgical procedure. In some embodiments, the event system 150 may receive auditory cues from the medical environment. Such auditory cues may be indicative of happenings in the medical environment and may help to determine, what if anything, may have gone wrong and what instrum ent / equipment may be needed in the medical environment. By predicting when instruments or equipment may be needed, the event system 150 may reduce the amount of time that the medical professional may spend in the medical environment waiting for the arrival of the instrum ents / equipment. The event system 150 may also predict preparatory tasks that may need to be completed for the next activity or phase or stage of the medical session to begin. Based on the prediction of the preparatory tasks, the event system 150 may update the time remaining to the milestone prediction. In some embodiments, the event system 150 may predict the time remaining to the preparatory task.

[0051] In some embodiments, the medical session may be divided into stages (e.g., presurgery stage, surgery stage, post-surgery stage, etc.). Each stage may include one or more activities that may be performed within that stage. The activity recognition model 145 may determine the stage of the medical session. In some embodiments, each activity may include sub-activities, one or more of which may be occurring at the same time. The event system 150 may track the sub-activities and update the time remaining to milestone predictions to reflect latest information. The event system 150 may send the time remaining to milestone prediction to an action system 160.

[0052] The action system 160 may take an action responsive to the time remaining to the milestone prediction. An example action may be sending one or more alerts (e.g., notification, communications, etc.) based on the time remaining to the milestone prediction received from the event system 150. In some embodiments, the action system 160 may be programmed to send an alert a threshold amount of time before the milestone. For example, in some embodiments, the action system 160 may be programmed to send an alert 10-15 minutes before the end of the medical session. As another example, if the predicted time to a milestone is 20 minutes and the action system 160 is programmed to send an alert 15 minutes before the milestone, the event system may wait for 5 minutes after receiving the predicted time remaining to milestone before sending an alert.

[0053] The action system 160 may also determine who to send the alert to based on the milestone. For example, if the milestone is the end of the medical session, based on the current activity, the action system 160 may determine that the next activity is a turnover activity to clean the medical environment and prepare the medical environment for the next patient. The action system 160 may also determine the individual(s) responsible for the turnover activities and send an alert to those individual(s). As another example, the milestone may be end of the pre-surgery group of activities. The action system 160 may determine that the surgery-related activities are next, identify the individual(s) (e.g., surgeon) needed for the next group of activities, and send an alert to those individual(s).

[0054] The action system 160 may send the alerts in any of a variety of ways. For example, in some embodiments, the action system 160 may send a communication to a user (e.g., medical professional). In some embodiments, the action system 160 may send the communication in any suitable way. For example, in some embodiments, the action system 160 may send an email to a mobile device (e.g., cellphone, paging device, personal digital assistant, smart watch, etc.) of the user. In some embodiments, the action system 160 may send the alert to the user in other ways. In some embodiments, the action system 160 may use an Application Programming Interface (API) 165 to send the alert. In some embodiments, the action system 160 may send a push notification to a mobile device of the user. In some embodiments, the action system 160 may push the alert to a mobile application installed on the user’s mobile device. The action system 160 may use other mechanisms to send the alerts.

[0055] In some embodiments, the action system 160 may take actions that include displaying information. For example, the action system 160 may display the alert on a display (e.g., the display 140). The action system 160 may also display the predicted time remaining to the milestone (and any other suitable details of the medical session) on the display (e.g., the display 140). The action system 160 may display information related to an upcoming (e.g., next) stage of the medical session. The action system 160 may display information related to the next activity or phase of the medical session. The action system 160 may display information related to one or more preparatory tasks that need to be completed for transitioning to the next phase or stage of the medical session. For example, the action system 160 may display information related to what additional instruments may be needed, what additional tests may need to be performed, etc. In some embodiments, the action system 160 may display the predicted time remaining to the milestone (and any other suitable details of the medical session) on another display. In some embodiments, the action system 160 may take actions that include sending an instruction to perform a task. For example, the action system 160 may sendan instruction for a scheduling operation to trigger scheduling or planning for the next group of activities. The action system 160 may request performance of the preparatory tasks. For example, the action system 160 may request any additional instruments that may be needed to be brought into the medical environment. The action system 160 may send notifications to request personnel / equipment to perform additional tests or capture imagery from the patient in the medical environment. The action system 160 may adjust the schedules of when one or more medical professionals are needed in the medical environment based on the predicted time remaining to the milestones. The action system 160 may adjust the schedules of which one or more medical professionals are needed in the medical environment based on the predicted time remaining to the milestone. The action system 160 may also adjust the scheduling of tasks that may need to be performed and the medical professionals that are needed to perform those tasks based on the predicted time remaining to the milestone prediction. In other embodiments, the action system 160 may send other types of alerts or take other actions.

[0056] The content of the message or instruction included in the action taken by the action system 160 may vary depending on the type of action, the format of the action, the individual(s) that the actions are directed to, and / or the activities or actions that need to occur next. In some embodiments, the content of the action may indicate what the next activity is for which a particular individual is needed. The content of the message or instruction included in the action may also indicate the time period by when and where the next activity is expected to begin. The content of the message or instruction may also indicate what action the particular individual is expected to take. For example, an alert to a surgeon may indicate that the surgery of a patient is expected to begin in 15 minutes in operating room A. The content of the alert sent by the action system 160 may include other or additional pieces of data.

[0057] The data processing system 135 may include a physical or virtual computer system operatively coupled, or associated with, the medical environment 105. In some embodiments, the data processing system 135 may be coupled, or associated with, the medical environment 105 via a network, either directly or through an intermediate computing device or system. The network may be any type or form of network. The geographical scope of the network may vary widely and may include a body area network (BAN), a personal area network (PAN), a localarea network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network may assume any form such as point-to- point, bus, star, ring, mesh, tree, etc. The network may utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (SynchronousOptical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite may include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network may be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0058] The data processing system 135 may be located at least partially at the location of the medical environment associated with the medical environment 105 or remotely therefrom. At least some elements of the data processing system 135 may be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 135 may include other or additional elements that may be considered desirable to have in performing the functions described herein. The data processing system 135 may include, or be associated with, a system processor 170. The system processor 170 may execute one or more instructions associated with the system 100. The system processor 170 may include an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The system processor 170 may include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), or the like. The system processor 170 may include, or be associated with, a memory 175 operable to store or storing one or more non-transitory computer-readable instructions for operating components of the system processor and operating components operably coupled to the system processor. The one or more instructions may include at least one of firmware, software, hardware, operating systems, embedded operating systems, and the like. The system processor 170 or the system 100 generally may include at least one communication bus controller to effect communication between the system processor and the other elements of the system 100.

[0059] The memory 175 may include one or more hardware memory devices to store binary data, digital data, or the like. The memory 175 may include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units, or the like. The memory 175 may include at least one of a non-volatile memory device, a solid-state memory device, a flash memory device, a NAND memory device, a volatile memory device, etc. The memory 175 may include one or more addressable memory regions disposed on one or more physical memory arrays.

[0060] Turning to FIG. 2, an example graph 200 plots predictions for time remaining to a milestone. The milestone plotted in the graph 200 is the end of a case. The graph 200, thus, plots a time to the end of the medical session on Y-axis 205 against a time since a patient was brought into a medical environment on X-axis 210. The graph 200 includes a first plot 215 showing an actual amount of time remaining to the end of the case and a second plot 220 showing a predicted amount of time until the end of the case. When the patient is initially brought in, the predicted (and actual) time to the end of the case is high. As the amount of time that the patient spends in the medical environment increases, the amount of time to the end of the case decreases. Prediction may be made continuously at any point in time. “X” notations on the first plot 215 may correspond to start or end of certain activities. There may be a sharp adjustment (e.g., predictions may be updated) when an activity starts or ends. “Jitters” 225 on the second plot 220 may indicate locations where a deviation from the expected value occurred (e.g., activity completed sooner than expected, activity took longer than expected, metadata change occurred) causing the event system 150 to update the time remaining to milestone prediction, as discussed above. Thus, the event system 150 may use a combination of the metadata and the current activity in a case to predict time remaining to milestone and dynamically update the prediction as certain data points change.

[0061] Turning now to FIG. 3, an example flowchart outlining the operations of a process 300 is shown, in accordance with some embodiments of the present disclosure. The process 300 may be performed by a system having one or more processors executing computer- readable instructions stored on a memory. For example, the process 300 may be performed by the data processing system 135, and particularly, by the system processor 170 that executes non-transitory computer-readable instructions stored on a memory (e.g., the memory 175).

[0062] The process 300 begins with the processor (e.g., the system processor 170) receiving the metadata and any other inputs at operation 305. The metadata may be input into the metadata system 155 in advance or in real-time. The any other inputs may include the manual inputs 180, the inputs from the medical tool 120, and / or the inputs from the robotic medical system 125. At operation 310, the processor (e.g., the system processor 170) receives one or more data streams (e.g., videos) from the medical environment 105. In some embodiments, the one or more data streams may be captured from various viewpoints by the sensor system 110 using the sensors 115 located in the medical environment 105 (e.g., located in the medical environment such as the operating room). The one or more data streams may provide real-time data from the medical environment 105. The one or more data streams may or may not be pre-processed in the data stream processing system 130 and the output thereofmay be input into the activity recognition model 145. The activity recognition model 145 determines an activity in the medical environment 105 at operation 315 based on the output of the data stream processing system 130.

[0063] The activity recognition model 145 may determine an activity occurring at the current time within the medical environment 105. Because the one or more data streams provide real-time data, the activity recognition model 145 may determine the activities in the medical environment 105 in real-time. At operation 320, the processor (e.g., the system processor 170) uses the event system 150 to predict the time remaining to an end of the milestone. The current activity from the operation 315 and the metadata and any other inputs from the operation 305 are input into the event system 150 at operation 320. The event system 150 outputs a time remaining to the end of the milestone predicted based on the combined current activity and the metadata. The event system 150 may also determine when updates to either the metadata or the current activity are made and update the time remaining to the milestone prediction. The event system 150 may also predict any adverse events. At operation 325, the processor (e.g., the system processor 170) performs an action (e.g., takes an action) responsive to the predicted adverse event and / or the time remaining to milestone predicted at the operation 320. Specifically, the action system 160 may take action a threshold amount of time before the predicted milestone and / or adverse event. The action may include sending an alert to an individual or group of individuals. The action may include displaying data. The action may include sending an instruction for a task to be performed. The action system 160 may take other actions.

[0064] Referring to FIG. 4, a surgical system 400 is shown, in accordance with some embodiments. The surgical system 400 may be an example of the medical environment 105. The surgical system 400 may include a robotic medical system 405 (e.g., the robotic medical system 125), a user control system 410, and an auxiliary system 415 communicatively coupled one to another. A visualization tool 420 may be connected to the auxiliary system 415, which in turn may be connected to the robotic medical system 405. Thus, when the visualization tool 420 is connected to the auxiliary system 415 and this auxiliary system is connected to the robotic medical system 405, the visualization tool may be considered connected to the robotic medical system. In some embodiments, the visualization tool 420 may be directly connected to the robotic medical system 405.

[0065] The surgical system 400 may be used to perform a computer-assisted medical procedure on a patient 425. In some embodiments, surgical team may include a surgeon 430 A and additional medical personnel 430B-430D such as a medical assistant, nurse, andanesthesiologist, and other suitable team members who may assist with the surgical procedure or medical session. The medical session may include the surgical procedure being performed on the patient 425, as well as any pre-operative (e.g., which may include setup of the surgical system 400, including preparation of the patient 425 for the procedure), and post-operative (e.g., which may include clean up or post care of the patient), and / or other processes during the medical session. Although described in the context of a surgical procedure, the surgical system 400 may be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that may benefit from the accuracy and convenience of the surgical system.

[0066] The robotic medical system 405 may include a plurality of manipulator arms 435 A- 435D to which a plurality of medical tools (e.g., the medical tool 120) may be coupled or installed. Each medical tool may be any suitable surgical tool (e.g., a tool having tissueinteraction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that may be used for a computer-assisted surgical procedure on the patient 425 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer- assisted surgical procedure on the patient). Although the robotic medical system 405 is shown as including four manipulator arms (e.g., the manipulator arms 435A-435D), in other embodiments, the robotic medical system may include greater than or fewer than four manipulator arms. Further, not all manipulator arms may have a medical tool installed thereto at all times of the medical session. Moreover, in some embodiments, a medical tool installed on a manipulator arm may be replaced with another medical tool as suitable.

[0067] One or more of the manipulator arms 435A-435D and / or the medical tools attached to manipulator arms may include one or more displacement transducers, orientational sensors, positional sensors, and / or other types of sensors and devices to measure parameters and / or generate kinematics information. One or more components of the surgical system 400 may be configured to use the measured parameters and / or the kinematics information to track (e.g., determine poses of) and / or control the medical tools, as well as anything connected to the medical tools and / or the manipulator arms 435A-435D.

[0068] The user control system 410 may be used by the surgeon 430A to control (e.g., move) one or more of the manipulator arms 435A-435D and / or the medical tools connected to the manipulator arms. To facilitate control of the manipulator arms 435A-435D and track progression of the medical session, the user control system 410 may include a display that may provide the surgeon 430 A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 425 as captured by a medical tool (e.g., the medical tool 120, whichmay be an endoscope) installed to one of the manipulator arms 435A-435D. The user control system 410 may include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 425 and generated by a stereoscopic imaging system may be viewed by the surgeon 430 A. In some embodiments, the user control system 410 may also receive images from the auxiliary system 415 and the visualization tool 420.

[0069] The surgeon 430 A may use the imagery displayed by the user control system 410 to perform one or more procedures with one or more medical tools attached to the manipulator arms 435A-435D. To facilitate control of the manipulator arms 435A-435D and / or the medical tools installed thereto, the user control system 410 may include a set of controls. These controls may be manipulated by the surgeon 430A to control movement of the manipulator arms 435A-435D and / or the medical tools installed thereto. The controls may be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 430A to allow the surgeon to intuitively perform a procedure on the patient 425 using one or more medical tools installed to the manipulator arms 435A-435D.

[0070] The auxiliary system 415 may include one or more computing devices configured to perform processing operations within the surgical system 400. For example, the one or more computing devices may control and / or coordinate operations performed by various other components (e.g., the robotic medical system 405, the user control system 410) of the surgical system 400. A computing device included in the user control system 410 may transmit instructions to the robotic medical system 405 by way of the one or more computing devices of the auxiliary system 415. The auxiliary system 415 may receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical tools) attached to the robotic medical system 405, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices (e.g., the sensor system 110) may be located within the surgical system 400. These image capture devices may capture images from various viewpoints within the surgical system 400. These images (e.g., video streams) may be transmitted to the visualization tool 420, which may then passthrough those images to the auxiliary system 415 as a single combined data stream. The auxiliary system 415 may then transmit the single video stream (including any data stream received from the medical tool(s) of the robotic medical system 405) to present on a display of the user control system 410.

[0071] In some embodiments, the auxiliary system 415 may be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medicalpersonnel 430B-430D) who may not have access to the user control system 410. Thus, the auxiliary system 415 may include a display 440 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 425 and / or the surgical procedure, and / or any other visual content (e.g., the single combined data stream). In some embodiments, display 440 may be a touchscreen display and / or include other features to allow the medical personnel 430A-430D to interact with the auxiliary system 415.

[0072] The robotic medical system 405, the user control system 410, and the auxiliary system 415 may be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 405, the user control system 410, and the auxiliary system 415 may be communicatively coupled by way of control lines 445, which may represent any wired or wireless communication link as may serve a particular implementation. Thus, the robotic medical system 405, the user control system 410, and the auxiliary system 415 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0073] It is to be understood that the surgical system 400 may include other or additional components or elements that may be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0074] Turning to FIG. 5, an example block diagram of an example computer system 500 is shown, in accordance with some embodiments. The computer system 500 may be any computing device used herein and may include or be used to implement a data processing system or its components. The computer system 500 includes at least one bus 505 or other communication component or interface for communicating information between various elements of the computer system. The computer system 500 further includes at least one processor 510 or processing circuit coupled to the bus 505 for processing information. The computer system 500 also includes at least one main memory 515, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information, and instructions to be executed by the processor 510. The main memory 515 may be used for storing information during execution of instructions by the processor 510. The computer system 500 may further include at least one read only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510. A storage device 525, such as a solid-state device, magnetic disk or optical disk, may be coupled to the bus 505 to persistently store information and instructions.

[0075] The computer system 500 may be coupled via the bus 505 to a display 530, such asa liquid crystal display, or active-matrix display, for displaying information. An input device 535, such as a keyboard or voice interface may be coupled to the bus 505 for communicating information and commands to the processor 510. The input device 535 may include a touch screen display (e.g., the display 530). The input device 535 may also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 510 and for controlling cursor movement on the display 530.

[0076] The processes, systems and methods described herein may be implemented by the computer system 500 in response to the processor 510 executing an arrangement of instructions contained in the main memory 515. Such instructions may be read into the main memory 515 from another computer-readable medium, such as the storage device 525. Execution of the arrangement of instructions contained in the main memory 515 causes the computer system 500 to perform the illustrative processes described herein. One or more processors in a multiprocessing arrangement may also be employed to execute the instructions contained in the main memory 515. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0077] Although an example computing system has been described in FIG. 5, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0078] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited tophysically mateable and / or physically interacting components and / or wirelessly interactable and / or wirelessly interacting components and / or logically interacting and / or logically interactable components.

[0079] With respect to the use of plural and / or singular terms herein, those having skill in the art can translate from the plural to the singular and / or from the singular to the plural as is appropriate to the context and / or application. The various singular / plural permutations may be expressly set forth herein for sake of clarity.

[0080] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0081] Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above. Such variation may depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0082] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and / or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least therecited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0083] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0084] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0085] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: one or more processors, coupled with memory to: determine, based on at least one data stream, a non-operative activity during a medical session in a medical environment, the at least one data stream captured using at least one sensor located in the medical environment; predict, using a machine learning model and based at least on the non-operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session; and perform an action responsive to predicting the time remaining to the milestone.

2. The system of claim 1, wherein the action comprises one or more of presenting the time remaining to the milestone on a display, presenting information related to a subsequent stage of the medical session on the display, or presenting information related to a preparatory task on the display.

3. The system of claim 1, wherein the action comprises sending an alert requesting presence of a medical professional in the medical environment.

4. The system of claim 1, wherein the action comprises one or more of adjusting a schedule of which one or more medical professionals are needed in the medical environment based on the time remaining to a milestone, adjusting the schedule of when the one or more medical professionals are needed in the medical environment, or adjusting the schedule of one or more tasks to be performed by the one or more medical professionals.

5. The system of any of claims 1-4, wherein the milestone comprises one or more of an end of the medical session, an end of a surgical procedure being performed during the medical session, a start of the medical session, start of a surgical procedure, instrument need in the medical environment, an adverse event, need for intra-operative test during the surgical procedure, or need for intra-operative imagery during the surgical procedure.

7. The system of claim 1, wherein the metadata comprises at least one of patient data, medical personnel data, medical environment data, or medical session data associated with the medical session, and wherein the metadata is continuously updated as the medical session progresses.

8. The system of claim 1, wherein a first type of the metadata has a greater impact on the prediction for the time remaining to the milestone than a second type of the metadata, and wherein the one or more processors automatically adjust a weight assigned to each of the first type of metadata and the second type of metadata to assign a higher weight to the first type of metadata than the second type of metadata in predicting the time remaining to the milestone.

9. The system of claim 1, wherein the non-operative activity is associated with an expected time of completion of the non-operative activity, and wherein the one or more processors: determine an actual time of completion of the non-operative activity; determine that the actual time of completion of the non-operative activity deviates from the expected time of completion of the non-operative activity; and update the time remaining to the milestone responsive to determining the deviation.

10. The system of claim 1, wherein the metadata used to predict the time remaining to the milestone comprises initial metadata, and wherein the one or more processors: determine that the initial metadata has changed to an updated metadata; and update the time remaining to the milestone based on the updated metadata.

11. The system of claims 9 or 10, wherein updating the time remaining to the milestone is performed in real-time.

12. The system of claim 10, wherein the initial metadata comprises a first set of medical professionals during the medical session, wherein the updated metadata comprises a second set of medical professionals different from the first set of medical professionals, and wherein the one or more processors update the time remaining to the milestone based on the second set of medical professionals.

13. The system of claim 1, wherein the one or more processors:determine occurrence of an adverse event ; and update the time remaining to the milestone based on the occurrence of the adverse event.

14. The system of claim 1, wherein the non-operative activity is performed by a nonsurgeon.

15. The system of claim 14, wherein the non-operative activity comprises at least one of a turnover activity, a setup activity, a sterile preparation activity, a robot draping activity, a patient-in activity, a patient preparation activity, a port placement activity, a room preparation activity, a rollup activity, a docking activity, a surgery activity, an undocking activity, a rollback activity, a patient close activity, a patient undraping activity, a robot undraping activity, a patient-out activity, a cleaning activity, or an idle activity.

16. The system of claim 15, wherein the patient preparation activity comprises at least one of an intubation activity, a patient draping activity, or a first cut activity.

17. The system of claim 1, wherein the at least one sensor is configured to capture data of one or more modalities within the medical environment.

18. A non-transitory computer-readable media comprising computer-readable instructions stored thereon that when executed by one or more processors cause the one or more processors to: determine, based on at least one data stream, a non-operative activity during a medical session in a medical environment, the at least one data stream captured using at least one sensor located in the medical environment; predict, using a machine learning model and based at least on the non-operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session; and take action responsive to predicting the time remaining to the milestone.

19. The non-transitory computer-readable media of claim 18, wherein the action comprises one or more of one or more of presenting the time remaining to the milestone on a display, presenting information related to a subsequent stage of the medical session on the display,presenting information related to a preparatory task on the display, sending an alert requesting presence of a medical professional in the medical environment, adjusting a schedule of which one or more medical professionals are needed in the medical environment based on the time remaining to the milestone, adjusting the schedule of when the one or more medical professionals are needed in the medical environment, or adjusting the schedule of one or more tasks to be performed by the one or more medical professionals.

20. The non-transitory computer-readable media of claims 18 or 19, wherein the milestone comprises one or more of an end of the medical session, an end of a surgical procedure being performed during the medical session, a start of the medical session, start of the surgical procedure, instrument need in the medical environment, an adverse event, need for intraoperative test during the surgical procedure, or need for intra-operative imagery during the surgical procedure.

21. The non-transitory computer-readable media of claim 18, wherein the metadata comprises at least one of patient data, medical personnel data, medical environment data, or medical session data associated with the medical session, and wherein the metadata is continuously updated as the medical session progresses.

22. The non-transitory computer-readable media of claim 18, wherein the non-operative activity is associated with an expected time of completion of the non-operative activity, and wherein the one or more processors: determine an actual time of completion of the non-operative activity; determine that the actual time of completion of the non-operative activity deviates from the expected time of completion of the non-operative activity; and update the time remaining to the milestone responsive to determining the deviation.

23. The non-transitory computer-readable media of claim 18, wherein the metadata used to predict the time remaining to the milestone comprises initial metadata, and wherein the one or more processors: determine that the initial metadata has changed to an updated metadata; and update the time remaining to the milestone based on the updated metadata.

24. The non-transitory computer-readable media of claims 22 or 23, wherein updating the time remaining to the milestone is performed in real-time.

25. The non-transitory computer-readable media of claim 18, wherein the non-operative activity comprises at least one of a turnover activity, a setup activity, a sterile preparation activity, a robot draping activity, a patient-in activity, a patient preparation activity, a port placement activity, a room preparation activity, a rollup activity, a docking activity, a surgery activity, an undocking activity, a rollback activity, a patient close activity, a patient undraping activity, a robot undraping activity, a patient-out activity, a cleaning activity, or an idle activity.

26. A method comprising: determining, based on at least one data stream, a non-operative activity during a medical session in a medical environment, the at least one data stream captured using at least one sensor located in the medical environment; predicting, using a machine learning model and based at least on the non-operative activity and metadata associated with the medical session, a time remaining to a milestone associated with the medical session; and taking action responsive to predicting the time remaining to the milestone.

27. The method of claim 26, wherein the non-operative activity is associated with an expected time of completion of the non-operative activity, and wherein the method further comprises: determining an actual time of completion of the non-operative activity; determining that the actual time of completion of the non-operative activity deviates from the expected time of completion of the non-operative activity; and updating the time remaining to the milestone responsive to determining the deviation.

28. The method of claim 26, wherein the metadata used to predict the time remaining to the milestone comprises initial metadata, and wherein the method further comprises: determining that the initial metadata has changed to an updated metadata; and updating the time remaining to the milestone based on the updated metadata.

Images