Digital management and coordination of surgical theater tasks

Abstract

Various of the disclosed embodiments provide guidance to team members in a surgical theater and improve surgical theater efficiency by providing dynamic and context- aware action assignments to various of the theater team members. Real-time monitoring of the surgical theater with one or more sensors located within the surgical theater may facilitate identification of personnel, the state of the personnel's respective task actions, and the general state of the theater. This monitoring may be coupled with tailored task overlap model management and rendering systems disclosed herein to provide team members with immediate feedback, as well as to encourage gradual adjustments nudging team members to more optimal performances.

Description

DIGITAL MANAGEMENT AND COORDINATION OF SURGICAL THEATER TASKSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of, and priority to, United States Provisional Application No. 63 / 526,971 , filed upon July 14, 2023, entitled “DIGITAL MANAGEMENT AND COORDINATION OF SURGICAL THEATER TASKS”, and which is incorporated by reference herein in its entirety for all purposes.TECHNICAL FIELD

[0002] Various of the disclosed embodiments relate to computer-implemented approaches for managing surgical theater tasks and for improving team member performance.BACKGROUND

[0003] Within the complicated, time-sensitive, and high stakes environment of the surgical theater, team members must coordinate disparate tasks among themselves during dynamic situations, appreciating that one team member’s actions may have consequences for another team member at some remote point in the future. Indeed, improper action, or lack of action, taken during nonoperative periods may have consequences for downstream operative periods, and vice versa. Failing to perform a task, performing tasks in the improper order, or performing tasks in an uncoordinated manner, can delay downstream actions and extend the duration of the surgeries and operating room turnarounds. Prolonged surgeries can themselves precipitate a cascade of additional failures, including complicating scheduling of future surgeries in the theater and subjecting the patient to additional, unplanned, anesthetic exposure. While all these challenges and considerations exist in traditional, non-robotic theaters, robotic system surgical theaters, with their additional hardware and configuration requirements, can further exacerbate these complications. Further still, team members may be asked to readily transition from performing procedures in robotic system surgical theaters to procedures in non-robotic theaters, and vice versa, requiring that the members readily adapt their approach for a variety of disparate tasks to the varying contexts.

[0004] While team members must perform some tasks in sequence with one another (e.g., one team member must complete closure before another can escort the patient from the theater), many tasks may be performed in parallel between the team members (e.g., one team member may begin applying anesthesia while another prepares a surgical instrument). As real-time verbal discussion and coordination between the team-members to achieve the benefits of parallelization are impractical, various hospitals and vendors perform workflow analyses to identify efficient task allocations during operative and nonoperative periods. These efficient allocations may then be memorialized into a Task Overlap Model (TOM), a centralized record of the task assignments between the respective team members. While often used for nonoperative periods, analogous documents may be created in some situations for operative periods. Such records may be presented as identical printouts shared among the team members, with the expectation that the team members will review the printout throughout the relevant operative or nonoperative periods, so as to effect the desired efficient workflow.

[0005] While these static TOMs provide a considerable improvement over ad hoc, real-time coordination between the team members, having been shown to improve efficiency, increase case volume, and improve equipment utilization for hospitals, static TOMs generally fail to account for the dynamic character of the theater. Indeed, many teams merely take the TOM as a “suggestion” and instead rely upon real-time coordination amongst themselves, since the TOM cannot accommodate for variations unaccounted for in the original workflow analysis, nor readily adapt itself to unexpected changes in the state of the theater. Additionally, the TOM printout cannot be readily adjusted when equipment and personnel change. Indeed, while the presence of an additional, and unexpected, team member able to assist with a surgery would typically be considered an asset, if the member’s presence was not accounted for by the original TOM, the additional member’s presence may be extraneous and underutilized.

[0006] In addition, requiring operating room (OR) personnel to constantly refer to TOM printouts during operative and nonoperative periods imposes an undesirable distraction upon each member’s cognitive load, as the OR staff must constantly refer to the TOM to find their assigned task at each step. Similarly, manually comparing the TOM’s recommended time for a task to a timepiece, while also mentally noting the timethat the team member began the task, is often impractical and very often distracting. In addition, TOM printouts are often limited to a single page, as any more information than this would be overwhelming. However, this limited space precipitates a laconism, which can itself be ambiguous and confusing to the team. Reviewers and hospital management seeking to improve the team’s efficiency may have limited options for adjusting the TOM or for confirming the team’s adherence to a modified TOM. Rarely will an outside party understand why a team departed from a TOM or be able to readily infer causal relations from a video record.

[0007] While on-site consultants who participated in creating the TOMs may be initially available for a few days to assist the team when departures occur, expecting teams to later urgently solicit an update from a consultant (who may no longer be available, and if available, is generally limited to assisting only one team at a time) is not an ideal expectation. Indeed, the consultants that draft TOMs are often only available for a couple of days to provide training to the hospital staff and are typically not permanently stationed at the hospital (such lack of continuous monitoring and feedback also making it difficult to verify the team’s adherence to any proposed workflow changes). Accordingly, such consultants cannot generally track the team’s day-to-day improvement (or lack thereof), and consequently, when they are consulted, cannot provide granular advice or adjustment recommendations suitable for that particular team or team member. Thus, customizing TOMs for specific numbers of staff, procedure types, processes, and other variations is often impractical, time consuming, and inefficient as such customization may require expensive expert knowledge, which is not always available. Indeed, because they are so time consuming to produce, hospitals often only create TOMs for teams with known inefficiencies, denying “efficient” teams opportunities to further improve.

[0008] Accordingly, there exists a need for systems and methods to overcome challenges and difficulties such as those described above. For example, there exists a need for systems to process disparate forms of surgical theater data and to present the processed data to team members in an insightful manner facilitating their coordinated and efficient action.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various of the embodiments introduced herein may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements:

[0010] FIG. 1A is a schematic view of various elements appearing in a surgical theater during a surgical operation, as may occur in relation to some embodiments;

[0011] FIG. 1 B is a schematic view of various elements appearing in a surgical theater during a surgical operation employing a robotic surgical system, as may occur in relation to some embodiments;

[0012] FIG. 2A is a schematic depth map rendering from an example theater-wide sensor perspective, as may be used in some embodiments;

[0013] FIG. 2B is a schematic top-down view of objects in the theater of FIG. 2A, with corresponding sensor locations;

[0014] FIG. 2C is a pair of images depicting a grid-like pattern of orthogonal rows and columns in perspective, as captured from a theater-wide visual image sensor having a rectilinear view and a theater-wide visual image sensor having a fisheye view, each of which may be used in connection with some embodiments;

[0015] FIG. 3 is a schematic depiction of data that may be acquired during a surgically operative period, as may occur in connection with some embodiments;

[0016] FIG. 4 is a schematic depiction of data that may be acquired during a surgically nonoperative period, as may occur in connection with some embodiments;

[0017] FIG. 5 is a schematic block diagram illustrating various example task action assignments in an example TOM, as may occur in connection with some embodiments;

[0018] FIG. 6 is a schematic block diagram illustrating an example component processing topology for a dynamic TOM management system, as may be implemented in some embodiments;

[0019] FIG. 7A a schematic representation of successive TOM theater states when encountering a dependency block, as may occur in some embodiments;

[0020] FIG. 7B is a schematic block diagram illustrating example dependency relations between various task actions, as may occur in some embodiments;

[0021] FIG. 8A is a schematic block diagram illustrating various components in an example data structure for representing and monitoring TOM action completion by team members, as may be used in some embodiments;

[0022] FIG. 8B is a schematic block diagram illustrating example relations between action data structures within a digital TOM representation, as may occur in some embodiments;

[0023] FIG. 9 is a flow diagram illustrating various operations in an example action transition management process, as may be implemented in some embodiments;

[0024] FIG. 10 is a schematic graphical user interface (GUI) rendering of elements in a TOM task status display, as may be implemented in some embodiments;

[0025] FIG. 11A is a schematic representation of elements in a theater presenting the GUI of FIG. 10, as may occur in connection with various embodiments;

[0026] FIG. 11 B is an enlarged view of the mounted display rendering in the schematic representation of FIG. 10, as may occur in connection with various embodiments;

[0027] FIG. 11 C is an enlarged view of a console display rendering in the schematic representation of FIG. 10, as may occur in connection with various embodiments;

[0028] FIG. 11 D is an enlarged view of the handheld display rendering in the schematic representation of FIG. 10, as may occur in connection with various embodiments;

[0029] FIG. 12 is a collection of configuration variations of the GUI of FIG. 10, as may be deployed in some embodiments;

[0030] FIG. 13 is a flow diagram illustrating various operations in an example process for preparing a TOM representation, as may be implemented in some embodiments;

[0031] FIG. 14 is a flow diagram depicting various operations in an example process for managing the GUI representations of FIGs. 10 and 11A-D, as may be implemented in some embodiments;

[0032] FIG. 15 is a flow diagram depicting various operations in an example process for reallocating personnel, as may be implemented in some embodiments;

[0033] FIG. 16 is a schematic block diagram illustrating various example TOM representation states during personnel additions, as may occur in some embodiments;

[0034] FIG. 17A is a collection of schematic block diagrams depicting example incremental performance adjustment plans, as may be implemented in some embodiments;

[0035] FIG. 17B is a flow diagram illustrating various operations in an example process for performing incremental performance guidance and monitoring, as may be implemented in some embodiments; and

[0036] FIG. 18 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments.

[0037] The specific examples depicted in the drawings have been selected to facilitate understanding. Consequently, the disclosed embodiments should not be restricted to the specific details in the drawings or the corresponding disclosure. For example, the drawings may not be drawn to scale, the dimensions of some elements in the figures may have been adjusted to facilitate understanding, and the operations of the embodiments associated with the flow diagrams may encompass additional, alternative, or fewer operations than those depicted here. Thus, some components and / or operations may be separated into different blocks or combined into a single block in a manner other than as depicted. The embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosed examples, rather than limit the embodiments to the particular examples described or depicted.DETAILED DESCRIPTIONExample Surgical Theaters Overview

[0038] FIG. 1A is a schematic view of various elements appearing in a surgical theater 100a during a surgical operation as may occur in relation to some embodiments. Particularly, FIG. 1A depicts a non-robotic surgical theater 100a, wherein a patient-side surgeon 105a performs an operation upon a patient 120 with the assistance of one or more assisting members 105b, who may themselves be surgeons, physician’s assistants, nurses, technicians, etc. The surgeon 105a may perform the operation using a variety of tools, e.g., a visualization tool 110b such as a laparoscopic ultrasound, visual image acquiring endoscope, etc., and a mechanical instrument 110a such as scissors, retractors, a dissector, etc.

[0039] The visualization tool 110b provides the surgeon 105a with an interior view of the patient 120, e.g., by displaying visualization output from an imaging device mechanically and electrically coupled with the visualization tool 110b. The surgeon may view the visualization output, e.g., through an eyepiece coupled with visualization tool 110b or upon a display 125 configured to receive the visualization output. For example, where the visualization tool 110b is a visual image acquiring endoscope, the visualization output may be a color or grayscale image. Display 125 may allow assisting member 105b to monitor surgeon 105a’s progress during the surgery. The visualization output from visualization tool 110b may be recorded and stored for future review, e.g., using hardware or software on the visualization tool 110b itself, capturing the visualization output in parallel as it is provided to display 125, or capturing the output from display 125 once it appears on-screen, etc. While two-dimensional video capture with visualization tool 110b may be discussed extensively herein, as when visualization tool 110b is a visual image endoscope, one will appreciate that, in some embodiments, visualization tool 110b may capture depth data instead of, or in addition to, two-dimensional image data (e.g., with a laser rangefinder, stereoscopy, etc.).

[0040] A single surgery may include the performance of several tasks, which may themselves include one or more actions. For example, locating a tumor may constitute a first task, excising the tumor a second task, and closing the surgery site a third task. Each task may include multiple actions, e.g., a tumor excision task may require several cutting actions and several cauterization actions. While some surgeries require that tasks assume a specific order (e.g., excision occurs before closure), the order and presence ofsome tasks in some surgeries may be allowed to vary (e g., the elimination of a precautionary task or a reordering of excision tasks where the order has no effect). Transitioning between tasks may require the surgeon 105a to remove tools from the patient, replace tools with different tools, or introduce new tools. Some tasks may require that the visualization tool 110b be removed and repositioned relative to its position in a previous task. While some assisting members 105b may assist with surgery-related tasks, such as administering anesthesia 115 to the patient 120, assisting members 105b may also assist with these task transitions, e.g., anticipating the need for a new tool 110c.

[0041] Advances in technology have enabled procedures such as that depicted in FIG. 1A to also be performed with robotic systems, as well as the performance of procedures unable to be performed in non-robotic surgical theater 100a. Specifically, FIG. 1 B is a schematic view of various elements appearing in a surgical theater 100b during a surgical operation employing a robotic surgical system, such as a da Vinci™ surgical system, as may occur in relation to some embodiments. Here, patient side cart 130 having tools 140a, 140b, 140c, and 140d attached to each of a plurality of arms 135a, 135b, 135c, and 135d, respectively, may take the position of patient-side surgeon 105a. As before, one or more of tools 140a, 140b, 140c, and 140d may include a visualization tool (here visualization tool 140d), such as a visual image endoscope, laparoscopic ultrasound, etc. An operator 105c, who may be a surgeon, may view the output of visualization tool 140d through a display 160a upon a surgeon console 155. By manipulating a hand-held input mechanism 160b and pedals 160c, the operator 105c may remotely communicate with tools 140a-d on patient side cart 130 so as to perform the surgical procedure on patient 120. Indeed, the operator 105c may or may not be in the same physical location as patient side cart 130 and patient 120 since the communication between surgeon console 155 and patient side cart 130 may occur across a telecommunication network in some embodiments. An electronics / control console 145 may also include a display 150 depicting patient vitals and / or the output of visualization tool 140d.

[0042] Similar to the task transitions of non-robotic surgical theater 100a, the surgical operation of theater 100b may require that tools 140a-d, including the visualization tool 140d, be removed or replaced for various tasks as well as new tools,e.g., new tool 165, be introduced. As before, one or more assisting members 105d may now anticipate such changes, working with operator 105c to make any necessary adjustments as the surgery progresses.

[0043] Also similar to the non-robotic surgical theater 100a, the output from the visualization tool 140d may here be recorded, e.g., at patient side cart 130, surgeon console 155, from display 150, etc. While some tools 110a, 110b, 110c in non-robotic surgical theater 100a may record additional data, such as temperature, motion, conductivity, energy levels, etc., the presence of surgeon console 155 and patient side cart 130 in theater 100b may facilitate the recordation of considerably more data than is only output from the visualization tool 140d. For example, operator 105c’s manipulation of hand-held input mechanism 160b, activation of pedals 160c, eye movement with respect to display 160a, etc., may all be recorded. Similarly, patient side cart 130 may record tool activations (e.g., the application of radiative energy, closing of scissors, etc.), movement of instruments, etc., throughout the surgery. In some embodiments, the data may have been recorded using an in-theater recording device, which may capture and store sensor data locally or at a networked location (e.g., software, firmware, or hardware configured to record surgeon kinematics data, console kinematics data, instrument kinematics data, system events data, patient state data, etc., during the surgery).

[0044] Within each of theaters 100a, 100b, or in network communication with the theaters from an external location, may be computer systems 190a and 190b, respectively (in some embodiments, computer system 190b may be integrated with the robotic surgical system, rather than serving as a standalone workstation). As will be discussed in greater detail herein, the computer systems 190a and 190b may facilitate, e.g., data collection, data processing, etc.

[0045] Similarly, many of theaters 100a, 100b may include sensors placed around the theater, such as sensors 170a and 170c, respectively, configured to record activity within the surgical theater from the perspectives of their respective fields of view 170b and 170d. Sensors 170a and 170c may be, e.g., visual image sensors (e.g., color or grayscale image sensors), depth-acquiring sensors (e.g., via stereoscopically acquired visual image pairs, via time-of-flight with a laser rangefinder, structural light, etc.), or acombination of visual image and depth-acquiring sensors (e.g., red green blue depth RGB-D sensors). In some embodiments, sensors 170a and 170c may also include audio acquisition sensors or sensors specifically dedicated to audio acquisition may be placed around the theater. A plurality of such sensors may be placed within theaters 100a, 100b, possibly with overlapping fields of view and sensing range, to achieve a more holistic assessment of the surgery. For example, depth-acquiring sensors may be strategically placed around the theater so that their resulting depth frames at each moment may be consolidated into a single three-dimensional virtual element model depicting objects in the surgical theater. Similarly, sensors may be strategically placed in the theater to focus upon regions of interest. For example, sensors may be attached to display 125, display 150, or patient side cart 130 with fields of view focusing upon the patient 120’s surgical site, attached to the walls or ceiling, etc. Similarly, sensors may be placed upon console 155 to monitor the operator 105c. Sensors may likewise be placed upon movable platforms specifically designed to facilitate orienting of the sensors in various poses within the theater.

[0046] For clarity, as used herein, a “pose” refers to the translational position and rotational orientation of a body. For example, in a three-dimensional space, one may represent a pose with six total degrees of freedom. One will readily appreciate that poses may be represented using a variety of data structures, e.g., with matrices, with quaternions, with vectors, with combinations thereof, etc. Thus, in some situations, when there is no rotation, a pose may comprise only a translational component. Conversely, when there is no translation, a pose may comprise only a rotational component.

[0047] Similarly, for clarity, “theater-wide” sensor data refers herein to data acquired from one or more sensors configured to monitor a specific region of the theater (the region encompassing all, or a portion, of the theater) exterior to the patient, to personnel, to equipment, or to any other objects in the theater, such that the sensor can perceive the presence within, or passage through, at least a portion of the region of the patient, personnel, equipment, or other objects, throughout the surgery. Sensors so configured to collect such “theater-wide” data are referred to herein as “theater-wide sensors.” For clarity, one will appreciate that the specific region need not be rigidly fixed throughout the procedure, as, e.g., some sensors may cyclically pan their field of view so as to augmentthe size of the specific region, even though this may result in temporal lacunae for portions of the region in the sensor’s data (lacunae which may be remedied by the coordinated panning or fields of view of other nearby sensors). Similarly, in some cases, personnel or robotics systems may be able to relocate theater-wide sensors, changing the specific region, throughout the procedure, e.g., to better capture different tasks. Accordingly, sensors 170a and 170c are theater-wide sensors configured to produce theater-wide data. “Visualization data” refers herein to visual image or depth image data captured from a sensor. Thus, visualization data may or may not be theater-wide data. For example, visualization data captured at sensors 170a and 170c is theater-wide data, whereas visualization data captured via visualization tool 140d would not be theater-wide data (for at least the reason that the data is not exterior to the patient).Example Theater-Wide Sensor Topologies

[0048] For further clarity regarding theater-wide sensor deployment, FIG. 2A is a schematic depth map rendering from an example theater-wide sensor perspective 205 as may be used in some embodiments. Specifically, this example depicts depth values corresponding to an electronics / control console 205a (e.g., the electronics / control console 145) and a nearby tray 205b, and cabinet 205c. Also within the field of view are depth values associated with a first technician 205d, presently adjusting a robotic arm (associated with depth values 205f) upon a robotic surgical system (associated with depth values 205e). Team members, with corresponding depth values 205g, 205h, and 205i, likewise appear in the field of view, as does a portion of the surgical table 205j. Depth values 205I corresponding to a movable dolly and a boom with a lighting system’s depth values 205k also appear within the field of view.

[0049] The theater-wide sensor capturing the perspective 205 may be only one of several sensors placed throughout the theater. For example, FIG. 2B is a schematic top- down view of objects in the theater at a given moment during the surgical operation. Specifically, the perspective 205 may have been captured via a theater-wide sensor 220a with corresponding field of view 225a. Thus, for clarity, cabinet depth values 205c may correspond to cabinet 210c, electronics / control console depth values 205a may correspond to electronics / control console 210a, and tray depth values 205b maycorrespond to tray 210b. Robotic system 21 Oe may correspond to depth values 205e, and each of the individual team members 210d, 210g, 210h, and 21 Oi may correspond to depth values 205d, 205g, 205h, and 205i, respectively. Similarly, dolly 2101 may correspond to depth values 2051. Depth values 205j may correspond to table 21 Oj (with an outline of a patient shown here for clarity, though the patient has not yet been placed upon the table corresponding to depth values 205j in the example perspective 205). A top-down representation of the boom corresponding to depth values 205k is not shown for clarity, though one will appreciate that the boom may likewise be considered in various embodiments.

[0050] As indicated, each of the sensors 220a, 220b, 220c is associated with different fields of view 225a, 225b, and 225c, respectively. The fields of view 225a-c may sometimes have complementary characters, providing different perspectives of the same object, or providing a view of an object from one perspective when it is outside, or occluded within, another perspective. Complementarity between the perspectives may be dynamic both spatially and temporally. Such dynamic character may result from movement of an object being tracked, but also from movement of intervening occluding objects (and, in some cases, movement of the sensors themselves). For example, at the moment depicted in FIGs. 2A and 2B, the field of view 225a has only a limited view of the table 210j, as the electronics / control console 210a substantially occludes that portion of the field of view 225a. Consequently, in the depicted moment, the field of view 225b is better able to view the surgical table 21 Oj. However, neither field of view 225b nor 225a has an adequate view of the operator 21 On in console 210k. To observe the operator 210n (e.g., when they remove their head in accordance with “head out” events), field of view 225c may be more suitable. However, over the course of the data capture, these complementary relationships may change. For example, before the procedure begins, electronics / control console 210a may be removed and the robotic system 21 Oe moved into the position 210m. In this configuration, field of view 225a may instead be much better suited for viewing the patient table 21 Oj than the field of view 225b. As another example, movement of the console 210k to the presently depicted pose of electronics / control console 210a may render field of view 225a more suitable for viewing operator 210n, than field of view 225c. Suitability of a field of view may thus depend uponthe number and duration of occlusions, quality of the field of view (e.g., how close the object of interest is to the sensor), and movement of the object of interest within the theater. Such changes may be transitory and short in duration, as when a team member moving in the theater briefly occludes a sensor, or they may be chronic or sustained, as when equipment is moved into a fixed position throughout the duration of the procedure.

[0051] As mentioned, the theater-wide sensors may take a variety of forms and may, e.g., be configured to acquire visual image data, depth data, both visual and depth data, etc. One will appreciate that visual and depth image captures may likewise take on a variety of forms, e.g., to afford increased visibility of different portions of the theater. For example, FIG. 2C is a pair of images 250b, 255b depicting a grid-like pattern of orthogonal rows and columns in perspective, as captured from a theater-wide sensor having a rectilinear view and a theater-wide sensor having a fisheye view, respectively. More specifically, some theater-wide sensors may capture rectilinear visual images or rectilinear depth frames, e.g., via appropriate lenses, post-processing, combinations of lenses and post-processing, etc. while other theater-wide sensors may instead, e.g., acquire fisheye or distorted visual images or rectilinear depth frames, via appropriate lenses, post-processing, combinations of lenses and post-processing, etc. For clarity, image 250b depicts a checkboard pattern in perspective from a rectilinear theater wide sensor. Accordingly, the orthogonal rows and columns 250a shown here in perspective, retain linear relations with their vanishing points. In contrast, image 255b depicts the same checkboard pattern in the same perspective, but from a fish-eye theater-wise sensor perspective. Accordingly, the orthogonal rows and columns 255a, while in reality retaining a linear relationship with their vanishing points (as they appear in image 250b) appear here from the sensor data as having curved relations with their vanishing points. Thus, each type of sensor, and other sensor types, may be used alone, or in some instances, in combination, in connection with various embodiments.

[0052] Similarly, one will appreciate that not all sensors may acquire perfectly rectilinear, fisheye, or other desired mappings. Accordingly, checkered patterns, or other calibration fiducials (such as known shapes for depth systems), may facilitate determination of a given theater-wide sensor’s intrinsic parameters. For example, the focal point of the fisheye lens, and other details of the theater-wide sensor (principalpoints, distortion coefficients, etc.), may vary between devices and even across the same device over time. Thus, it may be necessary to recalibrate various processing methods for the particular device at issue, anticipating the device variation when training and configuring a system for machine learning tasks. Additionally, one will appreciate that the rectilinear view may be achieved by undistorting the fisheye view once the intrinsic parameters of the camera are known (which may be useful, e.g., to normalize disparate sensor systems to a similar form recognized by a machine learning architecture). Thus, while a fisheye view may allow the system and users to more readily perceive a wider field of view than in the case of the rectilinear perspective, when a processing system is considering data from some sensors acquiring undistorted perspectives and other sensors acquiring distorted perspectives, the differing perspectives may be normalized to a common perspective form (e.g., mapping all the rectilinear data to a fisheye representation or vice versa).Operational and Nonoperational Theater Data Overview

[0053] FIG. 3 is a schematic depiction of data that may be acquired during a surgically operative period, as may occur in connection with some embodiments. Specifically, over time 305, e.g., throughout the course of a day, an operating room, whether the robotic theater 100a or non-robotic theater 100b, may alternate between operative periods of surgical activity and nonoperative periods when surgery is not performed. In this example, during an initial preparatory pre-surgical period 310a at the beginning of the day, team members may prepare the room for the day’s surgeries. A first surgery may then be performed during the interval 315a by the same or different team members. An inter-surgical period 310b may then follow, during which the team (which may or may not include the same composition of team members as in the previous periods) prepares the theater for the next surgery, e.g., changing equipment configurations, wheeling patients in and out of the theater, etc. The alternating operative and nonoperative periods may then continue throughout the day, e.g., as reflected by surgical period 315b, the Nthsurgery 315c, and intervening ellipsis 310e (which may indicate the presence of additional surgical procedure intervals). At the day’s end, during a post-surgical period 31 Od, team members may, e.g., clean the theater, put awayequipment, and ensure that requisite items are available and in order for the following day’s surgeries, etc.

[0054] During each of the surgical periods 315a-c, theater operations may generally produce data divided into two parallel groups: surgical equipment acquired data 375a and theater-wide data 375b. Each of these datasets may correspond to the performance of respective tasks. For example, the surgical equipment acquired data 375a, which, as mentioned need not necessarily be from a robotic system, may be acquired in connection with surgical tasks 370a, 370b, 370c, and 370d (intervening ellipsis 370e indicating the possibility of additional intervening tasks). As shown, each of tasks 370a-d may be associated with corresponding surgical data, such as visual image video data 320a-d (in some situations, depth video data may also, or alternatively, be available). Similarly, kinematics data 325a-d may be acquired from a robotic platform, a surgeon console, the instruments themselves, etc. Such data may indicate, e.g., the motion and poses of various instruments, end effectors, tools, etc., throughout each respective task. Similarly, system events data 335a-d may be acquired in connection with each task, indicating when instruments are activated (e.g., an endoscope position lock, a cauterization tool, surgical scissor activation, etc.). For clarity, ellipses 320e, 325e, and 335e again indicate the possibility of additional intervening tasks and corresponding data.

[0055] Thus, surgical data 375a generally corresponds to data generated by the actions of one or more surgeons. In contrast, the theater-wide data 375b may be acquired from one or more theater-wide sensors placed around the theater and may generally depict actions by team members within the theater, particularly in the performance of theater tasks 330a-d (ellipsis 330e indicating the possibility of additional tasks). As the tasks are different, though sometimes related, tasks 330a-d and 370a-d. In this example, there are three data streams 340a-d, 350a-d, and 360a-d (ellipses 380 indicating the possibility of additional data streams in some embodiments from other sensors, though one will appreciate that fewer than three streams may also appear in some embodiments) corresponding to three different theater wide sensor systems having different corresponding poses within the theater (ellipses 340e, 350e, and 360e again indicating the possibility of additional datasets for each respective row of stream data). Though streams 340a-d, 350a-d, and 360a-d are shown here as visual image data, as previouslydiscussed, one will appreciate that the streams may additionally, or alternatively, include, e.g., depth data.

[0056] While many of tasks 330a-d are performed in connection with or in anticipation of tasks 370a-d (though, naturally, not all are), tasks 330a-d performed in the theater may or may or not temporally correspond with the surgical tasks 370a-d, may be different in number and different in their start and stop times. That is, though each of the surgical data 375a and theater-wide data 375b may be acquired relative to a common time keeping device, the start and end times of various tasks 330a-d and 370a-d may not correspond. For example, here, the surgical task 370c may require an imaging system. Accordingly, as indicated in the video data images 340b for task 330b, a team member has begun to move the imaging system into position for task 370c. Accordingly, despite their relation, the start and end times of the task 330b may occur well before task 370c. Thus, the start and end times, duration, and number of tasks 330a-d and tasks 370a-d, while sometimes correlated, may often differ. Finally, for clarity, note that, while shown here as linearly succeeding one another in time, one will appreciate that some tasks may proceed in parallel, e.g., where multiple team members are simultaneously performing their own role-specific tasks within the theater.

[0057] FIG. 4 is a schematic depiction of data that may be acquired during the surgically nonoperative period 310c. Here, the three data streams are still active, acquiring the respective datasets 425a-e, 430a-e, and 435a-e during the performance of various inter-surgical tasks 420a-e (again, each of ellipses 420f, 425f, 430f, and 435f indicating the possibility of additional intervening tasks and datasets).Example Surgical Task Overlap Model Representation

[0058] FIG. 5 is a schematic block diagram illustrating example actions and role assignments in a TOM, as may be used in connection with some embodiments. Generally, in this example, a surgery may be divided into five divisions: “SET UP”, “PREP”, “SURGERY”, “POST-SURGERY” and “FINALIZE.” One will appreciate different divisions for different surgeries or inter-surgical periods. Within each division are theaterlevel tasks (e.g., “team arrives” and “set up” within the “SET UP” division) with one or more collections of action groups (e.g., groups 505a, 510a, etc.). As indicated, each ofthe action groups is assigned to one of the team members. Ellipses within various of the action groups indicate that the action group may have more actions than are shown in the confines of this FIG. 5. Actions associated with the ellipses are accordingly instead recited and clarified in this text. Similarly, each action may be associated with a recommended time, which, in the case of a printout, may be presented to the team member with the expectation that they will time their own actions accordingly. For clarity, one will appreciate that the terms “task”, “action,” and “task action” are used interchangeably herein when referring to an activity assigned by the TOM to a role, as one will appreciate that an assigned task may encompass one or more actions in the theater and an assigned action may itself encompass one or more constituent actions in the theater (e.g., the action “clean equipment” may include the cleaning of multiple pieces of equipment).

[0059] A TOM, such as that shown in FIG. 5, when rendered as a printout, can be cognitively taxing upon the reviewing team member. Specifically, it is not necessarily the case that actions in a group must be always completed in a specific order, or that actions in a future theater-level task cannot be worked upon until all the actions by all the team members in the current theater-level task are complete. However, such dependency relations between actions within a role or between roles, or the absence of such dependency relations, is not explicitly presented in the TOM printout. Rather, the static TOM imposes the onus upon the team members to recognize (and, typically, to memorize) such implicit relations and to act accordingly.

[0060] Indeed, separating action groups into the vertical columns of each theaterlevel task may be the only crude means for indicating to the team members that various actions are expected to be performed relatively contemporaneously with one another between the various roles. This undesirably prevents members who have completed their tasks from taking advantage of their efficiency, instead requiring the team to proceed in “lock step” from theater-level task to theater-level task. The onus is placed upon the team members to appreciate the existence of “implied” conditions between the actions of the respective roles, or the absence thereof, and to proceed efficiently in view of these unstated constraint topologies. This is often an unreasonable expectation of teammembers already cognitively taxed within the dynamic, time-sensitive, and stressful environment of the surgical theater.

[0061] Accordingly, as will be described in greater detail herein, various embodiments remedy these defects by translating the static TOM representation, in conjunction with various features disclosed herein, to a dynamic form. In some embodiments, such translation may enforce atomicity requirements upon each action as well as make explicit conditional dependency relations between the actions. Atomicity here refers to actions being defined such that once a team member begins the action, they know that they may complete the action without worrying about any blocking dependency relation from other of the team members’ actions. Similarly, while failure to complete an action may not mean that it must be restarted from the beginning, atomicity, as used herein, will dictate that other actions, dependent upon this action, may not begin until this action is entirely complete. Atomicity may also facilitate action recognition categories more amenable to machine learning recognition from the theater-wide data. In this manner, the TOM may be represented in a form suitable for monitoring by computer logic or machine learning systems, facilitating adaptive, optimized feedback, responsive to the theater’s dynamic character. Beginning with a static template, like FIG. 5, some embodiments may create a new representation amenable to real-time monitoring and adjustment. Accordingly, for completeness, this section walks through the details of FIG. 5’s example TOM so as to provide the reader with orienting context for future discussion. One will appreciate that for simplicity or to accommodate atomicity, various distinct actions discussed in this example static TOM may be grouped into a single action in the dynamic representation. Conversely, individual actions in a static TOM may be split apart into smaller units, e.g., to accommodate the atomicity described above.

[0062] Here, as mentioned, the division “SET UP” includes the action group collections: “Team arrives” and “Set Up.” “Team arrives” corresponds to the team’s initial actions within the theater. “Set up” refers to the group of actions, which will prepare the room for surgery. Within “team arrives”, action group 505a indicates that the circulator is to arrange the room, check the equipment, pair the operating bed with various communicating devices, check the patient, and perform the instrument count. Action group 510a indicates that the scrub tech should scrub in. Action group 515a indicatesthat the second scrub tech / assistant should check the preference card. Action groups 520a and 525a similarly indicate that the anesthesiologist and surgeon, respectively, should check the patient. For clarity, some actions may be performed by one or more team members, and so completion by one member, may negate the need for completion by the other (similarly, such actions may be completed by the members working together or by one member in isolation). Conversely, some actions, while sharing common names and even purposes, may require role-specific completion. Here, for example, each of the “see patient” actions of groups 520a and 525a are constrained to their roles, and so the anesthesiologist’s completion of their action will not absolve the surgeon of likewise performing their own “see patient” (in contrast, e.g., to actions such as “dock robot”, where completion by any team member will satisfy that action for all other team members).

[0063] Action group 505b indicates that the circulator is to transport the patient into the theater. Action group 510b indicates that the scrub tech should set up their assigned portions of the theater and drape the robotic surgical system. Action group 515b indicates that the second scrub tech I assistant should likewise set up their portion of the theater. Action group 520b indicates that the anesthesiologist should set up their portion of the theater. Action group 525b indicates that the surgeon should arrive into the theater, check in with the OR staff, determine case specific needs, and check the robotic surgeon console (though, again, one will appreciate application of various disclosed embodiments in non-robotic theaters).

[0064] The division “PREP” in this example includes the following action group collections: “Patient in room”, “Prep Completed”, and “Ports Placed.” “Patient in room” here refers to actions such as bringing the patient into the room and preparing the theater for the surgery. “Prep Completed” refers to actions which will complete such preparations and “Ports placed” refers to actions generally associated with the placement of the ports. As explained, while these relatively simplistic groupings of actions may be cognitively satisfying in a static TOM, various dynamic TOM structures disclosed herein need not be so limited (e.g., a circulator having completed all of their actions in the group 505d may be directed to proceed to one or more actions in group 505e, even though other members are still performing actions in the “Prep Completed” collection). Dynamic TOMs andaccompanying systems disclosed herein accordingly afford many opportunities for capturing efficiencies that would otherwise be lost in the static approach.

[0065] Action group 505c indicates that the circulator is to perform the “time in” action (referring to the time when the patient has entered to the OR, the time of which may be recorded by the circulator, and upon which many other actions may depend), alert the surgeon of their presence, position the equipment and patient, brief the other team members, assist anesthesia, prepare a foley, and perform skin preparation. Action group 510c indicates that the scrub tech should continue setup and confirm that the 20cc syringe is available for the robotic surgical system’s priming (again, though this example is for a robotic surgical system, various of the disclosed embodiments may be readily applied to non-robotic theaters). Action group 515c indicates that the second scrub tech I assistant should assist the circulator with the patient and assist the first scrub tech. Action group 520c indicates that the anesthesiologist should begin patient intubation and action group 525c indicates that the surgeon should position the patient.

[0066] Action group 505d indicates that the circulator is to connect cords, position the robotic surgical system, and time out. Timing out may refer to a period before a start of the surgery where all care team members gather and agree upon the particulars of the operation (e.g., that they have the correct patient, the correct planned operation upon the correct anatomy, etc.) so as to prevent any errors. This, and similar coordinating actions, may operate as “semaphores” to gatekeep team members from proceeding too quickly relative to other members (e.g., serving to briefly align their parallel actions). Accordingly, customarily, the “time out” action will involve all present members of the care team. While the requirement that all members perform the time out together may also be enforced in various embodiments disclosed herein, as will be described, some embodiments may instead apply time outs, and similar semaphore actions, to groups of less than all of the team members, or to individual team members, and possibly at different times, when it is possible for verification to be made without all team members doing so at once. For example, each team member may verify the appropriateness of the procedure, such as the choice of patient, anatomy, and procedure, when their action dependencies first permit them to do so, rather than waiting for other members to “catch up” so that they might verify together. In these embodiments, the system may check that each of theseparate verifications are in agreement before permitting surgery-related actions, and notify the team if one or more team members provided a verification inconsistent with the other team members’ verifications or which is inconsistent with the surgical plan.

[0067] Continuing the static TOM summary, action group 51 Od indicates that the scrub tech should drape the patient, gown the surgeon(s), pass cords, and enter the time out. Action group 515d indicates that the second scrub tech / assistant should scrub in, pass cords, end the time out, and place ports. Action group 520d indicates that the anesthesiologist should position the table and enter the time out. Action group 525d indicates that the surgeon should scrub in, enter the time out, and place ports (naturally, the actions need not necessarily be completed in an order depicted here within a group, another ambiguity which may be cognitively taxing upon the team member).

[0068] Action group 505e indicates that the circulator is to drive in the robotic surgical system, adjust equipment and secure cords. Action group 510e indicates that the scrub tech should assist with the instruments and organize the sterile field. Action group 515e indicates that the second scrub tech I assistant should dock the robotic surgical system and insert instruments. Action group 520e indicates that the anesthesiologist should monitor the patient and that this action shall continue throughout the next two collections. As will be discussed, various of the disclosed dynamic approaches may obviate the need for atypical representations like group 520e, which risk confusing various members, and as here, may employ a brevity in their description that would be better replaced with more granular instructions and sub-actions. Action group 525e indicates that the surgeon should perform docking, targeting, and insert the instruments.

[0069] The division “SURGERY” in this example includes only the “console time” collection of action groups. Action group 505f indicates that the circulator is to perform charting and check the next case chart as time allows (again, a somewhat ambiguous license). Action group 51 Of indicates that the scrub tech should check the patient, ports, and robotic surgical system arm clearance, as well as pass sutures and needles. The scrub tech should also keep track of the countables (instruments, clamps, temporary markers, etc., that should not remain within the patient following the surgery) inside thepatient. Action group 515f indicates that the second scrub tech / assistant should assist the other team members. Note that the statement “assist other team members” is laconic and vague, placing the onus upon the second scrub tech to recognize what peer actions are incomplete with which they may assist (and additionally resulting in variation of interpretation across teams and theaters, which may complicate transitioning team members between teams). In contrast, various of the disclosed dynamic approaches provide specific action guidance to the second scrub tech, directing them to the particular peer action with which to assist, e.g., in accordance with priorities disclosed herein. For example, various embodiments would replace this action with more specific instructions, e.g., assigning the second scrub tech I assistant to the same action as the outstanding action with which they should be assisting. Action group 525f indicates that the surgeon should perform the operation at the console.

[0070] The division “Post-Surgery” in this example includes only the “surgeon off console” collection of action groups. Action group 505g indicates that the circulator is to drive out and undrape the robotic surgical system, disconnect cords, turn off insufflation, debrief with the team, complete the robotic surgical system post-case debrief, count, and check of the instrument lives. Action group 510g indicates that the scrub tech should set up the Mayo™ closing suture, dressings, and water basin, as well as sign out and prime the robotic surgical system’s instruments and imaging tools. Action group 515g indicates that the second scrub tech I assistant should retrieve specimens, undock the robotic surgical system, and close the patient. Action group 525g indicates that the surgeon should scrub, close the patient, and confirm the plan of the next case (if applicable). Again, this TOM is vague in indicating whether the close of group 515g is the same as that of group 525g, whereas various disclosed dynamic embodiments may provide specific guidance to ensure each team member approaches that action simultaneously, albeit atomically, if that is the intention.

[0071] A “return to room time out announcement” may occur at the time between the “Post-Surgery” and “Finalize” divisions, again serving as a semaphoric division of team member actions. The division “FINALIZE” in this example includes only the “incision closure” collection of action groups. After the timeout, during the “finalize” division, action group 505h indicates that the circulator is to call the Post Anesthesia Care Unit (PACU),page the room turnover, and transport the patient. Action group 51 Oh indicates that the scrub tech should spray the instruments, move out the case chart, and assist with turnover. Action group 515h indicates that the second scrub tech / assistant should assist with turnover (again, such a vague statement placing the onus on the second scrub tech to infer where their assistance would be best utilized). Action group 520g indicates that the anesthesiologist should transport the patient. Finally, action group 525h indicates that the surgeon should complete the post-surgical paperwork and see the next patient.

[0072] In TOM printouts, time goals may be generally indicated to the team in connection with the TOM, e.g., via text printed above or below a columnar chart as in FIG. 5. Goal indications in such printouts may, confusingly, stretch across divisions or across action group collections, making reference to individual actions in disparate groups and across different roles. Not only is it difficult to abide by such constraints with a paper TOM, but it can also be especially difficult for team members or reviewers to verify compliance with the paper TOM. For example, turnover time in the “SET UP” division may be associated with 30 minute goal (from wheels out of the previous patient to wheels in of the new patient). The beginning of the “PREP” division until the first patient positioning action may be associated with a 20 minute goal. Similarly, the time from that patient positioning to the first incision may be associated with a 20 minute goal and from undocking of the robotic surgical system to the last closing incision may be associated with a 30 minute goal. An incision close to out-of-room goal of 10 minutes during the “finalize” division, may also be stated upon the printout. While they may be more helpful, more granular guidance than these temporal intervals is generally infeasible with the static TOM.

[0073] As indicated by this example, the TOM can be incredibly useful for ensuring theater efficiency, that appropriate actions are taken, and that failure cascades are avoided. Unfortunately, for static, e.g., paper systems, the TOM cannot be adapted to changing circumstances, which leaves team members to “guesstimate” how the original TOM would be modified to their current situation, often provides vague or incomplete instructions, and offers a poor medium for enforcing or monitoring performance time goals.

[0074] Thus, while perhaps suitable for an internal computer system representation, or for an offline reviewer, presenting the entirety of the TOM as in FIG. 5 during a surgery undesirably imposes upon each team member’s cognitive load, asking the member to identify the row and column relevant to them at a given moment, while remaining mindful of their history of their past actions to do so, and all while constantly consulting a timekeeping device to assess their progress relative to the recommended time in the printout.Example Dynamic TOM System Component Overview

[0075] Various embodiments include computer systems configured to process theater-wide data so as to monitor progress in the TOM and to provide appropriate corresponding real-time feedback to the team, thereby addressing many of the problems identified above. These embodiments provide meaningful TOM-based guidance and less cognitively taxing display of that guidance. Some of these embodiments employ machine learning analysis of theater-wide data and / or adaptive logic to accomplish these goals. For example, FIG. 6 is a schematic block diagram illustrating an example component processing topology relationship as may be implemented in connection with some embodiments. Specifically, in some embodiments, the dynamic TOM management system 605 may be implemented upon one of computer systems 190a and 190b, at an offsite server, upon a handheld device of a team member, etc.

[0076] The TOM management system 605 may include several machine learning systems 615 configured to perform various recognition tasks from real-time theater-wide sensor data 625 acquired from the theater. For example, a personnel detection machine learning system 615a may process 620a the real-time theater-wide data to infer the number, location, and roles of the present personnel. Similarly, an activity recognition machine learning system 615b may process 620b the theater-wide data 625 to recognize the one or more actions being presently performed by the various team members (while also suitable for detecting completion of actions, in some embodiments, completion may also be specified manually by the users, e.g., after they collectively complete a time out period, by hand input selecting of a computer interface, selection from a GUI, by recognized hand or vocal auditory gesture, etc.). Machine learning systems 615 mayalso include systems configured to perform tracking of the personnel, e.g., to monitor adherence to the TOMs. Thus, the system 605 may use multiple machine learning models and fuse multiple sources of data in the performance of its analysis. Some embodiments may implement machine learning systems 615a and 615b using transformer architectures and methods analogous to those discussed, e.g., in Bertasius, Gedas, Heng Wang, and Lorenzo Torresani. "Is Space-Time Attention All You Need for Video Understanding?" arXiv™ preprint arXivTM:2102.05095 (2021 ). While a modified You Only Look Once (YOLO) architecture (e.g., as described in Redmon, Joseph, et al. "You Only Look Once: Unified, Real-Time Object Detection." arXiv™ preprint arXiv™: 1506.02640 (2015)) may be appropriate in some circumstances for personnel detection, architectures and methods suitable for personnel detection via system 615a may also include transformer methods, e.g., as described in Carion, Nicolas, et al. "End- to-End Object Detection with Transformers." arXiv™ preprint arXiv™ :2005. 12872 (2020). Tracking, which may be used in either of machine learning systems 615a and 615b in some embodiments, may be accomplished, e.g., using methods and architectures analogous to those disclosed in Meinhardt, Tim, et al. "TrackFormer: Multi-Object Tracking with Transformers." arXiv™ preprint arXiv™:2101.02702. One will appreciate that these approaches are merely examples. Similarly, these architectures may be modified to receive visual image and depth sensor theater-wide data (e.g., as discussed herein with respect to FIGs. 1A-B and 2A-C), as well as be modified to receive visual image or depth frame video rather than discrete images or frames.

[0077] Rules and logic 610a may analyze the results of the personnel detection machine learning system 615a and activity recognition machine learning system 615b to adjust an internal representation of the state of the TOM 650. In some embodiments, a machine learning system 615c may be dedicated to recognizing a theater to TOM representation correspondence from the data acquired by the machine learning systems 615a and 615b. However, in some embodiments, the rules and logic 610a may suffice for interpreting and responding to the personnel and activity detection data from machine learning systems 615.

[0078] As will be described in greater detail herein with respect to the examples of FIGs. 8A-B, 9, and 15 rules and logic 610a may manage the internal representation ofthe state of the TOM 650, e.g., as Markov models or as analogous discrete state representations of the theater dynamics. Rules and logic 610a may also interact with a database 605a of historical data (e.g., as part of the performance of operations disclosed herein with respect to FIGs. FIG. 17A-B), both acquiring 605c past case data, e.g., to prepare a default TOM based on past similar surgical procedures (e.g., the same type of procedure), and to update 605b the historical database 605a, e.g., with the results of the current procedure (e.g., durations of various actions so as to inform a subsequently determined historical benchmark). For example, the system may use the machine learning system 615a to determine a current number of personnel in the theater and their roles and prepare a corresponding default template for the anticipated surgery. In some embodiments, the system may look up an appropriate corresponding default template from the database 605a for the anticipated surgical procedure, based upon the detected personnel and equipment, which may be adapted for the detected personnel and equipment (e.g., reassigning tasks across roles where fewer than all the personnel of the default template are present). In some embodiments, rather than generate a TOM for detected personnel, the system may generate a TOM with candidate personnel and equipment for the hospital administration or team members to schedule or approve.

[0079] Rules and logic 610a may also be used to adjust the TOM representation 650 and its renderings 605d in the theater. As discussed herein, the system may also cause the recommended amount of time to complete each task (which may be incrementally updated over performances) to be displayed 605e in renderings 605d (as well as determine incremental performance goals and adherence, as described in greater detail herein with respect to FIGs. 17A-B). If parameters change in the theater (e.g., if personnel enter or leave), then the system may reassign tasks among the presently available capacity of the surgical team members (e.g., using a table lookup of viable actions, by consulting representation 650, by using linear programming methods, etc.).

[0080] At the beginning of the surgery, an original “default” representation of the TOM 650 may be constructed based upon a template tailored to the care team size (e.g., a digitization of one of the static printout TOMs used previously, with atomic actions as discussed herein and modified to reallocate actions among the number of detected personnel). Throughout the surgery, the internal representation of the TOM 650 maychange according to the order in which actions are completed, and as theater parameter changes are detected, e.g., whether the surgeon is present to help with the setup, whether there are one or two scrub techs, whether there is a “floater” team member available to help, etc. These variations in the size of the care team may be detected, e.g., by the personnel detection machine learning system 615a. Upon detection of changes in the theater, the internal representation of the TOM 650 may, if necessary, likewise be adjusted, e.g., reassigning tasks or adjusting recommended task completion times to reflect the change in available working capacity. Initial recommended task completion times for the actions in the TOM may be determined from historical data, present case metadata, surgeon and staff experience levels, available staff, a speed with which previous actions were completed, the speed with which dependent blocking actions of an upcoming action are being completed, etc., e.g., using various of the methods disclosed herein. Thus, the initial recommended time may not always be the “ideal” or “most efficient” time for the action’s performance, but rather, a presently determined realistic goal or level of improvement of the given team (e.g., as described herein with respect to FIGs. 17A-B). Some embodiments may also include an Application Programming Inteface (API) 610b with which developers may access various resources within the system 605, e.g., the current state representation 650, the results of the machine learning systems 615, etc. Such real-time access to state representation 650 may, e.g., facilitate improved scheduling applications in the hospital context.

[0081] Thus, the system 605 may utilize historical case metadata as determined from database 605a and vision-based people detection algorithms, e.g., using the machine learning system 615a, to automatically generate customized TOMs for the care team.Example TOM Dependency Blocks

[0082] By imposing certain conditions upon action and dependency representations within representation 610a, various of the disclosed embodiments overcome various of the problems identified above with respect to static TOMs. One benefit of these representations is the ready identification and processing of action dependency blocks. For clarity, FIG. 7A provides a schematic representation of successive theater stateswhen encountering a dependency block (one will appreciate that not all actions that may apply in a real-world TOM are shown in FIGs. 7A and 7B so as to facilitate the reader’s understanding). As will be discussed in greater detail herein, when a task upon which subsequent tasks depend is incomplete or missed, the task may continue to be assigned to the responsible team member (or another suitable member), so that the task action will be timely completed so as to avoid outstanding dependencies. For instance, an anesthesiologist will not be able to start intubation until the circulator transports the patient. If certain tasks by other team members depend upon this task, those members may be assigned to help complete the current task or other tasks that are not blocked by this task. This reassignment process may be presented via state transitions as described herein.

[0083] In this example, as in FIG. 5, FIG. 7A depicts five rows for five roles and five corresponding team members in the theater (the reader will appreciate that this example is provided to facilitate understanding and that often the care team size will differ from OR to OR and from hospital to hospital). The columns of FIG. 7A correspond to generally successive instants in time (though the depicted temporal intervals of this example should not be construed as referring to equal, or even similar, divisions of time). Thus, at the instant in time corresponding to column 720a, all the team members are completing actions in the “Prep Completed” action group collection of FIG. 5. Specifically, the members are each in their respective “time out” actions of action groups 505d, 51 Od, 515d, 520d, and 515d. As mentioned, some actions may serve as a “semaphore,” waiting until all of the team members are in the same action before allowing them each to proceed to the next action. Here, during this “time out” the team may confer with one another to be sure their expectations are in alignment for the upcoming actions, e.g., the upcoming case. Specifically, the team may pause for a short period before, e.g., incision, so as to confirm that they are, e.g., about to perform the correct procedure, upon the correct portion of the patient’s anatomy, and indeed, upon the correct patient.

[0084] Thus, once in agreement, the team members may proceed to their next action at the time corresponding to column 720b. At this instant, some members remain in the “Prep Completed” action group collection of the TOM (particularly, performing their port placement actions), while others have progressed to actions in the “Ports Placed”action group collection (in accordance with the action groups 505e, 51 Oe, 515e, 520e, and 525e of FIG. 5). Accordingly, column 720b is labeled both “Prep Completed” and “Ports Placed.” In this example, during this period, the circulator and the scrub tech timely complete their tasks “Adjust equipment” (per group 505e) and “Organize sterile field” (per group 510e), respectively. Similarly, the anesthesiologist may have begun their “Monitor patient” action (per group 520e), which is generally independent of other task actions. However, the second scrub technician and surgeon have begun placing ports and, unfortunately, they are taking longer than usual to complete their port placement. By the time of column 720c, it is clear that the second scrub technician and surgeon have exceeded the recommended time for port placement. Surpassing the recommended time for port placement may eventually precipitate a warning, e.g., in a GUI (e.g., as will be described with respect to FIG. 10 herein) or a corresponding auditory warning, as indicated by the message 750a, as the prolonged actions continue into the time of column 720c. As will be discussed in greater detail herein, depending upon the user interface, these notifications may be tailored to only the relevant team members (e.g., those directly affected) or may be broadcast to the entire team.

[0085] The consequences of this delay may be understood by turning briefly to the schematic action dependency diagram of FIG. 7B. Specifically, this figure shows a plurality of task actions 705a-j for four roles and directional arrows 710a-k indicating various dependency relations between the actions (one will appreciate that the figure is schematic for the reader’s understanding and that not all applicable dependencies or actions may be shown, nor may the shown actions or dependencies correspond exactly to their real-world counterparts). As indicated, the circulator’s action “Drive in Robot” 705c requires that the second scrub tech and surgeon first complete their respective port placement actions 705b, 705a (the dilatory actions of columns 720b and 720c). Similarly, neither the second scrub tech nor the surgeon can complete their respective “dock robot” actions 705d, 705g (which, may refer to accomplishing the same task together, or alternatively, within the theater) until the circulator has driven 705c the robotic surgical system up to the patient. After robot docking 705d, 705g the surgeon will need to complete targeting 705i before the instruments may be inserted 705e, 705h (for clarity, one will appreciate that completion of some team member’s actions may precipitate acompletion of other team member’s actions, as when the second scrub technician performs docking 705d on their own, making it unnecessary for the surgeon to likewise performing the docking 705g). Only following instrument insertion 705e, 705h, can the surgeon begin the console action 705j (per action group 525f) and the scrub technician the “check patient, ports, and robot arm clearance” action 705f (per action group 51 Of).

[0086] Thus, as indicated, a number of transitive dependent actions may be required to be completed before actions 705f and 705j may be performed. As a consequence, returning to FIG. 7A, the continuing delayed port placement at the time of column 720c has resulted in both the circulator and the scrub technician becoming undesirably idle (in contrast, the anesthesiologist’s “monitor patient” action, being independent, may proceed unhindered).

[0087] Once port placement is finally complete, per “Drive in Robot” 705c of FIG. 7B, the scrub technician, second scrub technician, and surgeon, may be placed in brief idle states in column 720d. At the time of column 720e, the circulator, second scrub technician, and surgeon may transition to their next tasks, while the scrub technician remains idle. While each of the idle times of columns 720d and 720e may be a natural consequence of the dependency requirements of FIG. 7B, note that the circulator and scrub technician’s idle times of column 720c would not be necessary, but for the delay in port placement.

[0088] Thus, some embodiments, in assessing improper start times, prolonged performance, etc., may consider whether abnormal idle times are present to explain the inefficiency and the dependency relations and action durations that may have precipitated those idle times. Dependencies may thus also help explain the actual causal origin for an inefficiency (e.g., a delayed start time by the scrub technician may be the fault of the second scrub technician and surgeon’s dilatory port placement, rather than any unilateral performance of the scrub technician). From this, the system may infer, e.g., that port placement should be more highly prioritized (receiving prioritized assignment when new members enter the theater, if their roles are amenable to such assignment, and being more urgently subject, e.g., to the improvement methods discussed herein with respectto FIGs. 17A-B) since timely port placement may improve efficiency for a variety of downstream actions.

[0089] At the time of column 720f, the scrub technician may finally have an action without blocking dependencies, here “assist with instruments” (not shown in FIG. 7B, but corresponding to the actions 705e and 705h). Once these actions complete, at the time of column 720g, the team may enter the “Console Time” collection of action groups (including the check patient action 705f and console action 705j), though, per the port placement delay, the start time of these actions may be much delayed from what would be preferred.Example Action Data Structure and Monitoring

[0090] While there exist a number of ways to represent an action state, e.g., as part of the TOM representation 650, including explicit indications of each action’s implicit dependencies (e.g., the unstated relations, sometimes blocking, in FIGs. 7A-B), FIG. 8A provides a schematic block diagram illustrating various components as may be used in a data structure for representing and monitoring action completion by team members in some embodiments. This data structure, when used in connection with atomic action selection, may be particularly effective for facilitating automated TOM management. While representation at the action level will be discussed herein to facilitate the reader’s understanding, other viable representations may be used, such as a vector of current and historical (e.g., completed) action states in combination with a network data structure (e.g., as a dictionary hash map of each action to its zero or more dependent actions) indicating action dependencies. In either case, embodiments translate the implicit conditional dependency relations of the static TOM (e.g., those previously discussed with respect to FIGs. 7A-B), which previously required semantic knowledge on the part of the team members, to a set of explicit conditional dependency relations in connection with actions, which in some embodiments may be atomic as described herein. Again, action atomicity may allow the system to readily adapt when members enter or leave the theater, since completion of the action once begun will not disrupt the dependency relations, nor will abandonment of the action (though, as mentioned, members may resume an abandoned action without “starting the action over”).

[0091] In this example representation, each action may be represented by zero or more conditional relations. Specifically, actions are shown schematically here via circles, e.g., the circle 805f corresponds to a presently considered action represented by the data structure 805 (which may, e.g., be JavaScript Object Notation, a Structured Query Language database table, etc.). The action 805f may be preceded by an action 805g as well as succeeded by another action 805h, as previously described, though naturally the first action for a role will not have a predecessor, nor will the final, or “terminal”, action for the role have a successor. Thus, a dependency relations component 850a, such as an array or hash map, may point to other actions upon which this action 805f conditionally depends. For example, clearly if action 805f has a predecessor action 805g, then action 805f will depend 810a upon action 805g (for clarity, in accordance with their order of performance, dependency arrows, as used herein, point to the action which depends upon the action from which the arrow originates). Similarly, as indicated by dependency arrows 810b, 810c, and 810e (ellipsis 810d indicating the possibility of additional dependencies) associated with the actions 820a, 820b, and 820c, respectively, action 805f may depend upon the prior completion of one or more other actions by the same team member or different team members before it may be begun in accordance with the atomicity completion requirement.

[0092] In some embodiments, the dependency relations field 850a may only document preceding dependencies of action 805f (those actions upon which action 805f depends) as one may determine the actions dependent upon a considered action by reviewing all other actions. However, in some embodiments, in addition for avoiding the need to perform such global searches, knowledge of what other actions are dependent upon the present action may also be useful to know locally. Accordingly, in lieu of canvassing all other action data structures to see if action 805f is one of their dependencies, in some embodiments, the dependency relations field 850a may also document the actions 820d, 820e, and 820f (ellipsis 815d indicating the possibility of additional dependencies) which depend (as indicated by the respective arrows 815a, 815b, 815c, 815e) upon action 805f. Naturally, a successive action 805h, if it exists, may, like the preceding action 805g, also be documented as such within the data structure 805. While such local recordation is redundant from a global perspective, a record ofthese dependency relations may speed processing and facilitate various other analyses discussed herein.

[0093] Additional fields which may reside in each instance of the data structure 805 include a field 805a documenting whether the action 805f is being presently performed and therefore “active.” The field 805b may likewise indicate whether the action has been completed or not. This completion field may then inform when all the dependencies are satisfied such that transition to a new action is appropriate. Metadata fields 805c may be used to indicate associations between action 805f and other TOM parameters (e.g., desired start times). A field 805d may indicate viable roles for performing the action 805f, thus facilitating assignment or reassignment when a team member enters or departs the theater (this field may also indicate whether an action is a “common” action that can be completed by multiple roles or an action constrained to a specific role). A field 805e may indicate a current time spent upon the task when it is active (naturally, one will appreciate variations, as where only a start time is recorded and the duration inferred by consultation with a global clock). The field 805i may indicate a recommended time in which the user should complete the action 805f. Note that the recommended time 805i may include a plurality of recommended times in some embodiments, each associated with a different individual or role. Recommended time 805i may also correspond to an intermediate or final target duration as discussed in greater detail herein with respect to FIGs. 17A-B.

[0094] For further clarity in the reader’s understanding, FIG. 8B is a schematic block diagram illustrating example relations between action data structures as may occur in some embodiments. Here, a first set of actions 870a-e may be generally associated with a first role 865a (ellipses 855a, 860a indicating the possibility of additional preceding and succeeding actions respectively), a second set of actions 880a-e, consecutively dependent per arrows 885a-d, may be generally associated with a second role 865b (ellipses 855b, 860b indicating the possibility of additional preceding and succeeding actions respectively), and a third set of actions 890a-e, consecutively dependent per arrows 895a-d, may be generally associated with a third role 865c (ellipses 855c, 860c indicating the possibility of additional preceding and succeeding actions respectively), ellipsis 865d again indicating the possibility of additional roles and actions.

[0095] As discussed, arrows 875a-d indicate the dependency relations of successive actions performed by a single role (e.g., corresponding to the temporal dependencies 810a and 815a). For example, the action 870c cannot begin until the preceding action 870b has concluded, as indicated by arrow 875b. Similarly, the action 870d cannot begin until the action 870c has concluded, per arrow 875c.

[0096] However, as discussed, some actions may depend upon other actions not within the same role. For example, here, the action 870b also requires that the actions 880a and 890a, occurring in different roles, be completed, as indicated by dependency arrows 860a and 860b, respectively. Accordingly, if the member in role 865a completes the action 870a well in advance of actions 880a and 890a, the team member assuming the role 865a may be obligated to enter an idling period (or be reassigned to assist with another action) until those actions upon which the next action depends are complete (e.g., as discussed herein in connection with the example of FIGs. 7A-B). In some embodiments, the system may first seek to identify any actions the free team member could perform, only placing the team member in the idle state when no such independent and parallel actions are available (again, atomicity in action selection may more readily facilitate opportunities to reallocate resources in this manner than if actions were defined more loosely, permitting assignments for mere partial completion). For example, the system may follow the dependency chains in the structures of FIGs. 8A and 8B to identify actions, which are independent of one another, and then prioritize their completion over the blocked actions.

[0097] Similarly, as indicated, dependencies, in addition to being transitive, may be reciprocal between roles, as the action 870b depends upon completion of the action 880a (via arrow 860a), but the action 880c likewise depends upon completion of the action 870b (via arrow 860c). Similarly, dependencies may be stretched over long periods. Here, e.g., the action 870e depends upon the action 890b (as indicated by arrows 860f), even though they are normally expected to be separated by a long period. Per arrows 860d and 860e, actions 870e and 890e likewise depend upon action 880d. In some embodiments, however, dependency structures (e.g., in conjunction with atomicity conditions imposed upon the actions) may be structured so that actions in the TOM are local and Markovian.

[0098] By selecting dependencies and action definitions facilitating action atomicity, then dependency structures like that in FIG. 8B may be readily handled by various of the monitoring and feedback systems disclosed herein. Additionally, as described in greater detail herein, atomicity may facilitate simplified state representations of a surgical TOM state. For example, in some embodiments, a vector with a history of all completed actions and a current active role assignments (e.g., a one-hot vector representation) would suffice to capture the most salient aspects of any given moment when progressing through the TOM.

[0099] As an example of how the system may avail itself of the atomicity of data structures like that described in FIG. 8A, FIG. 9 is a flow diagram illustrating various operations in an action transition management process 900, as may be implemented in some embodiments. Specifically, at block 905a, the system may initialize all the data structures (e.g., instances of the structure 805) of each action in the TOM representation 650, such as setting each of the completion fields 805b to false. At block 905b, the system may select the initial actions in the TOM representation and set their status to active (e.g., by changing the property 805a from inactive to active).

[0100] At block 910a, the system may seek to verify whether all the actions (regardless of their active status) have been completed (e.g., the parameter 805b is marked complete for all the actions in the TOM representation), implying that the TOM is likewise finished. If incomplete actions remain, then the system may iterate over the active actions (e.g., actions with an property 805a set to active) at blocks 910b and 910c.

[0101] If the presently considered active action is not yet complete at block 915a, then the system may continue to consider the other active actions, as indicated. In contrast, when the considered active action is complete (e.g., based upon activity recognition machine learning system 615b) then the system may change the corresponding data structure’s status to inactive at block 915b. If the action is terminal, i.e. , it has no successive action in the TOM at block 930a, then the system may consider any other active actions as indicated. If the currently considered active action is not terminal, the system may consider whether any of the viable successive actions for the role may be activated at blocks 920a and 920c. Specifically, for each of the potentialsuccessive actions, at blocks 925a and 925c, the system may iterate over the presently considered successor actions’ dependencies to see if they are presently satisfied (e.g., complete) at block 925d. If all the dependencies are satisfied, then the newly considered action’s data structure may be activated at block 925b. In contrast, if all of the potential successive actions have at least one blocking dependency, then the system may present the idle state 920b to the member for the corresponding role until the dependencies are resolved (or, as will be further discussed, identify an alternative action for performance).

[0102] One will appreciate that the example process 900 has been chosen to facilitate the reader’s understanding, rather than to depict the most efficient approach to implementing a TOM update (e.g., a carefully crafted vector or matrix representation, e.g., of action and theater states, particularly where the actions and their dependencies are Markovian, may be much more efficient for performing various of the depicted operations). Accordingly, one will appreciate that process 900 and structure 05 are but one of many possible ways for tracking and updating the TOM, some of which may be much more efficient than this example.Example Graphical User Interface Elements

[0103] As mentioned, various of the disclosed embodiments contemplate a dynamic TOM able to intelligently adapt to the theater’s dynamic environment. To reduce cognitive load upon the team members, some embodiments may present only the relevant portions of the TOM to the team at any given moment in the procedure, or only for individual team members. For example, FIG. 10 is a schematic graphical user interface (GUI) 1005 rendering of a task status display, as may be implemented in some embodiments (e.g., as part of renderings 605d). A title 1005a may notify the viewer of the interface’s purpose, while a metadata region 1005b may provide details regarding the current surgery (or upcoming surgery if presented in a nonoperative period), e.g., the operating surgeon (“Dr. John Smith”), the type of procedure (“inguinal hernia”), and the modality of surgery (robotic rather than non-robotic).

[0104] A current division region 1005c may indicate the present division of the TOM in which the group actions are being performed (as in FIG. 7A, more than one division or action collection may be shown if team members are at different stages of the TOM). Thecare team region 1005d may indicate the number of team members understood to be available as, e.g., manually input, determined automatically from machine learning system analysis of theater-wide data, optical flow analysis of theater wide data, etc. A time elapsed region 1005e may indicate the amount of time that has elapsed for the entire surgical procedure (here, approximately 47 minutes and 15 seconds).

[0105] Below these general context providing regions may be rows associated with the team members and the currently assigned action of their respective active action group. In this example, rows for the circulator 1005i, scrub tech 1005j, first assistant 1005k, anesthesiologist 10051, and surgeon 1005m are provided in accordance with present time for the internal representation of the TOM 650. Here, the circulator’s current task (indicated by column label 1005f) is to perform skin prep, and the circulator has spent a minute and sixteen seconds on that task (indicated by current elapsed time label 1005g), while the recommended amount of time is two minutes (indicated by the recommended time label 1005h). For clarity, one will appreciate that the tasks shown here and values provided for the recommended time label 1005h are primarily provided to facilitate the reader’s understanding and may, or may not, correspond to actual tasks that would be performed in parallel or to actual recommended times for those tasks (e.g., dependency relations may be such that the tasks will not occur in parallel in some situations).

[0106] The scrub tech has been performing the setup task for seven minutes and nine seconds (where 15 minutes are recommended), the first assistant has been performing the positioning task for nine minutes and sixteen seconds (where five are recommended), the anesthesiologist has been performing the intubation task for thirteen minutes and forty eight seconds (where 15 minutes are recommended), and the surgeon has been performing the “position patient” task for five minutes and sixteen seconds. Recommended times rounded to the nearest minute or in discrete intervals may help reduce cognitive load for team members reviewing the interface 1005. Though not shown in this example, in some implementations, the GUI may provide a “role lane”, as in the TOM, indicating what action(s) the relevant team member have completed, what action they are performing now, and what action(s) they are expected to be performing in thefuture. This may be useful for anticipating, recognizing, or understanding dependency blocks from other team members’ actions.

[0107] The system may provide feedback via interface 1005 to the team while monitoring adherence to the TOM. For example, as a recommended time is exceeded, the system may cause appropriate instructions to be displayed so as to assist the team with timely finishing the action. The system may also improve workflow efficiency in accordance with guidelines specified by the team or by outsider reviewers. For example, where a team or team member is consistently taking more time upon an action, or actions, than a historical baseline, then with each subsequent encounter of the delinquent action, or actions, the system may gradually reduce the presented recommended time 1005h to more closely match the desired baseline value (e.g., as descried in greater detail herein with respect to FIGs. 17A-B). As will be discussed in greater detail with respect to FIGs. 17A-B, some actions may recevie higher priorities over other actions (the same or similar priorities may also be used for choosing among multiple possible action assignments for a team member).

[0108] Accordingly, if, e.g., an inefficiency results from the undesirably extended duration of a team member’s performance of three consecutive tasks, the system may not simply reduce each of the three consecutive tasks proportionally, in equal increments, or over a same time frame, but may instead focus on the team member’s improvement of only one or more for the tasks (e.g., as some of the consecutive actions may depend upon performance of a preceding one of the actions, improvement of the preceding action may itself effect an improvement in the subsequent action while imposing less stress or cognitive burden upon the team member by presenting them with three successive, or simultaneous, requests for improvement). In some embodiments, in these situations, the team member or team members may be presented with the list of underperforming actions so that the team member or team members may select the order in which they wish to make their improvement (e.g., first selecting an action they believe themselves most able to readily improve). Furthermore, in some embodiments, even when a first action is being performed efficiently and a second action inefficiently, in accordance with the prioritization, it may be easier to effect an overall improvement by further improving performance of the first task, rather than the second. Accordingly, in thesecircumstances, the system may provide adjusted recommended times for the “efficient” task so as to indirectly compensate for the “inefficient” task.

[0109] As another example, for clarity, a team may spend 2 hours on nonoperative tasks, which a hospital reviewer may wish to reduce to a more optimal number. Asking the team to transition directly from 2 hours to 1 hour may be difficult or infeasible. If, instead, actions within the nonoperative period are incrementally reduced from the current average time it takes the team members to the target time, then the team members may be able to make multiple smaller, less intrusive, adjustments to accomplish the ultimate goal. Specifically, such gradual, iterative adjustment are more likely to result in successful adherence than demanding a single, sweeping change, particularly as incremental adjustment may give each member time to recognize time-saving actions in other aspects of their workflow. Teams and team members may thus be encouraged to constructively “compete” with one another to reach optimal action times.

[0110] The rows 1005i-m may be color coded, animated, or otherwise provided with indicia to signify the assigned team member, the nature of the elapsed time and time remaining, etc. In this example, the first assistant has exceeded the recommended time for performing their task by over four minutes. Accordingly, the border of row 1005k may pulse with increasing frequency as the excess continues (e.g., its frequency being positively correlated with the excess). Similarly, or alternatively, the background color of the row 1005k may change to convey the urgency. In some embodiments, these changes may be viewed by all the team members so that they are aware of one another’s progress. However, in some embodiments, the changes may be viewed only by the affected team member or team members, e.g., to avoid distracting the other team members. As one team member’s delay may directly impact another team member’s upcoming task, notifying only affected individuals may facilitate localized corrections or accommodations between the team members. As the surgeon has only just recently passed the recommend time, only a mild indicia of urgency may be indicated for row 1005m (e.g., a slowly pulsing animation, a slightly different background hue, a soft audio indicia, etc.). In some embodiments, various audio cues may likewise, or alternatively, be used to warn team members that they are approaching, or have surpassed, a recommend time upon atask. For example, a beeping may occur with increasing frequency as a recommend time is approaching or continues to be increasingly surpassed.Example Coordinated Interface Display

[0111] The GUI 1005 elements may be presented at various locations throughout the surgical theater. For example, FIG. 11 A is a schematic representation of elements in a theater presenting the GUI 1005 of FIG. 10. Here, a first team member 1105c is using a tablet 1115e, which may be configured to perform various augmented reality functions in performance of the first team member 1105c’s present task. Particularly, FIG. 11 D provides an enlarged view of the tablet rendering 1115e in addition to various of the augmented reality elements 1125c and 1125d (e.g., augmented reality virtual elements indicating where the team member should go to perform an action, relevant metadata, etc.).

[0112] A second team member 1105b is likewise wearing an augmented reality headset 1115c, which, in addition to various augmented reality renderings 1115d seen by second team member 1105b, may also present a rendering 1110b of the GU1 1005 as an augmented reality element 1110b (e.g., as a billboard, textured plane, heads-up- display overlay on the team member’s field of view, etc.). Renderings of the GUI 1005 may likewise be provided on side dollies, e.g., the rendering 1110d in the display 1115f, on theater-mounted displays, e.g., the rendering 1110a on the ceiling mounted system 1115b (as shown in enlarged detail in FIG. 11 B), etc. Within the surgeon console 1115a, the surgeon 1105a may likewise be presented with a rendering of the GUI 1005 in some embodiments (e.g., during operative periods). For example as shown in FIG. 11 C, a rendering 1110e of the GUI may appear as an overlay in the surgeon’s heads-up display 1120

[0113] Each of the renderings 1110a-e may be coordinated, or may be particularized, to the members expected to monitor the rendering, e.g., as described in greater detail herein. For example, the rendering 1110d and rendering 1110a may present a general view, indicating the action progress status of any of the team members, whereas the views 1110b, 1110c, and 1110e may be personal to the viewer, only providing TOM progress information relevant to the particular team member.Example Interface Display Configurations

[0114] In addition to only providing urgency indicia to the particularly affected team members, in some embodiments, the GUI 1005 may be modified in accordance with individual team member and various other circumstances. For example, FIG. 12 is a collection of configuration variations of the GUI of FIG. 10 as may occur in some embodiments. In the “highlight” configuration 1205a one or more rows may receive highlighting indicia, e.g., a change in background color, a change in border color, animated backgrounds or borders, etc. to call the user’s attention to the particular row. This may be done when the row refers to the user’s own task and the user’s current elapsed task time is near, or has surpassed, the recommended time. In operative and nonoperative periods there will be dependency relations between the various actions of the TOM (e.g., the robotic surgical system must be rolled up to the patient before instrument insertion can occur). Thus, the personalized rendering may also highlight the times of other members upon whose actions this viewing member my depend for completion of their own actions.

[0115] In the “isolation” or “personalized” configuration 1205b only data relevant to a particular team member may be presented (or presented more prominently, e.g., with a higher opacity than the other rows). Such a limited view may be appropriate for attention-demanding tasks that do not depend upon other team members and where the team member’s cognitive load should be minimally imposed upon. Alternatively, in some embodiments, a particular team member’s personalized, local view may still include all of the rows of the other team members, as shown in FIG. 10, but the recommend time 1005h is only shown for that particular team member (e.g., if the view is personalized to the scrub tech in row 1005j, then only the recommended time of 15 minutes in row 1005j will be shown, but not the other recommended times) or only for that particular team member and for other team members performing actions affecting that team member (e.g., based upon a dependency relation with a future of the particular team member’s tasks). Such filtered presentation may help reduce cognitive load. Similarly, in some embodiments, only the elapsed time on task 1005g will be shown for the particular team member and the recommended times of the other team members may, or may not, be displayed.

[0116] In the “relative display” configuration 1205c only rows for related team member tasks may be presented. For example, where the reviewing team member’s next or upcoming task depends upon the actions of other team members, this configuration allows the team member to review only those dependent team members’ progress. Conversely, where a member’s current action impacts the actions of the members, this configuration may be used in the reverse fashion. As some team members have “idle” phases, this configuration may be useful for informing them of the team’s progress during that period, and may indicate with which of the other member’s actions the idling team member may assist.

[0117] In the “collective reassignment” configuration 1205d, two team members may be assigned the same task and share a single row, rather than two separate rows, to reflect as much. In this manner, rather than simply indicate that a team member is presently idle, the available resources of the idle member may be automatically reallocated to where they will do the most benefit (e.g., based upon a priority ordering logic for outstanding actions). In some embodiments, the recommended time may likewise be adjusted to reflect that two members are now performing the same action.

[0118] In the “upcoming assignments” configuration 1210 the GUI in its current state 1210a and in a future state 1210b (e.g., advancing each presently active action by one or more action transitions), may be shown, to apprise the user of the expected upcoming state of the surgery. Though shown here side-by-side, the GUIs may be translucently overlaid, future state rendering 1210b shown as a smaller overlay, etc. Temporally advancing the view of the current TOM in this fashion can help the team to reevaluate their approach when unexpected events occur, but without being overwhelmed by the entirety of the TOM, as in FIG. 5.

[0119] In the “holistic” configuration 1220, the GUI 1005 may be presented in the region 1220a and a supplemental region 1220b may indicate related theater information via elements 1225a-d. For example, as discussed with respect to FIG. 3, it may help team members other than the surgeon to be informed of the surgeon’s progress on one or more of tasks 370a-d (e.g., to notify the anesthesiologist while monitoring the patient’s status). Conversely, when the TOM is presented during an inter-surgical period (e.g.,one of periods 310a-d), it may help the team members performing the inter-surgical operations to know what surgeries are coming up and to receive real-time updates so that they can adjust the performance of their actions accordingly. For example, the system may interface with a scheduling system so that the team is aware when a patient has canceled their surgery, experienced an adverse event accelerating the need for their surgery, when personnel reassignments for upcoming surgeries are possible or necessary, etc. Indeed, such a configuration change may occur in connection with related actions, as when a surgeon’s action is to prepare for an upcoming surgery, and so the rendering is automatically adjusted to include information relevant to that upcoming surgery once the action’s temporal interval has begun.Example System Deployment Overview

[0120] FIG. 13 is a flow diagram illustrating various operations in an example process 1300 for preparing and deploying a dynamic TOM, as may be implemented in some embodiments. At block 1305a, the system may determine the surgery type (e.g., a modality, such as robotic, non-robotic laparoscopic, non-robotic open, etc., and / or a type of surgical procedures, such as prostatectomy, etc.). For example, a team member may specify the type, the system may pull the surgery type form a hospital scheduling database, etc. At block 1305b the system may determine the equipment corpus available in the theater (e.g., again consulting a scheduling database or performing machine learning detection) and at block 1305c may determine the personnel corpus (e.g., by consulting the scheduling database or performing machine learning detection methods to identify the personnel). For example, the system may check the scheduling roster to anticipate how many team members are excepted to be in the theater for the surgery and compare this expected number to a machine learning detected personnel count. In some embodiments, in lieu of blocks 1305a-c the system may receive or identify a default TOM template for a particular surgery (e.g., as derived by hand from a static TOM, represented, e.g., as JavaScript Object Notation, a Structured Query Language database table, etc.). Thus, a default state derived from a static TOM may serve as the foundational “source of truth” of the dynamic TOM.

[0121] At block 1305d, the system may then consult historical data for similarly situated operative or nonoperative periods. For example, past, static TOMs for a given type of surgery may be used as default starting templates from which the dynamic TOM may be generated. The historical data may also be used for identifying the recommended times for various tasks, e.g., based upon the team members’ efficiency goals, historical efficiency goals, etc.

[0122] At block 1305e, the system may filter the historical data to find relevant historical instances, e.g., those corresponding to the same type of surgery (robotic, non- robotic laparoscopic, non-robotic open, etc.), corresponding to the available equipment and personnel identified at blocks 1305b and 1305c, etc. Based upon the analysis of these instances, the initial configuration generation for the template TOM may be determined at block 1305f.

[0123] The system may then prepare a TOM at block 1305g (e.g., preparing a dependency structure of actions as discussed with respect to FIGs. 8A-B), and, in some embodiments, may seek verification at block 1305h (e.g., from one of the team members or from a hospital administrator). Where the reviewer does not approve the template TOM, then the system may prepare a revised version in accordance with the feedback at block 1305L For example, the reviewer may change the recommended durations for tasks, change the prioritization of tasks when multiple independent options will be available to a team member, change individual dependency relations or role assignments based upon a unique expected course of action in the theater, etc. In many embodiments, however, the system will generate the TOM without inviting human verification, relying instead, e.g., upon the processing of theater-wide data via the machine learning methods disclosed herein to verify the appropriateness of the initially prepared TOM.

[0124] Once approved (if approval is sought), at block 1305j, the system may publish the TOM, e.g., to one of the computer systems 190a and 190b (if process 1300 is not already being run upon them), which may in turn facilitate deployments to the various user views depicted in FIGs. 11A-D. The system may then monitor the surgery until all the TOM action group divisions are complete at block 1305k. During such review, at block 13051 the system may acquire the most recent available theater-wide data. At block1310a, the system may perform tracking and monitoring operations, e.g., to monitor the number and identity of the personnel in the theater from the data 375a and 375b, using the acquired data (e.g., using the machine learning systems 615a and 615b).

[0125] If there has been a parameter change, e.g., the number of available personnel, the available equipment, etc., have changed, then the system may transition from block 1310b to block 131 Of, to determine if the change is in accordance with the TOM.

[0126] If the change was not in accordance with the TOM, then, if relevant to the new parameter value, at block 1310c the system may update a tracking system to monitor the change in situation (e.g., the introduction of new personnel) and update the TOM at block 131 Od (e.g., using the methods discussed herein with respect to FIG. 15). At block 1310e, the system may again publish the TOM, e.g., again submitting the updated TOM to the computer systems 190a and 190b for deployment and rendering (when the process 1300 is not itself being performed on these systems).

[0127] At blocks 1305s and 1305t, the system may iterate over the team members, verifying if their actions are complete at blocks 1305u and 1305v, updating the TOM accordingly if so at block 1305w. For example, users may manually acknowledge completion of an action by, e.g., selecting their row in the GUI of FIG. 10, at block 1305v. Alternatively, or additionally, the system may analyze the data 375a and 375b to automatically infer action completion at block 1305u (e.g., using the machine learning methods discussed herein).

[0128] At block 1305x the TOM renderings (e.g., as in FIGs. 11A-D) may be updated in accordance with the current state of the TOM and the team’s progress. One will appreciate that these update renderings may be delegated to other computer systems, e.g., the computer systems 190a and 190b. After completion of all the action groups in the TOM at block 1305k, then at block 1305z, the system may perform the postdeployment analysis and data recordation for consideration in future procures (e.g., as part of future historical data acquisition at block 1305d).Example Rendering Management

[0129] FIG. 14 is a flow diagram depicting various operations in an example process 1405 for managing the GUI representations of FIGs. 10 and 11A-D, as may be implemented in some embodiments, e.g., as part of block 1305x. Renderings presenting only data relevant to a particular team member are referred to as localized, or local, renderings herein (e.g., one of the user-specific configurations of FIG. 12), whereas, renderings presenting all the TOM data relevant to the current time (e.g., as in FIG. 10), albeit, possibly restricted to the relevant window of presently performed actions (and possibly their neighbors) to reduce cognitive load, as discussed herein, are referred to as non-localized, or non-local, renderings. Generally, the non-localized representations may reflect most of the currently relevant information in the internal representation of the TOM 650, whereas the local representations may be adjusted in accordance with the specific role and task of the user.

[0130] At each update, the system may determine the current theater state at block 1405a, e.g., based upon the results of the personnel detection machine learning system 615a and activity recognition machine learning system 615b. At block 1405b, the system may verify that the TOM agrees with the theater state, making any updates if needed (e.g., following completion of an action). At block 1405c, the system may then prepare the non-localized rendering of the TOM and provide the non-local rendering to all the appropriate display system at blocks 1410a, 1410b, and 1410c (e.g., as may appear in renderings 1110a and 1110d).

[0131] After managing the non-local renderings, the system may iterate over the localized renderings (e.g., as may appear in renderings 1110b, 1110c, 1110e) associated with various roles at blocks 1415a and 1415b. In some embodiments, the non-local rendering may be provided universally or upon user request. If the user instead desires the non-local rendering upon their interface at blocks 1420a, then non-local rendering may be provided at block 1420b.

[0132] Where the local rendering is be instead presented, then the system may determine the specific personnel’s state at block 1420c (e.g., using the activity recognition machine learning system 615b or simply based upon the TOM state), prepare a corresponding TOM rendering specific to the user at block 1420d (e.g., per one of theformats of FIG. 12) based upon that state (and mindful to reduce cognitive load), then provide the rendering at block 1420e. The rendering system may then wait until the next system update to again perform the process 1405.Example Resource Reallocation Management

[0133] As mentioned, the system 605 may adjust the dynamic TOM 650 in response to new theater parameters, e.g., as determined at block 1310b. For example, the personnel detection system 615a may detect an unexpected departure of a team member or the introduction of an unexpected additional team member. When such parameter changes occur, the system may reallocate the available manpower to the pending tasks. FIG. 15 is a flow diagram depicting various operations in an example process 1505 for such reallocation of personnel, as may be implemented in some embodiments.

[0134] Until all the tasks in the TOM are complete at block 1505a, the system may check for excess unassigned team members at block 1505b (e.g., using the personnel detection system 615a or the logic 610a). Such excess may result from idle team members who have completed their tasks, but must wait before moving on to the next action for their role, or because a new team member has entered the theater. Where such additional capacity is available, then at block 1505c the system may determine the viable actions available for the team member. For example, if the new member’s role is known, then the system may consult a corresponding portion of known TOMs to make an assignment. At block 1505d, the system may determine peer active actions that may permit assistance or co-performance. At block 1505e, the system may then determine peer future actions amenable to assistance or reassignment at the present moment.

[0135] At block 1505f, the system may determine which of these identified actions are not blocked. That is, only some actions may be “reachable” from the current theater state, e.g., due to lack of equipment or personnel, and will consequently be excluded as non-viable. As another example, certain roles may be excluded from certain activities based upon sterility requirements. For example, it may not be possible for a circulator’s tasks to be performed by most team members, since those members (unlike the circulator) are obligated to remain sterile for their respective actions.

[0136] Thus, from the corpus of actions determined at blocks 1505c, 1505d, and 1505e, the system may then determine which of the actions are suitable for the current theater state and which are not at block 1505f. If any actions remain viable in view of the dependencies at block 1505g, then the system may order the viable actions by priority at block 1505h, and adjust the template (e.g., adding a row in the GUI 1005 if the member is new to the theater) at block 1505i . At block 1505k, the system may direct the new team member to perform the action (e.g., by publishing the updated TOM to the new member’s TOM rendering).

[0137] One will appreciate that such automatic adjustment removes the need to locate the consultant who generated the TOM (who, as mentioned, may not even be available) and invite them to observe the theater following a change in parameters. Automated construction of parallel task lanes tailored to the needs of care team members and guiding those members with a suitable graphical interface creates much less of an imposition upon the theater team. The prioritization of actions may also be based upon historical data and guidance specific to the recognized team member (e.g., in conjunction with the iterative improvement methods disclosed herein with respect to FIGs. 17A-B).Example TOM Management - Arriving Personnel / Departing Personnel

[0138] To further facilitate the reader’s understanding of how rules and logic 610a may manage parameter changes, changes in the theater state, etc., FIG. 16 schematically depicts state management adjustments as may be performed in accordance with various embodiments (again, one will appreciate that the FIG. 16 is provided to facilitate the reader’s understanding and that the depicted actions may not be handled in exactly the order as depicted here, having, e.g., different dependency relations in an actual deployment). Here, in an initial state 1605a, three personnel may be detected in the theater with the roles circulator, scrub technician, and anesthesiologist, with respective patient positioning, setup, and intubation actions assigned in accordance with the TOM. Accordingly, the state may generally include actions of action groups 505c, 510c, and 520c. Past completed actions may also be recorded in the order they occurred, for reference, as indicated (e.g., to verify their completion status for future actions having these past actions as dependencies, though such a record may not be used in someembodiments). For example, where state transitions are to be modeled as fully Markovian, then maintaining a record may facilitate the making of the next state independent of past states. As mentioned, in some embodiments, the TOM state may be presented as a single vector, such as a one-hot vector, to which one may append all the completed actions. Such as structure in combination with a dependency network (as in FIG. 8B, which may be represnted as a hashmap of dependency relations) may, in some embodiments, provide sufficient information for deciding what actions to assign next to each role, which of the next potential actions may be performed in parallel, as well as to accommodate newly arrived team members.

[0139] Absent deviations, the system will continue to push the current actions to the completed action history stack, and assign a new current action in accordance with the TOM. Similarly, at time T1 , the GUI of FIG. 10 may be presently “ticking down” in accordance with the depicted actions. One will appreciate that, rather than attempt to achieve real-time solutions, the internal representation of the TOM may instead be “fully saturated” with all possible roles, and role combinations, for the relevant period, even when only some of the roles are presently satisfied, so that upon arrival of a new team member, the new role may be “activated.”

[0140] Upon detection 1610a of a new team member, however, the system may perform a role identification (e.g., using personnel detection machine learning system 615a if the user does not identify their role themselves) and transition to state 1605b. In this example, the new personnel is identified as a second scrub technician. Based upon the existing TOM and the action priorities, the second scrub technician may be assigned the same “setup” action assigned to the scrub technician (e.g., in accordance with the action of action group 515c), thereby assisting the scrub technician, and a new row 1615a created. As indicated, the second scrub technician presently lacks any completed past action history and so the columns beyond the current action are presently empty. Also note that the anesthesiologist has completed their “intubate” task and so it is placed as a record 1625b in the completed action history queue, and the next action without any blocking dependency actions (here “position table”) is assigned.

[0141] Upon detection 1610b of another new team member, the new member’s role may likewise be identified and the system may transition to state 1605c, where a current action (“position patient”, which was being performed by the circulator) is instead assigned to the new team member in a new row 1615b. Here, the new member is identified as a surgeon. Because the circulator’s position patient task 1620b has a high priority, is a task shared with the surgeon, and because the circulator has other pending actions without blocking dependencies, “position patient” is here reassigned to the surgeon (again, in accordance with action group 525c). Thus, the circulator has transitioned to the “skin prep” action 1620a in accordance with the next action of action group 505c as skin preparation can be performed independently of patient positioning. This is in contrast to the assignment of “setup” to the second scrub technician in combination with the first scrub technician, since their other actions cannot begin until setup completes (and so it makes sense to assign that task to them in common). Also observe, in accordance with the atomicity requirement, that position patient 1620b does not appear in the circulator’s action history, since it is not yet complete. Rather, it may not be marked as “completed” and placed in the history until completed by the surgeon. Thus, after completion of skin prep 1620a, should the circulator have no other actions without outstanding dependencies, save actions which depend upon the position patient action, then the circulator may be idle, or assigned to assist the surgeon with completion of the position patient action.

[0142] Thus, in conjunction with the completion atomicity, by knowing what actions are completed, and consequently which outstanding actions are without incomplete dependency requirements, the system may readily assign new tasks when new team members arrive. Enforcing atomicity of each action, wherein, either the assigned action is either fully completed once taken on, or entirely abandoned, facilitates complete histories of completion states suitable for downstream dependency analysis and action assignment or reassignment (though, again, this atomicity may not require that action’s be “started over” if abandoned by one team member and passed to another, as occurs here between the circulator and surgeon). Similarly, in view of such atomicity, when team members leave the theater, rather than arrive, the procedures discussed above may be performed in reverse. For example, a removal would be analogous to transitioning fromT2 to T1 above in reverse. Because the actions are atomic, reassignment to a new team member will not block any other action, as action assignment changes will only occur at discrete time steps.Example Incremental Performance Adjustment Plans and Iterative Monitoring

[0143] As mentioned, various embodiments may seek to iteratively improve team performance via the TOM management system. The improvements may be accomplished by guiding the team through a sequence of intermediate goals from their current efficiency level to the recommended efficiency level. As mentioned, this was generally not possible in the old, static TOM approach, where a single recommended time was insisted upon for a given action.

[0144] For example, it may be desirable that turnover be completed in 30 minutes. If the team is currently completing turnovers in an hour upon average, then aiming for 30 minutes in a single step will often be unrealistic as many separate adjustments across several team members may be needed to effect the improvement. Instead, as mentioned, various embodiments plan a trajectory to decrease turnover time in stages, e.g., from 60 minutes to 50 minutes, from 50 minutes to 40 minutes, and from 40 minutes to 30 minutes, possibly over the course of several surgeries (indeed, it may require many iterations before a member’s habits are properly adjusted). Across these instances of the action, the system may note and record the team, group of one or more team members’, or individual team member’s progress and only continue to reduce the time when the current threshold has been satisfied (e.g., not proceeding to a 40 minute limit until the 50 minute limit has been satisfied at least once or a consecutive threshold number of times). The system may thus update the recommended time for individual tasks over multiple instances of the task’s performance to eventually accomplish a desired goal.

[0145] In accordance with these strategies, FIG. 17A is a collection of schematic block diagrams depicting example incremental performance adjustment plans as may be implemented in some embodiments. Specifically, various embodiments may employ a variety of different incremental performance adjustments so as to better ensure compliance by the team or team members. In these examples, a first adjustment plan 1705 seeks to take team member(s) from a current performance time (the full depictedduration marked “Current” in the figure, which may be, e.g., a mean or median of the member’s current performance durations for the action) to a shorter, target duration for an action (as indicated by arow 1705f), the duration of the target time here reflected by interval 1705a. In this example plan 1705, the intention is to progress the team member by equally spaced intervals 1705b-e. That is, initially, the system will seek to have the team member comply with the duration indicated cumulatively by the intervals 1705a-d. Once this goal has been satisfied, the system may then seek to have the team member(s) comply with the duration associated with intervals 1705a-c, then following that duration’s satisfaction, the system may seek to have the team member(s) comply with the duration associated with both intervals 1705a-b, until finally the team member is asked to achieve the duration of the final target interval 1705a.

[0146] While the plan 1705 may be suitable for many circumstances, different actions or circumstances may recommend different incremental adjustments, e.g., more granular intervals, more or fewer intervals, plans with successive intervals that do not monotonically decrease or increase, but permit “relief” periods when requirements are relaxed, etc. than those shown in the example plan 1705. For example, in the plan 1710, the goal is again for the team member(s) performance of an action, or group of actions, to move from a current time (the duration associated with the sum of all the intervals 1710a-i) to a target time (the duration associated with the interval 1710a), as indicated by the directional arrow 171 Oj. By using more granular intervals 171 Ob-i than the intervals in the plan 1705, the team member(s) may have more opportunities to make incremental adjustments to their workflow than in the plan 1705. This may be suitable where the urgency to reach the target duration of interval 1710a is sufficiently low that the team member(s) can gradually make their adjustment over many more instances of performing the action, than, e.g., may be expected to achieve the goal with the plan 1705. Smaller intervals may also impose a more realistic demand upon team member(s).

[0147] As yet another example, in the example plan 1715, the intervals 1715b-f are not equally spaced, but instead assume unequal durations, e.g., as an increasing or decreasing geometric sequence. That is, in this example, while the goal remains to move the team member’s performance from the current duration (the cumulative duration of intervals 1715a-f) to the target duration (the duration 1715a) in the direction indicated byarrow 1715g, each of the successive intervals 1715b-f is shorter than the previous interval. Such a plan may be suitable where initial compliance is expected to be readily attainable by a team member, but additional improvements may be increasingly difficult (e.g., such difficulty may be inherent in the nature of the task or may be contingent upon performances by other team members). In some embodiments, while still seeking to decrease interval by interval, the reverse order of intervals than depicted in the plan 1715 may be applied, where the intervals grow longer, rather than shorter, as shown. This alternative may be appropriate if the team member is expected to improve in their performance change over time, and consequently, greater successive improvement is a reasonable expectation (as may, e.g., often be the case for team members introduced to a new task they can be expected to readily master).

[0148] For clarity, though the example plans 1705, 1710, and 1715 are shown here as encouraging adjustment from a longer current time to a shorter target time, as discussed elsewhere herein, one will appreciate that for some actions, it may be desirable to instead encourage the team member to lengthen their duration, mutatis mutandis. For example, where a team member regularly overlooks important details during a task, blithely completing the task more quickly than is prudent, a reviewer may seek to impose a lower bound upon the team member’s performance duration to encourage a more thorough performance.

[0149] FIG. 17B is a flow diagram illustrating various operations in an example process 1720 for performing incremental performance guidance and monitoring as may be implemented in some embodiments (e.g., as part of logic 610a, as part of a computer system distinct from system 605, etc.). Specifically, at block 1720a the system may receive the one or more target durations (e.g., one of the durations associated with intervals 1705a, 1710a, 1715a) for one or more actions. At block 1720b, the system may receive the corresponding target member’s current performance for the respective actions. This may include, e.g., durations captured in a threshold number of past performances of the action from which the system may, e.g., take a mean or median as the current time in each of plans 1705, 1710, 1715, etc.).

[0150] As not all desired target action performance durations may be equally valuable to improving surgical outcomes, or because some targets will be known to be best addressed after other targets are regularly satisfied, in some embodiments, the targets may be associated with priorities at block 1720c. For example, a reviewer may seek to improve an action directly affecting patient health before also asking the team member(s) to improve a more administrative action. Priority may then be used, e.g., to determine the order in which plans are imposed upon the team member(s), the type of plan (e.g., one of plans 1705, 1710, 1715), the intervals chosen (wider intermediate intervals for urgent actions, smaller intervals for less urgent actions), etc.

[0151] To similarly inform priority, at block 1720d, the system may determine the dependency relations for each of the actions identified at block 1720a, e.g., using the internal TOM representation and structures discussed herein with respect to FIGs. 8A and 8B. These relations may be used in some embodiments to adjust the priority with which performances are sought to be improved. Here, for example, the priorities may be adjusted at block 1720e based upon the dependencies. If one team member’s improvement depends upon another’s, then the latter’s improvements may be prioritized over the former’s, as asking the former to improve when aspects of their actions are beyond their control would be unreasonable. Similarly, if other actions depend on action, then it may be prudent to improve it first (thus, in some embodiments, the action priorities for choosing assignments may also, by proxy, serve as an indication of which action’s performances should be first improved). At block 1720f, the system may determine the intermediate interval targets (e.g. the intervals 1705b-e) for each of the actions based upon the previously acquired information and priorities.

[0152] At block 1725a, data may be collected (e.g., using the system 605 disclosed herein) of the respective team members’ performances of the respective actions being monitored. Such performance monitoring may consider one or more performances of the action and the associated member’s success, or failure, in meeting the currently prescribed intermediate target duration. If all the members are meeting all the final targets for all the respective actions at block 1725b, then the system may record the result for future reference at block 1725c before concluding. For example, recording how often ateam member failed to satisfy an intermediate target may be useful to reviewers when considering the nature of the inefficient action.

[0153] In contrast, where final targets remain unmet (e.g., there are actions where the team member has yet to perform within the intervals 1705a, 1710a, or 1715a), then at blocks 1730a and 1730b, the system may iterate over the actions with outstanding target times (which, again, may be the final duration, e.g., for interval 1705a, or an intermediate goal, e.g., the duration of intervals 1705a-c) and determine at block 1730c if the targets were satisfied in the last round of performance at block 1725a. If the intermediate or final target has not been satisfied (e.g., once or a threshold number of required times), then the system may allow the team member(s) another round of performance at block 1725a so that they might further improve, before again assessing the member’s progress. Note that the target requirement of block 1730c may not be simply the satisfaction of a single performance at block 1725a, but of a threshold number of successful performances, or a threshold number of consecutive successful performances. Such requirements may themselves be imposed based upon the priority (e.g., requiring a higher minimum continuity of success for higher priority actions).

[0154] If the target was met in the last round of performance at block 1725a, then at block 1735a, the system may consider whether the target was the final target time or one of the intermediate interval times. If the current target is an intermediate time, then at block 1735b, the system may adjust the next goal to the next desired interval (e.g., instead of asking the member(s) to complete the task in the duration reflected by intervals 1710a- h, asking that they instead complete the actions within the duration reflected by intervals 1710a-g). Conversely, if this was the final goal and it has been satisfied, then the system may remove the target from consideration in the future at block 1735c.

[0155] In some embodiments, following completion of a performance improvement or completion of an incremental goal, the system may adjust priorities at block 1735d. For example, if a member is sufficiently improving upon an action, then it may be reasonable to ask another member, or the same member, to begin improving an action that was dependent upon the first team member’s performance. Adjustment in priority may be effected in a variety of ways, e.g., as discussed with respect to the plans of FIG.17A, by more or less granular intervals, a change in the target time, dynamic adjustment between the intervals (e.g., as in plan 1715), etc.Computer System

[0156] FIG. 18 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments. The computing system 1800 may include an interconnect 1805, connecting several components, such as, e.g., one or more processors 1810, one or more memory components 1815, one or more input / output systems 1820, one or more storage systems 1825, one or more network adaptors 1830, etc. The interconnect 1805 may be, e.g., one or more bridges, traces, busses (e.g., an ISA, SCSI, PCI, I2C, Firewire bus, etc.), wires, adapters, or controllers.

[0157] The one or more processors 1810 may include, e.g., an Intel™ processor chip, a math coprocessor, a graphics processor, etc. The one or more memory components 1815 may include, e.g., a volatile memory (RAM, SRAM, DRAM, etc.), a non-volatile memory (EPROM, ROM, Flash memory, etc.), or similar devices. The one or more input / output devices 1820 may include, e.g., display devices, keyboards, pointing devices, touchscreen devices, etc. The one or more storage devices 1825 may include, e.g., cloud-based storages, removable Universal Serial Bus (USB) storage, disk drives, etc. In some systems memory components 1815 and storage devices 1825 may be the same components. Network adapters 1830 may include, e.g., wired network interfaces, wireless interfaces, Bluetooth™ adapters, line-of-sight interfaces, etc.

[0158] One will recognize that only some of the components, alternative components, or additional components than those depicted in FIG. 18 may be present in some embodiments. Similarly, the components may be combined or serve dual-purposes in some systems. The components may be implemented using special-purpose hardwired circuitry such as, for example, one or more ASICs, PLDs, FPGAs, etc. Thus, some embodiments may be implemented in, for example, programmable circuitry (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in special-purpose hardwired (non-programmable) circuitry, or in a combination of such forms.

[0159] In some embodiments, data structures and message structures may be stored or transmitted via a data transmission medium, e.g., a signal on a communications link, via the network adapters 1830. Transmission may occur across a variety of mediums, e.g., the Internet, a local area network, a wide area network, or a point-to-point dial-up connection, etc. Thus, “computer readable media” can include computer-readable storage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0160] The one or more memory components 1815 and one or more storage devices 1825 may be computer-readable storage media. In some embodiments, the one or more memory components 1815 or one or more storage devices 1825 may store instructions, which may perform or cause to be performed various of the operations discussed herein. In some embodiments, the instructions stored in memory 1815 can be implemented as software and / or firmware. These instructions may be used to perform operations on the one or more processors 1810 to carry out processes described herein. In some embodiments, such instructions may be provided to the one or more processors 1810 by downloading the instructions from another system, e.g., via network adapter 1830.

[0161] For clarity, one will appreciate that while a computer system may be a single machine, residing at a single location, having one or more of the components of FIG. 18, this need not be the case. For example, distributed network computer systems may comprise multiple individual processing workstations, each workstation having some, or all, of the components depicted in FIG. 18. Processing and various operations described herein may accordingly be spread across the one or more workstations of such a computer system. For example, one will appreciate that a process amenable to being run in a single thread upon a single workstation may instead be separated into an arbitrary number of sub-threads across one or more workstations, such sub-threads then run in serial or in parallel to achieve a same, or substantially similar, result as the process run within the single thread. Similarly, one will appreciate that while a non-transitory computer readable medium may stand alone (e.g., in a single USB storage device), or reside within a single workstation (e.g., in the workstation’s random access memory or disk storage), such a medium need not reside at a single geographic location, but may include, e.g., multiple memory storage units residing across geographically separated workstations ofa computer system in network communication with one another or across geographically separated storage devices.Remarks

[0162] The drawings and description herein are illustrative. Consequently, neither the description nor the drawings should be construed so as to limit the disclosure. For example, titles or subtitles have been provided simply for the reader’s convenience and to facilitate understanding. Thus, the titles or subtitles should not be construed so as to limit the scope of the disclosure, e.g., by grouping features which were presented in a particular order or together simply to facilitate understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, this document, including any definitions provided herein, will control. A recital of one or more synonyms herein does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any term discussed herein is illustrative only and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term.

[0163] Similarly, despite the particular presentation in the figures herein, one skilled in the art will appreciate that actual data structures used to store information may differ from what is shown. For example, the data structures may be organized in a different manner, may contain more or less information than shown, may be compressed and / or encrypted, etc. The drawings and disclosure may omit common or well-known details in order to avoid confusion. Similarly, the figures may depict a particular series of operations to facilitate understanding, which are simply exemplary of a wider class of such collection of operations. Accordingly, one will readily recognize that additional, alternative, or fewer operations may often be used to achieve the same purpose or effect depicted in some of the flow diagrams. For example, data may be encrypted, though not presented as such in the figures, items may be considered in different looping patterns (“for” loop, “while” loop, etc.), or sorted in a different manner, to achieve the same or similar effect, etc.

[0164] Reference herein to "an embodiment" or "one embodiment" means that at least one embodiment of the disclosure includes a particular feature, structure, orcharacteristic described in connection with the embodiment. Thus, the phrase "in one embodiment" in various places herein is not necessarily referring to the same embodiment in each of those various places. Separate or alternative embodiments may not be mutually exclusive of other embodiments. One will recognize that various modifications may be made without deviating from the scope of the embodiments.

Claims

CLAIMSWe claim:1 . A computer-implemented method for managing presentation of action data to one or more members of a surgical team, the method comprising: causing a first set of indicia to be displayed, the first set of indicia comprising: a first role indicia, the first role indicia associated with a first role; a first current action indicia, the first current action indicia associated with a first action, the first action associated with a task overlap model; a first time elapsed upon action indicia; and a first recommended time indicia.

2. The computer-implemented method of Claim 1 , wherein, the first current action indicia indicates that the first action is assigned to a first team member, the first team member associated with the first role, wherein, the first time elapsed upon action indicia indicates a time elapsed in the first team member’s performance of the first action, and wherein, the first recommended time indicia indicates a first recommended time for performing the first action.

3. The computer-implemented method of Claim 2, the method further comprising: causing a second set of indicia to be displayed, the second set of indicia comprising: a second role indicia, the second role indicia associated with a second role; a second current action indicia, the second current action indicia associated with a second action, the second action associated with the task overlap model; a second time elapsed upon action indicia; and a second recommended time indicia.

4. The computer-implemented method of Claim 3, wherein, the second current action indicia indicates that the second action is assigned to a second team member, the second team member associated with the second role, wherein, the second time elapsed upon action indicia indicates a time elapsed in the second team member’s performance of the second action, and wherein, the second recommended time indicia indicates a second recommended time for performing the second action.

5. The computer-implemented method of one of Claims 2-4, the method further comprising: determining the first team member’s progress performing the first action by, at least in part: applying a first machine learning system to surgical theater data to detect one or more team members within the theater; and applying a second machine learning system to the surgical theater data to detect performance of the first action.

6. The computer-implemented method of Claim 5, wherein the method further comprises: detecting a completion of the first action by the first team member; and in response, at least in part, to detecting the completion of the first action by the first team member: adjusting the internal representation of the task overlap model; and updating the first action indicia to reflect a new action, the new action different from the first action, the new action determined, at least in part, based upon dependency relations within the task overlap model.

7. The computer-implemented method of Claim 6, the method further comprising:determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time; and in response, at least in part, to determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time, adjusting a recommended time associated with a third action, the third action not yet performed.

8. The computer-implemented method of Claim 7, wherein the third action is associated in the task overlap model with a role different from the first role.

9. The computer-implemented method of Claim 5, wherein the method further comprises: detecting a change in a number of team members of the surgical team; in response, at least in part, to the detecting the change in the number of team members, adjusting one or more action assignments of the task overlap model.

10. The computer-implemented method of Claim 9, wherein, the change in the number of team members comprises an entrance of a new team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.11 . The computer-implemented method of Claim 9, wherein, the change in the number of team members is a departure of a team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.

12. The computer-implemented method of Claim 9, wherein, adjusting one or more action assignments of the task overlap model comprises assigning a new action to a team member.

13. The computer-implemented method of one of Claims 1-4, wherein causing the first set of indicia to be displayed, comprises: preparing a local representation of at least a portion of the task overlap model based upon the first role, wherein instructions of the local representation are only relevant to the first role; and causing the local representation to be rendered upon a display.

14. The computer-implemented method of one of Claims 1-4, wherein the method further comprises: determining the first recommended time for performing the first action, wherein determining the first recommended time for performing the first action comprises: determining a historical performance duration associated with the first action; determining a final target performance duration associated with the first action; determining one or more intermediate target durations, the one or more intermediate target durations between the historical performance duration and the final target performance duration; and selecting the first recommended time for performing the first action from the one or more intermediate target durations.

15. The computer-implemented method of Claim 14, wherein the one or more intermediate target durations are equally distanced from one another.

16. The computer-implemented method of Claim 14, wherein the one or more intermediate target durations are spaced relative to one another in accordance with a substantially geometric progression.

17. The computer-implemented method of Claim 14, wherein determining the one or more intermediate target durations comprises:determining a priority of the first action; and selecting a spacing of the one or more intermediate target durations based upon the priority.

18. The computer-implemented method of Claim 17, wherein determining the final target duration of performance of the action comprises: selecting the final target duration based upon the determined priority.

19. The computer-implemented method of Claim 14, the method further comprising: monitoring the one or more team member’s successive compliance with the one or more intermediate target durations at least until the one or more team members comply with the final target duration, wherein, the determining the one or more intermediate target durations is based upon dependency relations in a task overlap model.

20. The computer-implemented method of Claim 14, wherein determining the final target duration of performance of the action is based upon dependency relations in the task overlap model.21 . A non-transitory computer-readable medium comprising instructions configured to cause at least one computer system to perform a method for managing presentation of action data to one or more members of a surgical team, the method comprising: causing a first set of indicia to be displayed, the first set of indicia comprising: a first role indicia, the first role indicia associated with a first role; a first current action indicia, the first current action indicia associated with a first action, the first action associated with a task overlap model; a first time elapsed upon action indicia; and a first recommended time indicia.

22. The non-transitory computer-readable medium of Claim 21 , wherein, the first current action indicia indicates that the first action is assigned to a first team member, the first team member associated with the first role, wherein, the first time elapsed upon action indicia indicates a time elapsed in the first team member’s performance of the first action, and wherein, the first recommended time indicia indicates a first recommended time for performing the first action.

23. The non-transitory computer-readable medium of Claim 22, the method further comprising: causing a second set of indicia to be displayed, the second set of indicia comprising: a second role indicia, the second role indicia associated with a second role; a second current action indicia, the second current action indicia associated with a second action, the second action associated with the task overlap model; a second time elapsed upon action indicia; and a second recommended time indicia.

24. The non-transitory computer-readable medium of Claim 23, wherein, the second current action indicia indicates that the second action is assigned to a second team member, the second team member associated with the second role, wherein, the second time elapsed upon action indicia indicates a time elapsed in the second team member’s performance of the second action, and wherein, the second recommended time indicia indicates a second recommended time for performing the second action.

25. The non-transitory computer-readable medium of one of Claims 22-24, the method further comprising:determining the first team member’s progress performing the first action by, at least in part: applying a first machine learning system to surgical theater data to detect one or more team members within the theater; and applying a second machine learning system to the surgical theater data to detect performance of the first action.

26. The non-transitory computer-readable medium of Claim 25, wherein the method further comprises: detecting a completion of the first action by the first team member; and in response, at least in part, to detecting the completion of the first action by the first team member: adjusting the internal representation of the task overlap model; and updating the first action indicia to reflect a new action, the new action different from the first action, the new action determined, at least in part, based upon dependency relations within the task overlap model.

27. The non-transitory computer-readable medium of Claim 26, the method further comprising: determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time; and in response, at least in part, to determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time, adjusting a recommended time associated with a third action, the third action not yet performed.

28. The non-transitory computer-readable medium of Claim 27, wherein the third action is associated in the task overlap model with a role different from the first role.

29. The non-transitory computer-readable medium of Claim 25, wherein the method further comprises: detecting a change in a number of team members of the surgical team; in response, at least in part, to the detecting the change in the number of team members, adjusting one or more action assignments of the task overlap model.

30. The non-transitory computer-readable medium of Claim 29, wherein, the change in the number of team members comprises an entrance of a new team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.31 . The non-transitory computer-readable medium of Claim 29, wherein, the change in the number of team members is a departure of a team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.

32. The non-transitory computer-readable medium of Claim 29, wherein, adjusting one or more action assignments of the task overlap model comprises assigning a new action to a team member.

33. The non-transitory computer-readable medium of one of Claims 21 -24, wherein causing the first set of indicia to be displayed, comprises: preparing a local representation of at least a portion of the task overlap model based upon the first role, wherein instructions of the local representation are only relevant to the first role; and causing the local representation to be rendered upon a display.

34. The non-transitory computer-readable medium of one of Claims 21 -24, wherein the method further comprises:determining the first recommended time for performing the first action, wherein determining the first recommended time for performing the first action comprises: determining a historical performance duration associated with the first action; determining a final target performance duration associated with the first action; determining one or more intermediate target durations, the one or more intermediate target durations between the historical performance duration and the final target performance duration; and selecting the first recommended time for performing the first action from the one or more intermediate target durations.

35. The non-transitory computer-readable medium of Claim 34, wherein the one or more intermediate target durations are equally distanced from one another.

36. The non-transitory computer-readable medium of Claim 34, wherein the one or more intermediate target durations are spaced relative to one another in accordance with a substantially geometric progression.

37. The non-transitory computer-readable medium of Claim 34, wherein determining the one or more intermediate target durations comprises: determining a priority of the first action; and selecting a spacing of the one or more intermediate target durations based upon the priority.

38. The non-transitory computer-readable medium of Claim 37, wherein determining the final target duration of performance of the action comprises: selecting the final target duration based upon the determined priority.

39. The non-transitory computer-readable medium of Claim 34, the method further comprising:monitoring the one or more team member’s successive compliance with the one or more intermediate target durations at least until the one or more team members comply with the final target duration, wherein, the determining the one or more intermediate target durations is based upon dependency relations in a task overlap model.

40. The non-transitory computer-readable medium of Claim 34, wherein determining the final target duration of performance of the action is based upon dependency relations in the task overlap model.41 . A computer system comprising: at least one processor; and at least one memory, the at least one memory comprising instructions configured to cause the computer system to perform a method for managing presentation of action data to one or more members of a surgical team, the method comprising: causing a first set of indicia to be displayed, the first set of indicia comprising: a first role indicia, the first role indicia associated with a first role; a first current action indicia, the first current action indicia associated with a first action, the first action associated with a task overlap model; a first time elapsed upon action indicia; and a first recommended time indicia.

42. The computer system of Claim 41 , wherein, the first current action indicia indicates that the first action is assigned to a first team member, the first team member associated with the first role, wherein, the first time elapsed upon action indicia indicates a time elapsed in the first team member’s performance of the first action, and wherein, the first recommended time indicia indicates a first recommended time for performing the first action.

43. The computer system of Claim 42, the method further comprising:causing a second set of indicia to be displayed, the second set of indicia comprising: a second role indicia, the second role indicia associated with a second role; a second current action indicia, the second current action indicia associated with a second action, the second action associated with the task overlap model; a second time elapsed upon action indicia; and a second recommended time indicia.

44. The computer system of Claim 43, wherein, the second current action indicia indicates that the second action is assigned to a second team member, the second team member associated with the second role, wherein, the second time elapsed upon action indicia indicates a time elapsed in the second team member’s performance of the second action, and wherein, the second recommended time indicia indicates a second recommended time for performing the second action.

45. The computer system of one of Claims 42-44, the method further comprising: determining the first team member’s progress performing the first action by, at least in part: applying a first machine learning system to surgical theater data to detect one or more team members within the theater; and applying a second machine learning system to the surgical theater data to detect performance of the first action.

46. The computer system of Claim 45, wherein the method further comprises: detecting a completion of the first action by the first team member; andin response, at least in part, to detecting the completion of the first action by the first team member: adjusting the internal representation of the task overlap model; and updating the first action indicia to reflect a new action, the new action different from the first action, the new action determined, at least in part, based upon dependency relations within the task overlap model.

47. The computer system of Claim 46, the method further comprising: determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time; and in response, at least in part, to determining that the first team member’s elapsed time in performance of the first action has exceeded the first recommended time, adjusting a recommended time associated with a third action, the third action not yet performed.

48. The computer system of Claim 47, wherein the third action is associated in the task overlap model with a role different from the first role.

49. The computer system of Claim 45, wherein the method further comprises: detecting a change in a number of team members of the surgical team; in response, at least in part, to the detecting the change in the number of team members, adjusting one or more action assignments of the task overlap model.

50. The computer system of Claim 49, wherein, the change in the number of team members comprises an entrance of a new team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.51 . The computer system of Claim 49, wherein,the change in the number of team members is a departure of a team member, and wherein, adjusting one or more action assignments of the task overlap model comprises adjusting a recommended time for an action.

52. The computer system of Claim 49, wherein, adjusting one or more action assignments of the task overlap model comprises assigning a new action to a team member.

53. The computer system of one of Claims 41-44, wherein causing the first set of indicia to be displayed, comprises: preparing a local representation of at least a portion of the task overlap model based upon the first role, wherein instructions of the local representation are only relevant to the first role; and causing the local representation to be rendered upon a display.

54. The computer system of one of Claims 41 -44, wherein the method further comprises: determining the first recommended time for performing the first action, wherein determining the first recommended time for performing the first action comprises: determining a historical performance duration associated with the first action; determining a final target performance duration associated with the first action; determining one or more intermediate target durations, the one or more intermediate target durations between the historical performance duration and the final target performance duration; and selecting the first recommended time for performing the first action from the one or more intermediate target durations.

55. The computer system of Claim 54, wherein the one or more intermediate target durations are equally distanced from one another.

56. The computer system of Claim 54, wherein the one or more intermediate target durations are spaced relative to one another in accordance with a substantially geometric progression.

57. The computer system of Claim 54, wherein determining the one or more intermediate target durations comprises: determining a priority of the first action; and selecting a spacing of the one or more intermediate target durations based upon the priority.

58. The computer system of Claim 57, wherein determining the final target duration of performance of the action comprises: selecting the final target duration based upon the determined priority.

59. The computer system of Claim 54, the method further comprising: monitoring the one or more team member’s successive compliance with the one or more intermediate target durations at least until the one or more team members comply with the final target duration, wherein, the determining the one or more intermediate target durations is based upon dependency relations in a task overlap model.

60. The computer system of Claim 54, wherein determining the final target duration of performance of the action is based upon dependency relations in the task overlap model.

Images