Surgical site non-operational period analysis and optimization

Abstract

Various disclosed embodiments provide systems and methods for evaluating surgical team performance during non-operational periods within a surgical site. Specifically, one or more sensors disposed throughout a surgical site may acquire data during non-operational periods, and in some embodiments, data may be acquired at several times of procedures and non-operational periods within the site. The disclosed embodiments segment and analyze the data to infer undesirable or inefficient behavior over non-operational periods between surgical procedures, as well as to improve the progression of behavior. Systems and methods for evaluating various topologies and metrics for non-operating periods, as well as for monitoring and providing feedback are also disclosed.

Description

[0001] Cross-references to related applications This application claims priority and benefit to U.S. Provisional Application No. 63 / 526,966, filed July 14, 2023, entitled “SURGICAL THEATER NONOPERATIVEPERIOD ANALYSIS AND OPTIMIZATION,” the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0002] Various disclosed embodiments relate to systems and methods for evaluating the performance of surgical team members in the surgical setting during nonoperative periods and for providing meaningful feedback. Background Technology

[0003] Surgical settings present unique challenges and operating conditions that require team members to adapt quickly and efficiently to rapidly changing technologies and situations. Failure of one or more team members to perform efficiently under these challenging conditions can lead to errors and inefficiencies, potentially cascading downstream delays, incurring undesirable costs, and posing potential risks to patient health. This potential for direct and indirect harm extends beyond inefficiencies occurring during the surgical procedure itself, encompassing actions taken before, between, and after these procedures. During these “non-operational” periods, team members must reset the site from previous surgeries and configure it for upcoming procedures. Although often overlooked, poor performance during these non-operational periods can itself lead to a variety of costly and potentially harmful downstream adverse events. Failure to promptly sort and store equipment, delays in patient transport to and from the site, excessive and unnecessary movement while working near patients, and similar inefficiencies during non-operational periods can directly harm patients, incur costs, or indirectly cause harm or costs through their impact on downstream tasks.

[0004] Unfortunately, identifying such non-operational inefficiencies (much less correcting these inefficiencies) can be extremely difficult, often intractable. Team members are so focused on patient care and the execution of their current tasks that they rarely, if ever, have the opportunity to reflect on their own behavior, recognize the impact of their behavior on other team members and downstream activities, and recognize inefficiencies in the team's overall collaborative dynamics. While third parties can observe team performance in situ, or subsequently via in situ sensor data, these observers themselves are subject to reviewer fatigue, reviewer subjectivity, and their involvement introduces undesirable temporal and financial constraints on performance evaluation. Indeed, such reviewers must be professionally trained and can only review a small number of procedures at a time. Furthermore, this human-in-the-loop approach to coaching complicates verifying that team members are following any provided feedback, as there is frequent turnover among reviewers and team members frequently move between teams that are monitored by different reviewers.

[0005] While automated data collection via in situ sensors can increase the number of non-operational periods that can be reviewed, non-operational periods present unique data management challenges, as the same or similar data available during operational periods can no longer be available. For example, a reviewer can be able to review hand instrument kinematics data or robotic surgical system event data captured during a surgical procedure, whereas these data sets would not be available during non-operational periods when such equipment is typically inactive or not present. As inefficiencies during non-operational preparation phases can exacerbate changes in downstream operational period instrument kinematics data, it can be very difficult to discern this causality by manually inspecting different data sets in isolation.

[0006] Accordingly, there is a need for systems and methods that overcome challenges and difficulties such as those described above. For example, there is a need for systems and methods that process different forms of surgical site data collected during non-operational periods to facilitate review by a reviewer based on team member inefficiencies identified therein and to generate feedback. BRIEF DESCRIPTION OF DRAWINGS

[0007] The various embodiments introduced herein can be better understood with reference to the following detailed description together with the accompanying drawings, in which like reference numerals refer to like elements / elements and in which: FIG. 1A is a schematic illustration of various elements present in a surgical site during a surgical operation, as can occur with respect to some embodiments; FIG. 1B is a schematic illustration of various elements present in a surgical site during a surgical operation with a robotic surgical system, as can occur with respect to some embodiments; FIG. 2A is a schematic depth map presented from an example full-site sensor perspective, as can be used in some embodiments; FIG. 2B is FIG. 2A is a schematic top view of objects within the site with corresponding sensor locations; FIG. 2C is a pair of images depicting an orthogonal grid pattern in perspective from a full-site visual image sensor with a straight view and a full-site visual image sensor with a fisheye view, each of which can be used in conjunction with some embodiments; FIG. 3 is a schematic illustration of a series of surgical procedures within a surgical site, non-operational periods between procedures, and a corresponding full-site sensor data set for one such non-operational period, as can occur in conjunction with some embodiments; FIG. 4 is a schematic block diagram illustrating an example deployment topology for a non- operational period analysis system, as can be implemented in some embodiments; FIG. 5A is a schematic illustration of a collection of index intervals, as can be used to assess non-operational team performance in some embodiments; FIG. 5B is a schematic process diagram indicating the relationship of various intervals, including intervals from FIG. 5A , over the course of a day, as can be applied in some embodiments; FIG. 5C is a schematic block diagram indicating possible activity analysis category groupings, as can be used in conjunction with some embodiments; FIG. 6 is a table of example task action time definitions, as can be used in some embodiments; FIG. 7 is a table of additional example task action time definitions, as can be used in some embodiments; FIG. 8 is a schematic block diagram illustrating various metrics and their relationships in constructing a composite score, referred to as an Operating Room Analysis "ORA" score, as can be used in some embodiments; FIG. 9A is a schematic block diagram depicting a general non-operational analysis system process flow, as can be implemented in some embodiments; FIG. 9B is a schematic block diagram depicting elements in an example non-operational analysis system process flow in more detail than the flow depicted in FIG. 9A , as can be implemented in some embodiments; FIG. 9Cis a flow diagram illustrating various operations in an exemplary overall process of analyzing full-site sensor data during non-operation periods, as can be implemented in some embodiments; FIG. 10 is a flow diagram illustrating various operations in an exemplary non-operation segment detection process, as can be performed in some embodiments; FIG. 11A is a schematic block diagram illustrating an exemplary information processing flow for performing object detection, as can be used in connection with some embodiments; FIG. 11B is a flow diagram illustrating various operations in an exemplary process of performing object detection, as can be used in some embodiments; FIG. 12A is a schematic block diagram illustrating an exemplary object tracking information processing flow, as can be used in connection with some embodiments; FIG. 12B is a flow diagram illustrating various operations in an exemplary process of performing object tracking, as can be used in connection with some embodiments; FIG. 13A is a pair of schematic visual images and depth frame full-site data pairs from a full-site data video, with indications of optical flow derived correspondences, as can be used in some embodiments; FIG. 13B is a schematic top view of the scene depicted in FIG. 13A FIG. 13C is a pair of schematic visual images showing team member motion moving away from and toward an imaging sensor; FIG. 13D is a schematic top view depicting team member motion presented in the visual images of FIG. 13C FIG. 14 is a flow diagram illustrating various operations in an exemplary process of performing motion analysis of non-operation periods from full-site data, as can be used in connection with some embodiments; FIG. 15 is a flow diagram illustrating various operations in an exemplary process of performing clustering and anomaly determination analysis based on indicator values, such as the indicator values disclosed herein, as can be performed in some embodiments; FIG. 16 is a flow diagram illustrating various operations in an exemplary process of providing guidance feedback based on determined indicator values, as can be performed in some embodiments; FIG. 17 is a schematic representation of graphical user interface (GUI) elements in an exemplary dashboard interface layout for non-operation indicator quick review, as can be implemented in some embodiments; ​​FIG. 18A is a schematic representation of a GUI element in an example global non-operational indicator quick review dashboard, as can be implemented in some embodiments; FIG. 18B is a schematic representation of an arrow graphical element, as can be used in some embodiments, for example FIG. 18A in the element of FIG. 9; FIG. 18C is a schematic representation of an example global non-operational indicator quick review dashboard layout, as can be implemented in some embodiments; FIG. 19A is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 19B is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 20A is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 20B is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 21A is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 21B is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 21C is a chart of example interval indicator values collected in connection with an example prototype implementation of an embodiment; FIG. 22 is a schematic representation of an example GUI element for providing feedback derived from indicators, as can be used in some embodiments; and FIG. 23 is a block diagram of an example computer system, as can be used in connection with some embodiments.

[0008] The specific examples depicted in the drawings are chosen for purposes of illustration. Thus, the disclosed embodiments should not be limited, in scope to the specific details shown in the drawings and corresponding descriptions. For example, the drawings can not be to scale, the dimensions of some elements in the figures can have been adjusted to facilitate understanding, and the operations of embodiments associated with flowcharts can include more, fewer, or alternative operations than those depicted. Thus, some components and / or operations can be separated into different blocks or combined into a single block in a manner different from what is depicted. It is intended that embodiments encompass all modifications, equivalents, and alternatives falling within the scope of the disclosed examples, and are not limited to the specific examples described or depicted. DETAILED DESCRIPTION

[0009] Exemplary surgical suite overview FIG. 1A is a schematic of various elements present within a surgical operating site 100a during a surgical operating procedure, as can occur with respect to some embodiments. In particular, FIG. 1A depicts a non-robotic surgical operating site 100a in which a patient-side surgeon 105a performs a procedure on a patient 120 with assistance from one or more assisting members 105b, who can themselves be surgeons, physician assistants, nurses, technicians, etc. The surgeon 105a can perform the procedure using various tools, such as visualization tools 110b (such as laparoscopic ultrasound, visual image-capturing endoscopes, etc.) and mechanical instruments 110a (such as scissors, retractors, dissectors, etc.).

[0010] The visualization tools 110b provide the surgeon 105a with an internal view of the patient 120, for example by displaying a visualization output from an imaging device that is mechanically and electrically coupled with the visualization tools 110b. The surgeon can view this visualization output, for example through an eyepiece coupled with the visualization tools 110b, or on a display 125 configured to receive this visualization output. For example, when the visualization tools 110b are visual image-capturing endoscopes, the visualization output can be color or grayscale images. The display 125 can allow the assisting members 105b to monitor the progress of the surgeon 105a during the surgical procedure. The visualization output from the visualization tools 110b can be recorded and stored for subsequent review, for example using hardware or software on the visualization tools 110b themselves, capturing the visualization output in parallel with providing it to the display 125, or capturing the output from the display 125 after it appears on the screen, etc. While primarily discussed herein in terms of two-dimensional video capture using the visualization tools 110b (such as when the visualization tools 110b are visual image endoscopes), it should be understood that in some embodiments the visualization tools 110b can capture depth data, in addition to or instead of two-dimensional image data (for example through laser rangefinders, stereo vision, etc.).

[0011] A single surgical procedure can include performing several groups of actions, each group forming a discrete unit, referred to herein as a task. For example, locating a tumor can constitute a first task, excising the tumor is a second task, and closing the surgical site is a third task. Each task can include multiple actions, for example the tumor excision task can require several cutting actions and several cauterizing actions. While some surgical procedures require that tasks have a particular order (e.g., excision occurs before closing), the order and existence of some tasks can be allowed to vary in some surgical procedures (e.g., canceling a prophylactic task, or reordering the excision task where the order does not have an impact). Transitions between tasks can require the surgeon 105a to remove a tool from the patient, replace a tool with a different tool, or introduce a new tool. Some tasks can require the visualization tool 110b to be removed and repositioned relative to its position in a previous task. While some assisting members 105b can assist with tasks related to the surgical procedure, such as administering anesthesia 115 to the patient 120, assisting members 105b can also replicate these task transitions, for example anticipating the need for a new tool 110c.

[0012] Advances in technology have also allowed programs such as those depicted in FIG. 1A to be performed by robotic systems, and to perform procedures that cannot be performed in non-robotic surgical sites 100a. In particular, FIG. 1B is to employ robotic surgical systems such as the da Vinci® Surgical System (Intuitive Surgical, Inc., Sunnyvale, CA) to perform procedures such as those depicted in TMA schematic view of various elements present within a surgical site 100b during a surgical procedure performed by a surgical system, as can occur with respect to some embodiments. Here, a patient-side cart 130 (which has tools 140a, 140b, 140c, and 140d attached to each of a plurality of arms 135a, 135b, 135c, and 135d, respectively) can replace the patient-side surgeon 105a. As before, one or more of the tools 140a, 140b, 140c, and 140d can include a visualization tool (here, the visualization tool 140d), such as a visual image endoscope, a laparoscopic ultrasound, etc. An operator 105c (who can be a surgeon) can view the output of the visualization tool 140d through a display 160a on a surgeon's console 155. By manipulating handheld input mechanisms 160b and pedals 160c, the operator 105c can remotely communicate with the tools 140a-140d on the patient-side cart 130, thereby performing a surgical procedure on the patient 120. In fact, the operator 105c can or can not be in the same physical location as the patient-side cart 130 and the patient 120, as communication between the surgeon's console 155 and the patient-side cart 130 can occur across a telecommunication network in some embodiments. The electronics / console 145 can also include a display 150 for displaying patient vital signs and / or output of the visualization tool 140d.

[0013] Similar to the task transition of the non-robotic surgical site 100a, the surgical procedure of the site 100b can require removal or replacement of the tools 140a-140d, including the visualization tool 140d, and introduction of new tools (e.g., new tools 165). As before, one or more assisting members 105d can now anticipate such changes and collaborate with the operator 105c to make the necessary adjustments during the surgical procedure.

[0014] As with the non-robotic surgical site 100a, output from the visualization tool 140d can also be recorded here, e.g., at the patient side cart 130, the surgeon console 155, from the display 150, etc. While some of the tools 110a, 110b, 110c in the non-robotic surgical site 100a can record additional data (such as temperature, motion, conductivity, energy levels, etc.), the presence of the surgeon console 155 and the patient side cart 130 in the site 100b can facilitate the recording of much more data than just the output from the visualization tool 140d. For example, the manipulation of the hand-held input mechanism 160b by the operator 105c, activation of the foot pedal 160c, eye movement relative to the display 160a, etc. can all be recorded. Similarly, the patient side cart 130 can record tool activation (e.g., application of radiant energy, closure of scissors, etc.), movement of instruments, etc. throughout the surgical procedure. In some embodiments, the data can be recorded using an in-site recording device, which can capture and store sensor data at a local or networked location (e.g., software, firmware, or hardware configured to record surgeon kinematics data, console kinematics data, instrument kinematics data, system event data, patient status data, etc. during a surgical procedure).

[0015] The computer systems 190a and 190b can be in network communication with the sites 100a, 100b, respectively, within each of these sites, or from an external location (in some embodiments, the computer system 190b can be integrated with a robotic surgical system, rather than as a standalone workstation). As will be discussed in greater detail below, the computer systems 190a and 190b can facilitate, e.g., data collection, data processing, etc.

[0016] Similarly, many of the sites 100a, 100b can include sensors arranged throughout the site, such as sensors 170a and 170c, respectively, configured to record activity within the surgical site from the perspective of their respective fields of view 170b and 170d. The sensors 170a and 170c can be, for example, visual image sensors (such as color or grayscale image sensors), depth acquisition sensors (such as pairs of visual images acquired via stereoscopy, via time-of-flight with laser rangefinders, structured light, etc.), or a combination of visual image and depth acquisition sensors (such as a red-green-blue-depth (RGB-D) sensor). In some embodiments, the sensors 170a and 170c can also include audio acquisition sensors, or sensors specifically for audio acquisition can be arranged throughout the site. Multiple such sensors can be arranged within the sites 100a, 100b, which can have overlapping fields of view and sensing ranges, to enable more comprehensive assessment of the surgical procedure. For example, depth acquisition sensors can be strategically arranged throughout the site such that the depth frames they generate at each instant can be integrated into a single three-dimensional virtual element model that depicts the objects within the surgical site. Similarly, sensors can be strategically arranged in the site to focus on regions of interest. For example, sensors can be attached to the display 125, the display 150, or the patient-side cart 130, to focus the field of view on the surgical site of the patient 120, or to a wall or ceiling, etc. Similarly, sensors can be arranged on the console 155 to monitor the operator 105c. Sensors can also be arranged on a movable platform that is specifically designed to facilitate orienting the sensors in various poses within the site.

[0017] For clarity, as used herein, a "pose" refers to the translational position and rotational orientation of a subject. For example, in three-dimensional space, a pose can be expressed with a total of six degrees of freedom. It will be appreciated that a pose can be expressed using a variety of data structures, such as using a matrix, a quaternion, a vector, or a combination thereof, among others. Thus, in some cases, a pose can include only translational components when there is no rotation. Conversely, a pose can include only rotational components when there is no translation.

[0018] For clarity, herein, "full-site" sensor data refers to data collected from one or more sensors configured to monitor a particular area of a site (the area encompassing all or a portion of the site) that is external to a patient, personnel, equipment, or any other object within the site such that the sensor can sense the presence of the patient, personnel, equipment, or other object within at least a portion of the area or passing through at least a portion of the area throughout the surgical procedure. Sensors configured to collect such "full-site" data are referred to herein as "full-site sensors." For clarity, it should be understood that the particular area need not be strictly fixed throughout the procedure, e.g., some sensors can periodically pan their field of view to expand the size of the particular area, although this can result in temporary gaps in the portion of the area in the sensor's data (which can be made up by coordinated panning or fields of view of other nearby sensors). Similarly, in some cases, personnel or robotic systems can reposition full-site sensors throughout the procedure, changing the particular area, e.g., in order to better capture different tasks. Thus, sensors 170a and 170c are full-site sensors configured to produce full-site data. "Visualization data" refers herein to visual or depth image data captured from a sensor. Thus, visualization data can or can not be full-site data. For example, the visualization data captured at sensors 170a and 170c is full-site data, whereas the visualization data captured via visualization tool 140d will not be full-site data (at least because the data is not from outside the patient).

[0019] Exemplary full-suite sensor topology To further clarify the deployment of full-site sensors, FIG. 2A is a schematic depth map presented from an example full-site sensor perspective 205, as can be used in some embodiments. In particular, this example depicts depth values corresponding to electronics / console 205a (e.g., electronics / console 145) and nearby tray 205b and cabinet 205c. Depth values associated with first technician 205d are also within the field of view, who is currently adjusting a robotic arm (associated with depth value 205f) on a robotic surgical system (associated with depth value 205e). Team members with corresponding depth values 205g, 205h, and 205i are also present within the field of view, as is a portion of the surgical table 205j. Depth values 2051 corresponding to a mobile cart and 205k with a boom having a lighting system are also present within the field of view.

[0020] The full-site sensor capturing perspective 205 can simply be one of several sensors arranged about the site. For example, FIG. 2Bis an illustrative top-down view of objects in the place at a particular moment during a surgical procedure. In particular, the perspective 205 can be captured via a full-place sensor 220a with a corresponding field of view 225a. Thus, for clarity, the cabinet depth value 205c can correspond to the cabinet 210c, the electronics / console depth value 205a can correspond to the electronics / console 210a, and the tray depth value 205b can correspond to the tray 210b. The robotic system 210e can correspond to the depth value 205e, and each of the individual team members 210d, 210g, 210h, and 210i can correspond to the depth values 205d, 205g, 205h, and 205i, respectively. Similarly, the cart 2101may correspond to the depth value 2051. The depth value 205j can correspond to the operating table 210j (here, the outline of the patient is shown for clarity, although the patient has not yet been arranged on the operating table corresponding to the depth value 205j in the example perspective 205). For clarity, a top-down representation of the boom corresponding to the depth value 205k is not shown, but it will be appreciated that this boom can likewise be considered in various embodiments.

[0021] As indicated, each of the sensors 220a, 220b, 220c is associated with a different field of view 225a, 225b, and 225c, respectively. The fields of view 225a-225c can sometimes have complementary characteristics, providing different perspectives of the same object, or providing a view of an object from one perspective when it is not in another perspective or is occluded in another perspective. The complementarity between perspectives can be dynamic, both in space and in time. This dynamic characteristic can result from movement of tracked objects, but also from movement of intermediary occluding objects (and, in some cases, from movement of the sensors themselves). For example, in the example perspective 205, the individual team member 210d is occluded from the perspective of the full-place sensor 220a by the individual team member 210g. However, the individual team member 210d is visible from the perspective of the sensor 220b, which has a narrower field of view 225b than the field of view 225a of the full-place sensor 220a. Thus, the individual team member 210d is visible in the depth value 205d, but not in the depth value 205d. FIG. 2A and FIG. 2BAt the depicted moment, field of view 225a has only a limited view of surgical table 210j because electronics / control console 210a largely obstructs this portion of field of view 225a. Thus, at the depicted moment, field of view 225b can have a better view of surgical table 210j. However, neither field of view 225b or 225a can have a sufficient view of operator 210n in console 210k. To observe operator 210n (e.g., when they remove their head in accordance with a "head away" event), field of view 225c can be more suitable. However, during the data capture process, these complementary relationships can change. For example, prior to the start of the procedure, electronics / control console 210a can be removed and robotic system 210e moved to position 210m. In this configuration, field of view 225a can be more suitable than field of view 225b for observing patient surgical table 210j. As another example, moving console 210k to the current depicted posture of electronics / control console 210a can make field of view 225a more suitable than field of view 225c for observing operator 210n. Thus, the suitability of a field of view can depend on the amount and duration of obstructions, the quality of the field of view (e.g., how close the object of interest is to the sensor), and the movement of the object of interest within the site. Such changes can be temporary and of short duration, such as when team members move within the site temporarily obstructing the sensors, or long term or persistent, such as when equipment is moved to fixed positions for the duration of the procedure.

[0022] As previously noted, the full-site sensors can take various forms and, for example, can be configured to capture visual image data, depth data, both visual and depth data, etc. It will be appreciated that the visual and depth image capture can likewise take various forms, for example, to improve visibility of different portions of the site. For example, FIG. 2Care a pair of images 250b, 255b that depict an orthogonal grid of rows and columns in perspective from a full-field sensor with rectilinear view and a full-field sensor with fisheye view, respectively. More specifically, some full-field sensors can capture rectilinear vision images or rectilinear depth frames, e.g., via appropriate lenses, post-processing, combinations of lenses and post-processing, etc., while other full-field sensors can capture fisheye or distorted vision images or rectilinear depth frames, e.g., via appropriate lenses, post-processing, combinations of lenses and post-processing, etc. For clarity, image 250b depicts a checkerboard pattern in perspective from a rectilinear full-field sensor. Thus, the orthogonal rows and columns 250a shown in perspective here maintain a linear relationship with their vanishing points. In contrast, image 255b depicts the same checkerboard pattern in the same perspective but from a fisheye full-field sensor perspective. Thus, the orthogonal rows and columns 255a, while maintaining a linear relationship with their vanishing points in reality (as they appear in image 250b), appear in the sensor data with a curvilinear relationship with their vanishing points. Thus, in connection with various embodiments, each type of sensor, as well as other sensor types, can be used individually or, in some cases, in combination.

[0023] Similarly, it should be appreciated that not all sensors can capture perfect rectilinear, fisheye, or other desired mappings. Thus, a checkerboard pattern or other calibration fiducial, such as a known shape for a depth system, can facilitate determining the intrinsics of a given full-field sensor. For example, the focal point of a fisheye lens, as well as other details of a full-field sensor (principal point, distortion coefficients, etc.) can vary between devices and even the same device at different times. Thus, various processing methods can need to be recalibrated for a particular device at an organization, anticipating device variation when training and configuring systems for machine learning tasks. Moreover, it should be appreciated that once the intrinsics of a camera are known, a rectilinear view can be obtained by de-distorting a fisheye view (e.g., this can facilitate normalizing different sensor systems into a similar form recognized by a machine learning architecture). Thus, while a fisheye view can allow a system and user to perceive a wider field of view more easily than in a rectilinear perspective, when a processing system considers data from some sensors that capture de-distorted perspectives and other sensors that capture distorted perspectives, the different perspectives can be normalized into a common perspective form (e.g., mapping all rectilinear data to a fisheye representation, or vice versa).

[0024] Exemplary surgical suite non-operative data As discussed previously, a nuanced and meaningful assessment of team member actions and performance during non-operational periods within a site can uncover opportunities to improve efficiency and avoid inefficient behaviors that can impact downstream operational and non-operational periods. To provide context, FIG. 3The description depicts the state of a single operating room over time 305 (e.g., a whole day). In this example, during the initial pre-surgical period 310a, the team can prepare the operating room for the day's procedures, gather appropriate equipment, review scheduling tasks, etc. After performing the first surgery 315a of the day, a non-operational operating room period 310b follows, during which the team performs a handover from the operating room configuration for performing surgery 315a to the configuration for performing surgery 315b. This alternating non-operational and operational period can continue throughout the day; for example, here the non-operational operating room period 310c will follow the second surgery 315b, etc. After the last procedure of the day 315c is performed, the team can perform any final maintenance operations during the post-operational period 310d, such as securing and packing equipment, deactivating equipment, uploading data, etc. The ellipsis 310e indicates the possibility of additional intermediate operations and non-operational states (of course, in some locations, there may only be one surgery per day). Due to the sequential nature of operations in a location, errors in upstream periods can cause errors and delays to propagate through downstream periods. For example, improper equipment alignment during the pre-operative period 310a can lead to delays during surgery 315a. This delay itself may necessitate shortening the non-operational period 310b, leaving team members insufficient time to perform appropriate cleaning procedures, thus putting the patient's health at risk during surgery 315b. Therefore, inefficiency early in the day can lead to delays, poor execution, or rescheduling of downstream actions. Conversely, efficiency early in the day can provide tolerance for downstream contingencies, promoting more predictable operational outcomes and other benefits.

[0025] Each of the site states (including both operational periods 315a, 315b, etc. and non-operational periods 310a, 310b, 310c, 310d, etc.) can be divided into a set of tasks. For example, non-operational period 310c can be divided into tasks 320a, 320b, 320c, 320d, and 320e (intermediate tasks are indicated by ellipsis 320f). In this example, there are at least three site-wide sensors in the operating room, each capturing at least visual image data (however, it should be understood that there may be fewer than three streams, or more, as indicated by ellipsis 370q). Specifically, the first all-field sensor captures a set 325a of visual images (e.g., visual image video) during the first non-operational task 320a, a set 325b of visual images during the second non-operational task 320b, a set 325c of visual images during the third non-operational task 320c, a set 325d of visual images during the fourth non-operational task 320d, and a set 325e of visual images during the last non-operational task 320e (again, it may be an intermediate set of frames acquired for other tasks, as indicated by ellipsis 325f).

[0026] At the same time, during each task of the second non-operation period 310c, the second full-site sensor can collect data sets 330a-330e (ellipsis 330f depicts possible intervening sets), and the third full-site sensor can collect sets 335a-335e (ellipsis 335f depicts possible intervening sets). Thus, it should be understood that, for example, the data in sets 325a, 330a, and 335a can be collected by the three full-site sensors simultaneously during task 320a (similarly for each column of collected data in the other columns associated with each respective non-operation task). Again, while visual images are shown in this example, it should be understood that other data, such as depth frames, can also be collected in each set instead or in addition.

[0027] Thus, in task 320a (which can be an initial "clean-up" task following surgical procedure 315b), the sensors associated with sets 325a-325e depict the team members and patient from a first perspective. In contrast, the sensors capturing sets 335a-335e are located on the opposite side of the site and provide a fisheye view from a different perspective. Thus, the second sensor has a more limited view of the patient. The sensors associated with sets 330a-330e focus on the patient, however, the perspective of this sensor in set 330a does not well depict the team members, while set 325a can provide a clear view of the team members.

[0028] Similarly, in task 320b (which can be a "back-out" task in which the robotic system is moved away from the patient), the full-site sensors associated with sets 330a-330e depict the patient no longer under anesthesia, but do not depict the state of the team members repositioning the robotic system. In contrast, sets 325b and 335b each depict the team members as well as the new pose of the robotic system at a point away from the patient and surgical table (although the sensors associated with the flow sets 335a-e are better positioned to observe the robotic system in the back-out pose).

[0029] In task 320c (which can be a "hand-off" or "patient departure" task), the team members escort the patient out of the operating room. While the full-site sensor associated with set 325c has a clear view of the departing patient, the full-site sensor associated with set 335c can be too far away to observe the departure in detail. Similarly, set 330c only indicates that the patient is no longer on the surgical table.

[0030] In task 320d (which can be a "set-up" task), the team members position equipment to be used in the next operational period (e.g., the last surgical procedure 315c if there are no intervening periods in ellipsis 310e).

[0031] Finally, in task 320e (which can be a "sterile preparation" task prior to the start of the next surgical procedure (again, for example, surgical procedure 315c) and prior to the initial port placements), the full-site sensors associated with set 330e can perceive the pose of the robotic system and its arms, as well as the state of the new patient. In contrast, sets 325e and 335e can provide more broad situational information about the state of the site.

[0032] Thus, the overall benefits of multiple sensor perspectives should be appreciated, as the combined view of streams 325a-325e, 330a-330e, and 335a-335e can provide overlapping situational awareness. Again, as noted previously, not all sensors can collect data in exactly the same way. For example, the sensors associated with sets 335a-335e can collect data from a fisheye perspective, while the sensors associated with sets 325a-325e and 330a-330e can collect straight-line data. Similarly, there can be fewer or more full-site sensors and streams than depicted here. In general, since each set is time-stamped, the review system can correlate the representations of the respective streams, even if they are in different forms. Thus, data for different areas of the site can be reconciled and reviewed. Unfortunately, as noted previously, unlike time periods 315a-315c, the surgical instruments, robotic systems, etc. can no longer capture data during the non-operational time periods (e.g., time periods 310a-310d). Thus, systems and reviewers accustomed to analyzing the large data sets available during time periods 315a-315c can find it particularly difficult to review time periods 310a-310d, as they can need to rely only on the different full-site streams 325a-325e, 330a-330e, and 335a-335e. Even if the reader can have appreciated this while considering the figure, manually reconciling different but contemporaneously captured perspectives can place a cognitive burden on human reviewers.

[0033] Exemplary non-operative activity data processing overview Various embodiments employ a processing pipeline to facilitate analysis of non-operative periods, and can include methods that facilitate iterative improvement of performance of surgical teams during these periods. In particular, some embodiments include a computer system configured to automatically measure and analyze non-operative activity in a surgical operating room, and recommend customized actionable feedback to operating room staff or hospital management based on historical dataset patterns, for example to improve workflow efficiency. Such systems can also help hospital management assess the impact of new personnel, equipment, facilities, etc., and extend their review to a larger number and more diverse types of surgical sites and procedures, thereby reducing variability in workflow. As discussed, various embodiments can be applied to surgical sites with multiple modalities, such as robotic, non-robotic laparoscopic, non-robotic open, etc. The various methods disclosed are not limited to non-operative periods associated with particular types of surgical procedures (e.g., prostatectomy, cholecystectomy, etc.).

[0034] FIG. 4 is a schematic block diagram of an exemplary deployment topology 450 of a non-operative period analysis system of a particular embodiment. As described herein, during real-time capture 450a, data can be collected from one or more full-site sensors at one or more perspectives. A multi-modality (e.g., visual image and depth) sensor suite within a surgical site (whether robotic or non-robotic) produces a wide variety of data. As described herein, integrating this data into basic and comprehensive OR metrics can more easily facilitate analysis. To determine these metrics, the data can be provided to a processing system 450b (described in more detail below) to perform automated inferences 450c, including detecting objects (such as personnel and equipment) within the site, and segmenting the full-site data into different steps 450d (e.g., which can correspond to the groupings and their respective actions discussed herein with reference to FIG. 4). Discretizing the full-site data into steps 450d can facilitate more meaningful and more granular determination of metrics from the full-site data via various workflow analyses 450e, for example to determine surgical site efficiency, provide actionable coaching recommendations, etc. FIG. 5A-5C

[0035] ​After generating such metrics during workflow analysis 450e, the embodiments also disclose software and algorithms for presenting metric values ​​to users and for detecting anomalies within metric values ​​relative to historical patterns. These analytical results can then be used to provide guidance and feedback via various applications 450f. Software applications 450f can present the various metrics and derived analyses disclosed herein in various interfaces as part of executable feedback, a more rigorous and comprehensive solution than using only human reviewers. It should be understood that such applications can be provided on any suitable computer system, including desktop applications, tablets, augmented reality devices, etc. This feedback can lead to adjustments in site operations, allowing subsequent applications of steps 450a-450f to identify new or more subtle inefficiencies in the team's workflow. Therefore, the cycle can continue again, enabling iterative, automated OR workflow analysis to facilitate incremental improvements in team performance, allowing the team to make contextual adaptations based on the corresponding adjustments. This iterative application can also help reviewers better track the impact of feedback on the team, analyze the effects of changes on site composition and scheduling, and enable the system to consider historical patterns in future evaluations and metric generation.

[0036] Exemplary non-operative interval division FIG. 5A This is a schematic representation of a set of indicator intervals, which, in some embodiments, can be used to evaluate the performance of non-operational teams. It should be understood that these intervals can be referenced as described above. FIG. 3The alternation of operational and non-operational periods within the described space is applied periodically. For example, initially, surgical operation 315b may correspond to interval 550e. After operation 315b is completed, the actions and corresponding data within the space can be assigned to consecutive intervals 550a-550d during the subsequent non-operational period 310c. Data and actions in the next surgical operation (e.g., surgical operation 315c if there is no intermediate period in ellipsis 310e) can again be categorized into a second instance of interval 550e, and so on (therefore, data from each of the non-operational periods 310b will be assigned to instances of intervals 550a-550d). Intervals can also be grouped into larger intervals, as is the case here with the "push-to-push" interval 550f, which groups intervals 550b and 550c, sharing the start time of interval 550b and the end time of interval 550c. As described in more detail herein, integrating site-wide data into this taxonomy, combined with the various other operations disclosed herein, can more easily facilitate analysis in a manner suitable for more efficient review. For example, organizing data in this way can facilitate comparisons across different days of the month, across sites, surgical configurations (robotic and non-robotic), and teams, with particular focus on specific intervals within these intervals 550a-550d that occur during corresponding non-operational periods. Although not part of the non-operational period, determining the duration of surgical procedures within interval 550e may still be useful in some embodiments, as this duration can indicate the efficiency or inefficiency of non-operational periods before or after them. Therefore, in some embodiments, some of the disclosed metrics may take into account events and actions within interval 550e, even if the ultimate goal is to assess the efficiency of non-operational periods.

[0037] To help readers understand more clearly, FIG. 5B It indicates that it comes from FIG. 5A A schematic diagram illustrating the relationships between the elements throughout the day. Specifically, as discussed earlier, FIG. 5A Instances of intervals in can be based on FIG. 3Alternating operational and non-operational periods are created periodically. In some embodiments, when considering full-day data (e.g., including pre-operational non-operational period 310a, post-operational non-operational period 310d, and data from all intermediate periods), the system can accordingly anticipate an initial interval “start of day to patient entry” 555a to account for actions within the pre-operational period 310a. This interval may, for example, begin from the first person entering the facility that day and end with the patient entering the facility for the first surgical procedure. Thus, as indicated by arrow 555c in the figure, this could lead to the first instance of transitioning to the “patient enters skin incision” interval 550d. From there, as indicated by the cyclical relationship, data can be periodically grouped into instances of intervals 550a-550e, for example, according to alternating periods 315a, 310b, 315b, 310c, etc., up to period 315c.

[0038] After the last surgical procedure of the day (e.g., surgery 315c) and the last instance of interval 550a following that surgery, the system may cease additional periodic data allocation between instances of intervals 550a-550e and instead transition to the final "patient departure to end of day" interval 555b (which can be used to evaluate the post-operative non-operational period 310d), as indicated by arrow 555d in the figure. The "patient departure to end of day" interval 555b may end when the last team member leaves the site or when data collection ceases. It should be understood that various disclosed computer systems can be trained to distinguish actions in interval 555b from corresponding data in interval 550b (of course, in some embodiments, the end of the data stream can also be used to infer the existence of interval 555b). Although it is the end of an action in a day, in some embodiments, analysis of interval 555b may still be appropriate because actions taken at the end of the day may affect performance on the following day.

[0039] Exemplary task-to-interval assignment and action time interval In some embodiments, the duration of each of intervals 550a-550e can be determined based on the corresponding start and end times of various tasks or actions within the venue. Of course, when intervals 550a-550e are used consecutively, the end time of the preceding interval (e.g., the end of interval 550c) can be the start time of the following interval (e.g., the start of interval 550d). When coupled with a task-action grouping ontology, venue-wide data can be readily grouped into meaningful partitions for downstream analysis. This can facilitate, for example, feedback verifying whether team members are following recommendations, and the consistency of computer-based verification across different venues, team configurations, etc. As will be explained below, some task actions may occur over a period of time (e.g., cleaning), while other task actions may occur at a specific moment (e.g., a team member's entry).

[0040] Specifically, FIG. 5C Four high-level task action categories are described: post-surgical 520, handover 525, pre-surgical 510, and surgical 515. Surgical 515 may include tasks or actions 515a-515i, as will be discussed below. FIG. 6 and FIG. 7Various exemplary time definitions for actions are provided, but a brief overview is given here for the reader's convenience. Specifically, task "first cut" 515a may correspond to the time of making the first incision on the patient (e.g., consider duration 605a). Task "port placement" 515b may correspond to the duration between the time when the first port is placed into the patient and the time when the last port is placed (e.g., consider duration 605b). Task "advancement" 515c may correspond to the duration during which a team member begins moving the robotic system to a position where the robotic system assumes a posture it will use in at least the initial part of the surgical procedure (e.g., consider duration 605c). Task "room preparation" 515d may correspond to the duration from the first surgical preparation action specific to the surgery being performed and may terminate at the last preparation action specific to the surgery being performed (e.g., consider duration 605d). Task "docking" 515e may correspond to the duration from when a team member begins docking the robotic system and terminates when the robotic system is docked (e.g., consider duration 605e). Task “Surgical Procedure” 515f can correspond to the duration from the start of the first incision to the patient’s final closure (e.g., consider the corresponding expected surgical procedures 705a-705c, specifically robotic surgery 705a and non-robotic surgery 705b and 705c). Of course, in many taxonomies, these action boxes can be further subdivided into even more action and task divisions depending on the analyst’s desired focus (e.g., if action “Port Arrangement” 515b is associated with inefficiency, a supplementary taxonomy could be used where each port arrangement is a distinct action with its own measured duration to refine the analysis). However, since non-operational actions are the subject of review here, the general task “Surgical Procedure” 515f (e.g., one of the durations 705a-705c) may suffice, even though surgery comprises many component actions. Task “Disengagement” 515g can correspond to the duration that begins when team members begin disengaging the robotic system and terminates when the robotic system is disengaged (e.g., consider duration 705d). Task “retreat” 515h can correspond to the time when team members begin to remove the robotic system from the patient and the duration during which the robotic system assumes a posture it will maintain until the handover begins (e.g., consider duration 705e). Task “patient closure” 515a can correspond to the duration during which the surgeon observes the patient during retreat (e.g., duration 705f) (e.g., it should be understood that, by this example, some action durations may overlap and occur in parallel).

[0041] Within the post-surgical category group 520, the task "Robotic undraping" 520a can correspond to the duration from the time a team member first begins undraping the robotic system to the time the robotic system is undraped (e.g., consider duration 705g). The task "Patient departure" 520b can correspond to the time or duration of the patient leaving the location (e.g., consider duration 705h). The task "Patient undraping" 520c can correspond to the duration that begins when a team member begins undraping the patient and ends when the patient is undraped (e.g., consider duration 705i).

[0042] Within the handover category group 525, task "Cleaning" 525a can correspond to a duration that begins when the first team member starts cleaning the equipment in the area and ends when the last team member (who may be the same team member) completes the final cleaning of any equipment (e.g., consider duration 705j). Task "Idle" 525b can correspond to a duration that begins when team members are not performing any other tasks and ends when they begin performing another task (e.g., consider duration 705k). Task "Handover" 505a can correspond to a duration that begins when the first team member begins resetting the area from the last procedure and ends when the last team member (who may be the same team member) completes the reset (e.g., consider duration 615a). Task "Setup" 505b can correspond to a duration that begins when the first team member begins changing the posture of equipment to be used in surgery and ends when the last team member (who may be the same team member) completes the final equipment posture adjustment (e.g., consider duration 615a). Task “Aseptic Preparation” 505c can correspond to a duration that begins when the first team member begins cleaning the surgical area and ends when the last team member (who may be the same team member) finishes cleaning the surgical area (e.g., consider duration 615c). Again, although shown in a linear order here, it should be understood that task actions within a category may be performed in a different order than shown, or in some cases, may refer to time periods that may overlap and proceed in parallel (e.g., when performed by different team members).

[0043] Within the pre-surgical category group 510, the task "Patient Entry" 510a can correspond to the duration of the start and end times of the patient's first entry into the facility (e.g., consider duration 620a). The task "Robotic Drooping" 510b can correspond to the duration that begins when a member begins draping the robotic system and ends when draping is complete (e.g., consider duration 620b). The task "Intubation" 510c can correspond to the duration that begins when the patient begins intubation and ends when intubation is complete (e.g., consider duration 620c). The task "Patient Preparation" 510d can correspond to the duration that begins when a team member begins preparing the patient for surgery and ends when preparation is complete (e.g., consider duration 620d). The task "Patient Drooping" 510e can correspond to the duration that begins when a team member begins draping the patient and ends when the patient is draped (e.g., consider duration 620e).

[0044] Although not discussed herein, it should be understood, as previously stated, that additional or different task actions may exist. For example, the durations of "imaging" 720a and "walking in" 720b, although not... FIG. 5C It is part of an exemplary taxonomy, but can also be determined in some embodiments.

[0045] Therefore, as FIG. 5C As shown by the corresponding arrow in the image, FIG. 5A The intervals can be assigned as follows. "Skin closure to patient departure" 550a can begin at the last closure operation 515j of the previous surgical interval and terminate when the patient leaves the area (e.g., from the end of the last suture in box 515i to the patient's departure in box 520b). Similarly, "patient departure to case opening" interval 550b can begin when the patient leaves the area in box 520b and terminate when aseptic preparation for the next case begins in box 505c.

[0046] The "Case Opening to Patient Entry" interval 550c can begin at the start of aseptic preparation in box 505c and terminate when a new patient begins entering the area in box 510a. The "Patient Entry to Skin Incision" interval 550d can begin at the start of a new patient entering the area in box 510a and terminate at the start of the first incision in box 515a. The surgical procedure itself can occur during interval 550e, as shown in the figure.

[0047] As mentioned earlier, the "pull-out to push-in" interval 550f is the interval that begins with "patient departure to case opening" 550b and ends with "case opening to patient entry" 550c.

[0048] Exemplary non-operative metric generation and scoring In identifying (e.g., using the references in this article) FIG. 9A-9C and FIG. 10 Following the non-operational segment (as discussed in this paper's references), the system and methods are identified (e.g., using the references in this paper). FIG. 9A-9C , FIG. 12A-12B , FIG. 13A-13D and FIG. 14 The system and methods discussed) The number and location of objects (such as people) within each segment (e.g., using references in this article) FIG. 9A-9C and FIG. 11A-11B Following the discussion of the systems and methods (and their corresponding movements), the system can generate one or more index values. As previously stated, FIG. 5A-5C The interval, category, and duration and relative time of the task action can themselves serve as indicators.

[0049] Various implementations can also determine the "comprehensive" score based on a variety of other defined indicators. These indicators are expressed as functions of Formula 1: s = f ( m (1) in s This refers to the comprehensive indicator score, which can be limited to a certain range, such as from 0 to 1, from 0 to 100, etc. f (·) indicates a mapping from individual indicators to a composite score. For example, m This can be a vector of metrics calculated using the various data streams and models disclosed herein. In some embodiments of such a comprehensive score, the constituent metrics may fall into one of the time workflows, scheduling, human resources, or other groupings disclosed herein.

[0050] Specifically, FIG. 8 This is a schematic block diagram illustrating the various metrics and their relationships when constructing an "ORA score," as can be performed in some embodiments. Within time grouping 805, the "efficiency" score metric 805a can combine non-operational metrics that measure the efficiency of time workflows in the operating room (OR), such as... FIG. 5A The duration of one or more of the six time interval indicators. More specifically, the average or median of non-operational indicators from all cases collected from a team, facility, or hospital can be compared with the top 20% of teams, facilities, or hospitals in the database as a benchmark (e.g., manually indicated by reviewers or indicated from historical patterns via iteration of topology 450). The "consistency" indicator 805b can be combined (e.g., summed or averaged or median) across all non-operational indicators (e.g., from the current team, facility, or hospital) collected from the current team, facility, or hospital. FIG. 5AThe standard deviation of the six time interval indicators. The "adverse event" indicator 805c can be combined (e.g., summed) with negative outliers, for example, according to... FIG. 5A-5B The interval index detects negative anomalies. Anomalies can be identified using statistical analysis algorithms (e.g., those referenced in this article). FIG. 15 , FIG. 16 , FIG. 19A-19B and FIG. 20A-20B The discussion covers clustering, distribution analysis, regression, etc., for detection. Negative anomalies can be identified as... FIG. 5A-5B At least one of the non-operational interval indicators differs from the median or mean of the relevant team, facility, or hospital by more than a threshold (such as standard deviation) (e.g., based on a threshold specified by an expert reviewer or based on historical patterns from previous iterations of topology 450). Examples of such anomalies are discussed in this paper, for example, in reference to FIG. 19A-19B and FIG. 20A-20B .

[0051] Within scheduling group 810, the “case volume” rating metric 810a includes the average or median number of cases per OR operation per day for a team, site, or hospital, normalized to the expected case volume of a typical OR (e.g., again as specified in the historical dataset benchmark, such as the average or median). The “first case handover” rating metric 810b is the ratio of the first case handed over during the operation day to the total number of first cases captured from the team, site, or hospital. Alternatively, a more general “case handover” metric is the ratio of all cases that were handed over to the total number of cases executed by the team, site, or hospital. The “delay” rating metric 810c is the average or median deviation of each case from the scheduled start time (in minutes) as either positive (later than the scheduled start time of the action) or negative (earlier than the scheduled start time of the action), normalized to an acceptable delay (e.g., a historical average or median benchmark). Of course, the definitions of negative or positive numbers can be reversed if other contextual parameters are adjusted similarly (e.g., where a delayed start is defined as negative and an early start as positive).

[0052] Within the Human Resources metrics group 815, the “Number of People Completing the Task” rating metric 815a combines the average or median number of people required to complete the non-operational task for each case at each time point across all cases collected from the team, site, or hospital (i.e., the maximum number of people detected within the OR at any given moment throughout the procedure), normalized by the recommended number of people for each task (e.g., historical baseline median or average). The “OR Flow” rating metric 815b measures the average amount of movement within the OR for each case, averaged across all cases collected from the team, site, or hospital (itself the median or average), and normalized by the recommended flow rate (e.g., based on the historical baseline described above). For example, this metric could receive (two-dimensional or three-dimensional) optical flow and convert such raw data into a single numerical value, such as an entropy representation, average amplitude, median amplitude, etc.

[0053] Within the “Other” metrics group 820, the “Room Layout” rating metric 820a includes the ratio of robotic cases with multipart advances or retreats normalized to the total number of robotic cases for a team, venue, or hospital. That is, ideally, each advance or retreat of the robotic system should involve a single movement. Conversely, this “multipart” rolling implies inefficiency when team members move the robotic system back and forth, and therefore the number of such multipart rollings relative to all advance and retreat events can provide an indication of the proportion of inefficient attempts. As this example indicates, some metrics may be specific to robotic venues, just as some metrics may be specific to non-robotic venues. In some embodiments, correspondences between venue-type-specific metrics can be specified to facilitate comparisons. The “Modal Conversion” rating metric 820b includes the ratio of cases with both robotic and non-robotic modes normalized to the total number of cases for a team, venue, or hospital. For example, this metric could calculate the number of conversions (e.g., a transition from a planned robotic configuration to a non-robotic configuration and vice versa) and then divide the total number of such cases with such conversions by the total number of cases. Regardless of whether the transition occurs during the operational or non-operational period, such transitions can reflect inefficiencies during the non-operational period (e.g., improper actions during a previous non-operational period can render a planned robotic procedure infeasible during the operational period). Therefore, this indicator can capture inefficiencies related to unexpected complications in the planning, equipment, or original surgical plan.

[0054] While each of the indicators 805a-805c, 810a-810c, 815a-815c, and 820a-820b can be considered individually to assess performance during non-operational periods, or multiple indicators can be combined (as discussed above with reference to Formula 1), some embodiments consider a “ORA score” 830 that reflects the overall sum of all these indicators 825. For example, when compared with… FIG. 5A-5CWhen data on durations of one or more intervals are presented together, ORA scores can provide reviewers with an easily identifiable way to quickly and intuitively assess the relative performance of surgical teams, surgical sites, hospitals, and hospital systems during non-operational periods across sites, teams, surgical procedure types (non-operational periods before and after prostatectomy, hernia repair, etc.), surgical modalities (non-operational periods for preparation or subsequent reset of non-robotic laparoscopic procedures, non-robotic open procedures, robotic procedures, etc.), and hospital systems.

[0055] Therefore, although some embodiments may employ more complex relationships between indicators 805a-805c, 810a-810c, 815a-815c, and 820a-820b (e.g., using any suitable mathematical functions and operations) when forming the ORA score 830, in this example, each of the indicators can be weighted by corresponding weighted values ​​850a-850j, such that the composite 825 is the weighted sum of each of the indicators. The weights can be selected, for example, by hospital administrators or reviewers based on which indicators are identified as more important to the current efficiency improvement needs. For example, in a system where reviewers want to evaluate reports on the impact of staff shortages on efficiency, weight 850g can be increased relative to other weights. Therefore, when the cross-procedure ORA score 830 is combined... FIG. 5A-5C When comparing the duration of one or more intervals across a group of surgical procedures, reviewers can more easily identify a relationship between the number of patients and the duration of undesirable intervals. Of course, other options and combinations of weighting adjustments, as well as specific considerations for particular interval durations, should be understood to assess other performance characteristics.

[0056] Exemplary metric scoring method - ORA significance assessment Some higher ORA composite scores can be correlated with increased system utilization of hospitals in the database. u and the decrease in OR minutes for each case t They are positively correlated, for example, as shown by Formula 2: Therefore, the ORA comprehensive score can be used for a variety of analytics and feedback applications. For example, the ORA comprehensive score can be used to detect negative trends and prioritize hospitals, facilities, teams, or team members that need workflow optimization. The ORA comprehensive score can also be used to monitor workflow optimization, such as to verify whether requested adjustments have been followed and whether the expected improvements have actually occurred. The ORA comprehensive score can also be used to provide an objective measure of efficiency when a team performs a new type of surgery for the first time.

[0057] Exemplary metric scoring method - additional metrics Additional metrics for evaluating workflow efficiency can be generated by combining time, number of staff, and motion metrics. For example, the composite score can consider scheduling efficiency (e.g., a combination of one or more of case volume 810a, first case handover 810b, and case delay 810c) and one or both of modality transition 820b and the duration of the “idle time” metric, where the “idle time” metric is the average or median idle time (individual member or collaborative team) over a period of time (e.g., during action 525b).

[0058] Although for convenience, it is sometimes described as considering the behavior of one or more team members, it should be understood that the metrics described herein can be used to compare the performance of individual members, teams, venues (across different teams and modalities), hospitals, hospital systems, etc. Similarly, metrics calculated at the individual, team, or hospital level can be aggregated for higher-level evaluations. For example, to compare hospital systems, metrics for team members within each system across hospitals can be determined, and then averaged (e.g., mean, median, weighted sum by team member characteristics, etc.) for inter-system comparisons.

[0059] Exemplary non-operative data processing workflow FIG. 9A This is a schematic block diagram depicting a general processing flow, as may be implemented in some embodiments. Specifically, this example flow employs various machine learning fusion systems to generate basic OR metrics (such as duration of time intervals, personnel presence, personnel movement, equipment movement, etc.) from raw multimodal, all-location sensor data, from which other metrics can be generated, such as... FIG. 8 (The indicators described in the text).

[0060] In some embodiments (e.g., where the data has not been preprocessed), non-operational segments can be detected from all-day, all-location data using the non-operational segment detection module 905a. Subsequently, the number of people involved in each detected non-operational segment / activity in the all-location data can be detected using the personnel number detection module 905b (e.g., a spatiotemporal machine learning algorithm that uses a 3D convolutional network to process time-varying visual images and depth data, such as data presented in a video). The motion assessment module 905c can then measure the amount of motion observed in each non-operational segment / activity (e.g., the motion of people, equipment, etc.) (e.g., using optical flow methods, machine learning tracking systems, etc.). Metrics can then be generated using the metric generation component 905d, such as the metrics disclosed herein (e.g., [the metric is missing here]). FIG. 5A-5C The duration of each interval and action, and FIG. 8The metrics discussed herein are defined as indicators. As described herein, while in some embodiments the indicator results can be presented directly to reviewers, some embodiments may alternatively provide some preliminary analysis and evaluation of the indicator values, determine the standard deviation relative to historical values, prioritize larger tolerance deviations for priority presentation to reviewers, and determine whether the indicator value (e.g., motion) indicates that a more granular analysis of the data is expected (e.g., determining team member movement paths, object collision event detection, etc.). Therefore, the indicator analysis component 905e can subsequently analyze the generated indicators, for example, to identify anomalies relative to historical patterns.

[0061] FIG. 9B It is a description FIG. 9A The process described herein is a schematic block diagram of elements in a more detailed example of the processing flow, as may be implemented in some embodiments. It should be understood that each depicted component may be a logical component, or may be one or more machine learning systems, as will be discussed in more detail below. Computer system 910b may receive full-field sensor data 910a and first perform non-operational period detection 910c (e.g., identifying periods 310a, 310b, 310c, 310d, although some systems may be configured to detect only non-operational periods with the characteristics of periods 310b and 310c). Once a portion of the full-field data corresponding to a non-operational period is detected, the data may be further segmented at block 910d into corresponding action tasks or intervals (e.g., intervals 550a-550d and / or groups 510, 515, 520, 525 and corresponding action tasks).

[0062] Using an object detection (and in some embodiments, tracking) machine learning system 910e, the system can perform object detection using machine learning methods, such as detecting equipment 910f or personnel 910h (ellipsis 910g indicates the possibility of other machine learning systems). In some embodiments, only personnel detection 910h is performed, as the desired metric only requires the number of personnel and their movements. The motion detection component 910i can then analyze the detected objects at box 910e to determine their corresponding movements, for example, using various machine learning methods disclosed herein, optical flow, or combinations thereof.

[0063] Using the number of objects, detected motion, and determined interval duration, the metric generation system 910j can generate metrics (e.g., the interval duration itself can be used as a metric). FIG. 8The values ​​in the data can be calculated, etc. The indicator values ​​can then be analyzed via component 910k to identify anomalies and other deviations, such as those deviating from historical data (e.g., previous iterations of topology 450). When performing historical comparisons at box 910k, the system can consider historical sensor data 915a, 915c, 915e, and historical indicator data 915f (e.g., clustering historical indicator values ​​around efficient and inefficient nodes, then assessing the distance of newly arriving data to these nodes). In this way, the system can infer that an entire team, group of members, or individual member is performing poorly in terms of historical indicator data compared to similar roles, team composition, or individual team members. Conversely, processed and raw full-site sensor data can be provided to historical data storage 915b 915e for future analysis. Similarly, indicator results and anomaly identifications can be recorded 915d in historical indicator database 915f for future reference.

[0064] The results of the analysis can then be presented to reviewers via component 910l (e.g., sent over a network to one or more applications 550f). For example, the application algorithm can utilize the identified metrics and non-operational data, and based on the metric analysis results, propose customized, actionable guidance for each individual in the team and the team as a whole (although in some embodiments, this guidance or feedback may first be determined on computer system 910b). Exemplary recommendations include, for example: changes to OR layout at various points in time, changes to OR scheduling, changes to communication systems between team members, changes to the number of employees involved in various tasks, etc. In some embodiments, this guidance and feedback can be generated by comparing metric values ​​with a finite corpus of known inefficiencies (or conversely, known efficient patterns) and corresponding proposed corrective actions (e.g., slow port placement and overstaffing may be related to inefficiencies addressed by reducing the number of personnel for that task).

[0065] To make it clearer, FIG. 9C This is a flowchart illustrating the various operations in an exemplary process 920 for analyzing full-area data. At box 920a, the computer system can receive full-area sensor data from the area to be inspected. At box 920b, the system can perform preprocessing on the data, such as harmonizing the full-area data into a common format, as is done when processing both fisheye and line sensor data simultaneously.

[0066] At box 920c, the system can perform operational and non-operational period identification, for example, identifying each of segments 310a-310d and 315a-315c from raw full-area sensor data. In some embodiments, this segmentation can be identified or verified via auxiliary data such as console data, machine kinematics data, etc. (e.g., active only during operational periods)).

[0067] The system can then iterate over the detected non-operational periods (e.g., periods 310a, 310b) at boxes 920d and 925a. In some embodiments, the iteration may also include operational periods, for example, to determine metric values ​​that can inform the analysis of non-operational periods, although many embodiments will only consider non-operational periods. For each period, the system can identify the relevant task and interval at box 925b, for example... FIG. 5A-5C The intervals, groups, and actions within.

[0068] At boxes 925c and 925e, the system can traverse the corresponding portions of the location data for the identified tasks and intervals, perform object detection at box 925f, motion detection at box 925g, and corresponding metric generation at box 925h. In some embodiments, at box 925f, only the number of people in the location can be determined, without knowing their roles or identities. Again, the metrics can therefore be generated at both the action task level and... FIG. 5A-5C Other interval generation described herein. In an alternative embodiment, the metric may be determined only for non-operational periods (e.g., where the duration of interval 550a-550e is the only metric to be determined).

[0069] After considering all relevant tasks and intervals for the current time period at box 925c, the system can create any additional indicator values ​​at box 925d (e.g., indicators that include values ​​determined across multiple tasks at box 925h as their component values). Once all time periods have been considered at box 920d, the system can perform overall indicator generation at box 930a (e.g., indicators whose component values ​​depend on the time period indicators of boxes 925d and 925h, such as certain composite indicators described herein).

[0070] At box 930b, the system can analyze the metrics generated at boxes 930a, 925d, and 925h. As discussed, many metrics (potentially at each of boxes 930a, 925h, and 925d) will take historical values ​​into account, for example, normalizing the specific values ​​at that location during their generation. Similarly, at box 930b, the system can identify anomalies by incorporating historical values ​​into the metric results (as described in more detail herein). Finally, at box 930c, the system can publish its analysis for use by, for example, application 450f.

[0071] Exemplary non-operative full-suite data processing - non-operative data recognition It should be understood that there are sufficient mechanisms to perform operational / non-operational period detection for components 905a or 910c and activity / task / interval segmentation (e.g., identification) for box 910d. FIG. 5A-5CVarious systems and methods (actions, tasks, or intervals). In practice, as previously described, in some embodiments, alternative signals in addition to the full-site data, or total signal monitoring within the full-site data, may be sufficient to distinguish between time periods 310a-310d and 315a-315d. For example, in some embodiments, team members can provide explicit notification. Similarly, the absence of kinematic and system event data from the robotic surgical system console or instrument can indicate a prolonged separation between the surgeon and patient or between the robotic platform and patient, which may be sufficient to indicate the commencement of a non-operational period within the surgical room (or to provide verification of parallel determinism by the machine learning system).

[0072] However, some embodiments alternatively or additionally consider employing machine learning systems to perform non-operational period detection. For example, some embodiments employ spatiotemporal model architectures, such as those similar to those described by Bertasius, Gedas, Heng Wang, and Lorenzo Torresani in “Is Space-Time Attention All You Need for VideoUnderstanding?” arXiv TM preprint arXiv TM The transformer architecture described in :2102.05095 (2021) can also be particularly useful for automated activity detection from long sequences of full-field sensor data. The spatial segment transformer architecture can be designed to learn features from frames of full-field data (e.g., visual image video data, depth frame video data, visual image and depth frame video data, etc.). The time segments can be based on a gated recurrent unit (GRU) method and are designed to learn action sequences within long videos, and can be trained, for example, in a fully supervised manner (again, where data labeling can be aided by activations from surgical instrument data). For example, the OR full-field data can first be annotated by human experts to create ground truth labels and then fed into the model for supervised training.

[0073] Some implementations may employ a two-stage model training strategy: first, a backbone transformer model is trained to extract features, and then a temporal model is trained to learn the sequence. The input to model training can be long-sequence full-field data (e.g., hours of visual image video), and the output can be each segment of interest (e.g., non-operational segments) or activity (e.g., ...). FIG. 5A-5C The timestamps (range and task). It should be understood that some models can operate on individual visual images, individual depth frames, groups of image frames (e.g., video segments), groups of depth frames (e.g., depth frame video segments), combinations of visual video and depth video, etc.

[0074] As another example, FIG. 10 This is a flowchart illustrating various operations in an exemplary process 1005 for performing non-operational period detection in some embodiments. Specifically, since the number of sensors across different locations or within the same location over time can vary, investing in training a machine learning system configured to receive only a specific number of across-location data inputs may not be desirable. Therefore, in these embodiments where the classifier is not configured to simultaneously consider across-location sensor data from all available streams, the system may instead consider these streams individually or in smaller groups and then analyze the collaborative results, such as incorporating smoothing operations, to assign a classification to the considered segments.

[0075] For example, after receiving full-field data at box 1005a (e.g., all three streams 325a-325e, 330a-330e, and 335a-335e), the system can traverse the data within the interval at boxes 1005b and 1005c. For example, the system can consider streams in continuous segments (e.g., 30-second, 1-minute, or 2-minute intervals), although the data within these segments may be downsampled based on the frame rate at which they were acquired. For each data interval, the system can traverse portions of the interval data associated with the stream from the corresponding sensor at boxes 1010a and 1010b (e.g., streams 325a-325e, 330a-330e, and 335a-335e, or each of their groups, possibly considering the same stream multiple times in different groups). For each stream, the system can determine whether the classification result belongs to an operational or non-operational interval at box 1010c. After considering all streams, the system can consider the final classification for that interval at box 1010d. For example, the system can perform majority voting on the individual flow classification of box 1010c, resolve ties, and smooth the results based on continuity with previous (and possibly subsequently determined) classifications.

[0076] After considering all field data at box 1005b, the system can integrate the classification results at box 1015a (e.g., performing smoothing and continuity reconciliation on all data, similar to what was discussed regarding box 1010d, but here for a larger smoothing window, such as 1 to 2 hours). At box 1015b, the system can perform any supplementary data validation before publishing the results. For example, if the supplementary data indicates time intervals with known classifications, hard-coded true positive classifications can be assigned, and smoothing can be rerun.

[0077] Exemplary non-operative full-suite data processing - object recognition Similar to the segmentation of non-operational and operational full-site data, it should be understood that there are multiple ways to perform object detection (e.g., at box 905b or component 910e). Again, in some embodiments, object detection only includes counting the number of people, and therefore it is possible to iteratively apply a You Only Look Once (YOLO) style network (e.g., as in "You Only Look Once: Unified, Real-Time Object Detection" by Redmon, Joseph et al.). arXiv TM Preprint arXiv TM :1506.02640 The network described in (2015) may suffice. However, some embodiments consider using visual images or groups of depth frames. For example, some embodiments employ a transformer-based spatial model to process frames of full-scene data, detecting all present humans and reporting their numbers. Carion, Nicolas, et al., in "End-to-End Object Detection with Transformers" arXiv TM Preprint arXiv TM :2005.12872 An example of this architecture is described in (2020).

[0078] To clarify this particular method, FIG. 11A This is a schematic block diagram illustrating an exemplary information processing flow that can be used in conjunction with some embodiments to perform object detection. Given a visual or depth frame image 1105f, the system can present the raw pixel or depth values ​​of the image to a convolutional network 1105a trained to generate image features 1105b. These features can then be provided to a transformer encoder-decoder 1105c, and the location and number of objects (e.g., people or equipment) in the image can be predicted 1105e using a binary matching loss 1105d, reflected here by bounding boxes within the augmented image 1105g (it should be understood that the system may not generate an actual augmented image, but only an indication of the object location, and in some embodiments an indication of the type of object found therein).

[0079] FIG. 11BThis is a flowchart illustrating various operations in an exemplary process 1100 for performing object detection, which may be used in conjunction with some embodiments. At block 1110a, the system may receive full-field data (visual image data, depth data, etc.). At blocks 1110b and 1110c, as in process 1005, the system may traverse non-operational periods, considering data in discrete continuous intervals (as previously mentioned, in some embodiments, operational periods may also be considered, for example, to verify the continuity of object detection and identification at the beginning or end of non-operational periods).

[0080] At boxes 1110d and 1115a, the system can consider the entire site data set. For example, some embodiments may consider each data capture moment, while others may consider every other capture or captures at certain intervals, because some site sensors may employ high data acquisition rates (in fact, not all sensors within a site may apply the same rate, and therefore normalization may be applied to consolidate the data). For such high rates, interpolating object positions between data captures may be unreasonable if the data capture rate is sufficiently greater than the movement speed of objects within the site. Similarly, data captures from some site sensors may not be perfectly synchronized, or may capture data at different rates, which necessitates the system interpolating or selecting data captures that are sufficiently corresponding in time to perform detection and index calculations.

[0081] At boxes 1115b and 1115c, the system can consider data from individual full-field sensor data streams and perform object detection at box 1115d, as described above. FIG. 11A As described, or using a YOLO network, etc. After performing object detection for each stream of the considered group, the system can perform post-processing at box 1115e. For example, if the relative poses of the sensors throughout the field are known, their corresponding object detections can be reconciled to better confirm the position of objects in the 3D representation. Similarly, relative data capture can be used to verify each other's determinations and to resolve occlusion based on temporal continuity (e.g., when a team member occludes the view of one sensor but not the view of another).

[0082] After considering all time groups at box 1110d, additional verification can be performed at box 1110e, for example, using time information from the intervals of box 1110d to reconcile occlusion and gaps in object detection at box 1115d. Once all non-operational periods of interest have been considered at box 1110b, the system can perform overall post-processing and verification fill at box 1120a. For example, knowledge about the presence of objects between time periods, or based on location or operation type, can inform the expected number and relative positions of objects to be identified. To this end, although some embodiments may only be interested in analyzing non-operational periods, the start and end of operational periods may help inform or verify object detection during non-operational periods and can be considered. For example, if four people are consistently identified throughout an operational period, the system should expect to identify four people at the end of the previous non-operational period and the beginning of the next non-operational period.

[0083] Exemplary non-operative full-suite data processing - object tracking Similar to segmenting raw data into non-operational periods (e.g., performed by non-operational period detection component 910c) and detecting objects (such as people) within these periods (e.g., via component 910e), it should be understood that there are multiple ways to perform tracking and motion detection. For example, as... FIG. 11B The object detection described, combined with optical flow analysis (with a complementary flow perspective to resolve ambiguities), can be readily used to identify the movement of each specific object throughout the site. As another example, some embodiments may employ multi-object machine learning tracking algorithms, which involve detecting and tracking multiple objects within a full-site data sequence. These methods can identify and locate objects of interest in each frame and then correlate these objects across frames to maintain tracking of their movement over time. For example, some embodiments may use algorithms similar to those described by Meinhardt, Tim et al. in "TrackFormer: Multi-Object Tracking with Transformers". arXiv TM Preprint arXiv TM : 2101.02702 The implementation method described in (2021).

[0084] As an example of the method of Meinhardt et al. FIG. 12A This is a schematic block diagram illustrating an exemplary tracking information processing flow, which may be used in conjunction with some embodiments. In the first visual image or depth frame 1205a, the system may apply a set of tracking frameworks 1210a, including a convolutional neural network, a transformer encoder and decoder, and initial object detection (e.g., in...). FIG. 11AWith the aid of object detection methods. Iteratively applying the tracking frame 1210b and 1210c to subsequent images or frames 1205b and 1205c can produce object detection (such as people), with a record of orientation across frames 1205a, 1205b, and 1205c (ellipsis 1205d reflects the presence of intermediate frames and tracking identification).

[0085] FIG. 12B This is a flowchart illustrating various operations in an exemplary process 1215 for performing object tracking, which may be used in conjunction with some embodiments. At block 1215a, the system may receive full-field data, for example, after identification during non-operational periods. At blocks 1215b and 1215c, the system may iterate through non-operational periods, and for each period, considerable detection and tracking methods are iterated at blocks 1220a and 1220b. For each method, the sensor data stream may be considered sequentially at blocks 1220c and 1220d, and the applicable detection and tracking method may be performed at block 1220e (it should be understood that, alternatively, in some embodiments, the stream may be integrated before applying the object detection and tracking system, for example, when integrating depth frames acquired simultaneously from multiple sensors into a single virtual model). As previously mentioned, some methods may benefit from considering the temporal and spatial continuity across full-field sensors, and therefore a reconciliation method for a specific tracking application may be applied at block 1220f.

[0086] Similarly, reconciliation between findings from tracking methods across time periods can be performed at box 1225a. For example, the determined locations of objects found by various methods can be averaged. Similarly, the number of objects can be determined by majority voting among methods, possibly weighted by uncertainty or confidence values ​​associated with the methods. Similarly, after considering all non-operational time periods, the system can perform overall reconciliation at box 1225b, for example, ensuring consistency between initial and final object counts and locations with adjacent time periods or action groups.

[0087] For comparison FIG. 12B and FIG. 9C It will be noted that object detection, tracking, or motion detection can be performed at the time-period level (and then, if necessary, associated with a task / action / range for metric calculation), or it can be performed after an action, task, or range has been identified, and specifically for the corresponding data.

[0088] Exemplary non-operative full-suite data processing - motion assessment While some tracking systems can readily facilitate motion analysis at the motion detection unit 910i, some embodiments can alternatively or in parallel perform motion detection and analysis using visual images and depth frame data. In some embodiments, simply measuring the amount of motion (amplitude, independent of the direction component) of any object (or only the object of interest) in three-dimensional space within the site can provide a meaningful metric for determining non-operational periods. However, more refined motion analysis can facilitate more sophisticated queries, such as team member path analysis, collision detection, etc.

[0089] As an example, optical flow-based motion evaluation FIG. 13A These are schematic visual image 1305a and depth frame 1305b, full-field data pairs, with indications of optical flow-derived correspondence, as may be used in some embodiments. Specifically, the data processing system can review the visual image data sequence to detect optical flow. Here, the system has detected that team member 1310b is moving from the right side of the image to the left side, as shown by arrow 1310a and the pixel boundaries around the pixels with optical flow around team member 1310b.

[0090] While some embodiments may consider motion solely based on optical flow from visual images, it is sometimes desirable to "normalize" the motion. Specifically, turning... FIG. 13C As shown in image 1340a, movement 1345a away from the camera may result in fewer pixels (the pixels depicting member 1350a) being associated with optical flow. Conversely, as shown in image 1340b, when team members 1350b are very close to the sensor, their movement 1345b may result in optical flow affecting more pixels.

[0091] Some embodiments may normalize the motion associated with optical flow to three-dimensional space, rather than allowing the number of visual image pixels involved in the flow to influence motion determination. That is, referencing FIG. 13D As team members move from positions 1355a and 1360a to positions 1355b and 1360b respectively, the magnitudes of motions 1345a and 1345b in three-dimensional space can be the same. Although the distance 1370b between positions 1360a-1360b and sensor 1365 is less than the distance 1370a from sensor 1365 to positions 1355a-1355b, some embodiments may seek to identify the same amount of motion 1345a and 1345b in each instance. Specifically, downstream metrics can treat the velocities of motions 1345a and 1345b equally, regardless of their distance from the capture sensor.

[0092] To achieve this, return FIG. 13AFor each portion of the visual image 1305a associated with optical flow, the system can consider the corresponding portion of the simultaneously acquired depth image 1305b, where team member 1310b and its motion (indicated by arrow 1315a) will also be apparent. That is, in this example, pixel 1310c associated with optical flow can correspond to depth values ​​1320 to 1315c. By considering these depth values ​​1320, the system can infer the distance to the object causing the optical flow (e.g., distance between 1370b and 1370a). In other words, referencing... FIG. 13B Once the distances 1330a and 1330b from the acquisition sensor 1335 are inferred from the depth data, the system can infer the "normalized" motion 1325c of the object moving from orientation 1325a to orientation 1325b in three-dimensional space. In some embodiments, instead of detecting optical flow in the two-dimensional visual image first, the optical flow information in the three-dimensional depth data is used, and the normalized motion is determined after necessary adjustments.

[0093] FIG. 14 This is a flowchart illustrating various operations in an exemplary process 1400 for performing motion analysis from full-field data, as may be applied in some embodiments. At boxes 1405b and 1405c, the system may iterate through the full-field data received at box 1405a. For example, the full-field data may be downsampled and considered as a discrete dataset of temporally continuous pairs of visual images and depth frames. If one or more optical flow artifacts (continuous regions in a visual image or depth frame where optical flow exceeds a threshold) are detected in the dataset at box 1405d, the system may iterate through these artifacts at boxes 1410a and 1410b. Many artifacts may not correspond to the objects of interest used to prepare metrics. For example, some accidental motion of equipment, some light adjustments, some door openings, etc., may be irrelevant to downstream analysis. Therefore, at box 1410c, the system may verify that the artifact corresponds to one or more of the objects of interest (e.g., via a machine learning system of component 910e using methods disclosed herein). FIG. 11A-11B and 12A- FIG. 12B The system and methods used can associate optical flow artifacts with people or equipment detected by the system. For example, pixels corresponding to optical flow can be compared with pixels identified in, for example, YOLO network object detection. In some cases, a single optical flow artifact can be associated with more than one object, such as when one moving object occludes another. Evaluation of the corresponding depth values ​​can reveal the identity of the corresponding object appearing in the artifact, or at least their corresponding location and trajectory.

[0094] Therefore, when the artifact corresponds to an object of interest (e.g., team members), the system can determine the corresponding depth value at box 1415a and normalize the detected motion to three-dimensional space (e.g., the same motion value independent of the distance to the sensor) at box 1415b, rather than in the two-dimensional plane of the optical flow of the visual image, for example using the methods described in this paper. FIG. 13A-13D The techniques discussed. The resulting motion can then be recorded at box 1415c for subsequent metric calculations discussed in more detail in this paper.

[0095] Exemplary non-operative full-suite metric analysis - anomaly detection After the metrics are generated (e.g., at the metric generation system 910j), some embodiments may seek to identify anomalous behavior (e.g., at the metric analysis system 910k) based on the aforementioned metrics (including...). FIG. 5A-5C This involves detecting anomalies in each team / operating room / hospital, etc., by analyzing the duration of actions and intervals, the number of people involved in each location, and the observed amount of movement. FIG. 15 This is a flowchart illustrating various operations in an exemplary process 1500 for anomaly analysis based on determined indicator values, as may be implemented in some embodiments.

[0096] At box 1505a, the system can acquire historical datasets, for example, for use with metrics that have component values ​​based on historical data (such as normalization). At box 1505b, the system can determine the metric results for the entire non-operational period (e.g., cumulative motion within that period, regardless of whether it occurs in association with any specific task or interval). At box 1505c, the system can determine the metric results for specific tasks and intervals within each non-operational segment (e.g., FIG. 5A-5C The duration of the action and interval in the interval). At box 1505d, the system can then determine the composite index value from the previously determined indexes (e.g., FIG. 8 The ORA score discussed in the article is 830.

[0097] At box 1505e, clusters of indicator values ​​corresponding to patterns of inefficient or efficient non-operational site states, and clusters of indicator values ​​corresponding to patterns of efficient or positive non-operational site states, can be included in the historical data of box 1505a. Such clusters can be used to find indicator scores and patterns of indicator scores, as well as to find distances to ideal clusters and undesirable clusters (e.g., where the distances are Euclidean distances and each indicator in a group is treated as a separate dimension).

[0098] Therefore, the system can traverse the indicators individually or in groups at boxes 1510a and 1510b to determine at box 1510c whether an indicator or group exceeds the tolerance relative to historical data clusters (of course, the nature of the tolerance can vary with each expected grouping and can be based on a historical benchmark, such as one or more standard deviations from the median or mean). In cases where this tolerance is exceeded (e.g., an indicator value or group of indicator values ​​is too close to an inefficient cluster or too far from an efficient cluster), the system can record the deviation at box 1510d for future guidance and feedback as described herein.

[0099] For clarity, as mentioned earlier, clusters can appear in an N-dimensional space, where N corresponding metrics are considered within the group (although other spaces and surfaces can also be used to compare metric values). This algorithm can be applied to detect anomalies in each team / operating room / hospital based on the aforementioned metrics. Clustering algorithms (e.g., based on K-means, using machine learning classifiers, etc.) can both reveal groups and identify anomalies; the former is used to identify common inefficient / efficient patterns in values, and the latter is used to identify, for example, deviations from ideal performance or acceptable avoidance of undesirable states.

[0100] Therefore, the system can determine at box 1515a whether an indicator is individually or in groups associated with inefficient or efficient clusters (e.g., within their threshold distance, such as the cluster's standard deviation, maximum principal component, etc.), and if so, record the cluster at box 1515b for future guidance and feedback. For example, raw indicator values, composite indicator values, anomalies, distance to the cluster or distance from the cluster, and relevant corrective actions can be presented in the GUI interface, for example, as described below regarding... FIG. 17 or FIG. 18A-18C Described.

[0101] Exemplary non-operative data analysis - coaching Following anomaly detection and clustering, in some embodiments, the system may also seek to integrate the results into a form suitable for feedback and guidance (e.g., via application 550f). For example, corrective actions may be known for tolerance breaches (e.g., at box 1510c) or proximity to undesirable indicator clusters (e.g., at box 1515a). Here, guidance may simply include known corrective actions when reporting breaches or cluster associations, for example.

[0102] Some embodiments can identify higher-level correlations in metric values, thereby enabling the proposal of corrective actions. For example, after considering a new dataset from a previously unconsidered hospital setting, various embodiments can identify a particular surgical specialty (e.g., urology) within that setting that has a large standard deviation in its non-operational time metrics. The various algorithms disclosed herein can leverage this large standard deviation, along with other data points and historical data, to suggest corrective actions for scheduling or staffing models. For instance, a regression model can be used, employing historical data to infer potential solutions based on data distribution.

[0103] As another example, FIG. 16 This is a flowchart illustrating various operations in an exemplary process 1600 for providing guidance feedback based on determined indicator values, as may be implemented in some embodiments. While this example focuses on the relationship between indicator values ​​and undesirable / inefficient patterns, it should be recognized that variations exist that instead determine the relationship with desirable / efficient patterns (with corresponding corrective measures when indicators deviate too far from these preferred states). Similarly, in some embodiments, indicators and groups of indicators can be directly compared to known patterns without first identifying tolerance deviations and cluster distances, as in exemplary process 1600.

[0104] Here, at boxes 1615a and 1615b, the system can iterate through all previously identified tolerance deviations (e.g., as determined at box 1510c) of groupings of one or more indicator results, and at box 1615c consider whether they correspond to known inefficiency patterns (e.g., by taking the inner product of the indicator value with a known inefficiency vector). For example, a delayed “case opening to patient admission” duration, combined with a specific delay 810c and case quantity 810a value, can, for example, indicate scheduling inefficiency, where adjusting the scheduling typically resolves undesirable states. Note that one or more indicators used to map to the inefficiency pattern used for correction can be the same as or different from one or more indicators that deviate from the tolerance (e.g., at box 1615a) or approach an undesirable cluster (e.g., at box 1620a), for example, the latter can in turn indicate that the former can correspond to an inefficiency pattern. For example, from FIG. 5A An anomaly in a duration metric can suggest a source FIG. 8 An inefficient pattern of combining multiple indicators.

[0105] Therefore, the system can traverse possible inefficiency patterns at boxes 1615c and 1615d to consider the similarity between the corresponding indicator values ​​and the inefficiency patterns. For example, the Euclidean distance from the indicator to the pattern can be obtained at box 1615e. At box 1615f, the system can record the similarity (e.g., distance) between the inefficiency pattern and the group of indicators associated with tolerance deviation.

[0106] Similarly, after considering tolerance deviations, the system can consider the combination of index scores for clusters near the undesirable / inefficient events at boxes 1620a and 1620b (e.g., as determined at box 1515a). As previously described, the system can traverse possible known inefficiency patterns at boxes 1620c and 1620d, again determine the correspondence between the inefficiency pattern and the corresponding index value (which may or may not be the same set of index values ​​identified in the cluster association at box 1620a) at box 1620e (again, for example, Euclidean or other suitable similarity indexes), and record the degree of correspondence at box 1620f.

[0107] Based on the distances and correspondences determined at boxes 1615e and 1620e, respectively, the system can determine the priority ranking of detected inefficiency patterns at box 1625a. At box 1625b, the system can return the inefficiency pattern associations with the most significant threshold number. For example, each inefficiency pattern can be associated with a priority (e.g., high-priority patterns might be those with the potential to cause downstream transmission, patient harm, equipment damage, etc., leading to inefficiency, while low-priority patterns might only cause time delays) and presented accordingly to the reviewers. Thus, each association can be scored as a weighted sum of the similarity between the scoring indicator value and the indicator value associated with the inefficiency pattern (which is then weighted by the severity / priority of the inefficiency pattern). In this way, the most serious potential faults can be identified and returned to the reviewers first. The iterative nature of topology 450 can facilitate the reconsideration and reweighting of the prioritization process 1600 as the reviewers observe the impact of the proposed feedback over time. Similarly, iterations can provide opportunities to identify additional corrective actions and correspondences with inefficiency patterns.

[0108] Exemplary GUI non-operative metric analysis feedback element The presentation of the analysis results (e.g., at box 910l) can take various forms in different embodiments. For example, FIG. 17 This is a schematic representation of GUI elements in a quick review dashboard interface for reviewing non-operational indicators, as may be implemented in some embodiments. In this example GUI 1705, selectors 1710a-1710d are provided for users to select the time range of non-operational period performance data they wish to analyze. In this example, the user has selected to review data captured over the past year. Following this time selection, the "Non-operational Indicators" area, the "Case Mix" area, and the "Metadata" area can be populated with values ​​corresponding to the non-operational periods of the selected data range.

[0109] The “Case Mixing” area provides a general description of the data filtered from the time selection. For example, as shown in label 1715a, the total number of cases under consideration (non-operational periods) is 205. These 205 cases are then broken down by surgical type via labels 1715b-1715d (specifically, of the 205 non-operational periods, 15 are associated with preparation for open surgery, 180 with preparation for robotic surgery, and 10 with preparation for laparoscopic surgery). The non-operational periods under consideration can be periods occurring before and after the 205 surgeries, periods only before, or periods only after, depending on the user's selection.

[0110] The “Metadata” area can also be filled with various parameters describing the selected data, such as the number of operating rooms involved (8 according to label 1720a), the number of specialties (4 according to label 1720b), the number of procedure types (10 according to label 1720c), and the number of different surgeons involved in the surgery (27 according to label 1720d).

[0111] Within the "non-operational indicators" area, the methods described in this article can be used (e.g., regarding...). FIG. 8 The description shows the overall comprehensive score (such as the ORA score) in region 1725a. Regions 1725b-1725f can show... FIG. 5A The corresponding statistical data for the interval, here is FIG. 5A Values ​​for various intervals.

[0112] Some implementations can also comprehensively present the results of the scoring metrics, for example, to allow reviewers to quickly review the feedback and identify effective and ineffective aspects of performance in non-operational settings. For example, FIG. 18A This is a schematic representation of GUI element 1805 that can be used for global rapid review feedback in some embodiments. Specifically, individual indicator scores, composite indicator scores, deviations from tolerances, and proximity to desirable or undesirable clusters can be indicated in the numerical area 1805d. The names of indicators, etc., can be indicated in the name area 1805a, and the expected feedback can be indicated in area 1805b. A rapid review icon 1805e may also be included to facilitate easy identification of the nature of the numerical feedback. The quality relationship arrow area 1805c can be used to indicate whether the value in area 1805d is above or below the operating point or tolerance, or whether it shows an upward or downward trend over time, and whether this is interpreted as indicating an improvement or decrease in efficiency.

[0113] Specifically, FIG. 18B This is a schematic representation of an arrow element, which can be used in some embodiments. FIG. 18AThe quality relationship arrows in area 1805c are used. The arrows may be color-coded to indicate whether a value is efficient (e.g., green) or inefficient (e.g., red). Thus, an upward arrow 1810a could indicate a value in area 1805d that is above the lower limit (e.g., when idle time after a task is successfully completed increases above the historical average). Similarly, a downward arrow 1810b could indicate a value in area 1805d that is below the upper limit (e.g., when preparation time decreases below the historical average). Conversely, a downward arrow 1810c could indicate a value in area 1805d that is below the expected minimum (e.g., when the number of personnel ready to perform the required steps is below the historical average). Similarly, an upward arrow 1810d could indicate a value in area 1805d that is above the expected upper limit (e.g., when preparation time increases beyond the historical average).

[0114] By associating relational values ​​with both arrow direction and highlighting (e.g., by color, bolding, animation, etc.), reviewers can easily browse a large number of values ​​and identify results that indicate efficient or inefficient feedback. Highlighting can also take various degrees (e.g., transparency values, bolding levels, animation frequency, etc.) to indicate the priority associated with efficient or inefficient values. For example, FIG. 18C This is a schematic representation of GUI elements in the quick review feedback interface 1820, as may be used in some embodiments. Here, individual quick review feedback (an instance of element 1805) can be arranged in a grid and sized to allow reviewers to perceive multiple items at once. Each element can be selectable, presenting value-specific details, including, for example, corresponding historical data, full-site data, intermediate indicator calculation results, etc. It should be understood that this diagram is only illustrative, and each "action" or "feedback" text can be replaced with one of the indicators described herein (e.g., from...). FIG. 5A-5C The duration of the interval) and corrective measures (although in some configurations, feedback may be omitted from all or some of the elements).

[0115] FIG. 19A This is a chart of exemplary analysis values ​​collected in conjunction with a prototype implementation of the embodiments. Specifically, FIG. 19A It shows the FIG. 5A The results were processed from various intervals. Here, outlier 1905a clearly indicates a deviation from the median duration of approximately 10 minutes (instead, it is approximately 260 minutes) in the "skin closure to patient departure" interval. FIG. 19B Similarly, a graph of exemplary operating room analysis values ​​collected in conjunction with a prototype implementation of the embodiments is shown. Here, the standard deviation interval may be displayed to guide reviewers in identifying outliers (e.g., whether they reflect durations longer or shorter than the standard deviation interval).

[0116] FIG. 20A This is a graph of exemplary values ​​collected in conjunction with a prototype implementation of the embodiments. Even without performing the anomaly detection and inefficient pattern recognition methods disclosed herein, various values ​​can be easily identified as anomalies through inspection. For example, within the "case opening to patient entry" interval, cases 2005a-2005c clearly indicate anomalies above the standard deviation. For the "patient entry to skin incision" interval, case 2010a is shorter than the standard deviation interval. For the "skin closure to patient exit" interval, cases 2015a-2015b exceed the standard deviation of the selected historical cases. For the "patient exit to case opening" interval, case 2020a significantly exceeds the standard deviation of the selected historical cases. For the "extraction to insertion" interval, cases 2025a and 2025b exceed the standard deviation of the selected historical cases. FIG. 20B Similarly, there are charts of exemplary operating room analysis values ​​collected in conjunction with prototype implementations of the embodiments.

[0117] Similarly, FIG. 21A This is a chart of exemplary operating room analysis values ​​collected in conjunction with an exemplary prototype implementation of the embodiments. FIG. 21B This is a chart of exemplary operating room analysis values ​​collected in conjunction with an exemplary prototype implementation of the embodiments, and it is in a horizontal format. FIG. 21C This is also a chart of exemplary operating room analysis values ​​collected in conjunction with an exemplary prototype implementation of the embodiments.

[0118] FIG. 22 This is a schematic representation of exemplary elements in a graphical user interface used to provide metric-based feedback, as may be used in some embodiments. In this example, the interface typically includes two elements: a full-field sensor playback element 2205; and an integrated timeline element 2210, which depicts the duration of various intervals within multiple non-operational periods. For example, each of the time interval decompositions 2210a-2210g could indicate the duration of interval 550a-550d for comparison (although seven periods are shown in this example, variations with more rows and more playback elements 2205 can be readily envisioned).

[0119] The full-site sensor playback element 2205 may contain metadata segment 2205a, which indicates the case identifier (“Case 1”), the site status (although the surgical procedure “gastric bypass” is shown here to anticipate the upcoming surgery, it may also be shown here additionally or alternatively). FIG. 5A-5C The non-operational actions and intervals), the date and time of data collection ("May 27, 20XX, 07:49:42"), and the number of personnel identified (here, "2", as shown, for example, according to component 910h and for example) FIG. 11A-11B(The method is determined). The full-area sensor playback element 2205 can also display data from one or more full-area sensors in playback segment 2205b, with bounding boxes (e.g., boxes 2205c and 2205d), overlays, contours, or other suitable indications of detected personnel. In the integrated timeline element 2210, multiple time intervals 2210a-2210g can be presented, indicating, for example, the duration of multiple intervals (e.g., the duration of intervals 550a-550d across seven non-operational periods). The playback in region 2205 can correspond to the selection of one of the intervals in the time interval decomposition 2210a-2210g (e.g., depicting the corresponding full-area data playback for that interval). In this way, reviewers can easily perceive the overall results while analyzing the state of specific instances (e.g., instances that might attract user attention based on, for example, corresponding determined indicator values ​​or pattern similarities).

[0120] Computer system FIG. 23 This is a block diagram of an exemplary computer system, which may be used in conjunction with some embodiments. The computing system 2300 may include interconnects 2305 connecting various components, such as one or more processors 2310, one or more memory units 2315, one or more input / output systems 2320, one or more storage systems 2325, one or more network adapters 2330, etc. The interconnects 2305 may be, for example, one or more bridges, traces, buses (e.g., ISA, SCSI, PCI, I2C, FireWire, etc.), wires, adapters, or controllers.

[0121] One or more processors 2310 may include, for example, Intel TM Processor chips, math coprocessors, graphics processors, etc. One or more memory components 2315 may include, for example, volatile memory (RAM, SRAM, DRAM, etc.), non-volatile memory (EPROM, ROM, flash memory, etc.), or similar devices. One or more input / output devices 2320 may include, for example, display devices, keyboards, pointing devices, touchscreen devices, etc. One or more storage devices 2325 may include, for example, cloud-based storage devices, removable universal serial bus (USB) storage devices, disk drives, etc. In some systems, memory components 2315 and storage devices 2325 may be the same component. Network adapter 2330 may include, for example, a wired network interface, a wireless interface, Bluetooth, etc. TM Adapters, line-of-sight interfaces, etc.

[0122] It should be understood that, with FIG. 23Compared to the components depicted herein, some embodiments may contain only some components, alternative components, or more components. Similarly, in some systems, components may be combined or have dual functions. Components may be implemented using dedicated hardwired circuit systems, such as, for example, one or more ASICs, PLDs, FPGAs, etc. Therefore, some embodiments may be implemented, for example, in a programmable circuit system (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in a dedicated hardwired (non-programmable) circuit system, or in a combination of such forms.

[0123] In some embodiments, data structures and message structures may be stored or transmitted via network adapter 2330 via a data transmission medium (e.g., signals on a communication link). Transmission can occur across various media, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections. Therefore, "computer-readable medium" can include computer-readable storage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0124] One or more memory components 2315 and one or more storage devices 2325 may be computer-readable storage media. In some embodiments, one or more memory components 2315 or one or more storage devices 2325 may store instructions that can perform or cause to perform the various operations discussed herein. In some embodiments, the instructions stored in memory 2315 may be implemented as software and / or firmware. These instructions may be used to perform operations on one or more processors 2310 to perform the processes described herein. In some embodiments, such instructions may be provided to one or more processors 2310 by downloading instructions from another system, such as via network adapter 2330.

[0125] For clarity, it should be understood that although a computer system can be a single machine located in a single location, it has... FIG. 23 One or more of the components, but this is not mandatory. For example, a distributed networked computer system may include multiple individual processing workstations, each with... FIG. 23Some or all of the components described herein. Therefore, the processes and various operations described herein can be distributed across one or more workstations of such a computer system. For example, it should be understood that a process suitable for running as a single thread on a single workstation can alternatively be divided into any number of sub-threads across one or more workstations, which then run serially or in parallel to achieve the same or substantially similar results as the process running in a single thread. Similarly, it should be understood that while non-transitory computer-readable media may exist independently (e.g., in a single USB storage device) or reside within a single workstation (e.g., in the workstation's random access memory or disk storage device), such media need not be located in a single geographic location, but may include, for example, geographically separated workstations (which network with each other) or geographically separated storage devices residing across a computer system.

[0126] Remarks The accompanying drawings and descriptions are illustrative. Therefore, neither the descriptions nor the drawings should be construed as limiting this disclosure. For example, titles or subtitles are provided merely for the reader's convenience and to facilitate understanding. Therefore, titles or subtitles should not be construed as limiting the scope of this disclosure, for example, by grouping features in a particular order or together merely for ease of understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In case of conflict, this document (including any definitions provided herein) shall prevail. The listing of one or more synonyms herein does not preclude the use of other synonyms. Examples used anywhere in this specification (including examples of any terms discussed herein) are merely illustrative and are not intended to further limit the scope and meaning of this disclosure or any exemplary terminology.

[0127] Similarly, although presented in a particular manner in the accompanying figures, those skilled in the art will understand that the actual data structure used to store information may differ from that shown. For example, the data structure may be organized in different ways, may contain more or less information than shown, may be compressed and / or encrypted, etc. Common or well-known details may be omitted in the figures and disclosure to avoid confusion. Similarly, the figures may depict a particular set of operations for ease of understanding, which are merely exemplary operations within a broader category of such sets of operations. Thus, it will be readily recognized that the same purpose or effect depicted in some flowcharts can often be achieved using additional, alternative, or fewer operations. For example, data may be encrypted, although not presented in this way in the figures; items may be considered in different loop patterns (“for” loops, “while” loops, etc.), or ordered in different ways to achieve the same or similar effect, etc.

[0128] The references to “embodiment” or “an embodiment” herein mean that at least one embodiment of this disclosure includes the specific features, structures, or characteristics described in association with that embodiment. Therefore, the phrase “in one embodiment” appearing in various places throughout this document does not necessarily refer to the same embodiment. Individual or alternative embodiments may not be mutually exclusive with other embodiments. It will be appreciated that various modifications can be made without departing from the scope of the embodiments.

Claims

1. A computer-implemented method for evaluating team member performance during non-operational periods within a surgical setting, the method comprising: The team members inspected the entire site data collected during the aforementioned non-operational periods; Determine the movement of the team members; as well as The first indicator result is determined at least in part based on the movement of the team members.

2. The computer-implemented method according to claim 1, wherein the full-scene data includes full-scene visual image data and full-scene depth data.

3. The computer-implemented method according to claim 2, wherein, The team members are associated with the roles they play within the premises, and wherein, Detecting the team members includes determining that the team members are associated with the roles.

4. The computer-implemented method according to claim 3, further comprising: Detect intervals within the entire site data; Determine the duration of the interval; as well as The second indicator result is determined at least in part based on the duration of the interval.

5. The computer-implemented method according to claim 4, wherein the interval includes one of the following: The skin closes until the patient leaves the area; The patient left the case-opening area; From the start of the case to the patient's entry into the interval; The patient enters the skin incision area; as well as Push out to push in the interval.

6. The computer-implemented method according to claim 4, wherein the method further comprises: The third indicator result is determined, and the third indicator result includes at least one of the following: Metrics results related to time-based workflows; Metrics results related to scheduling; as well as Results of indicators related to human resources.

7. The computer-implemented method of claim 6, wherein determining the third indicator result includes determining one or more of the following: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; and Modality transition index results.

8. The computer-implemented method of claim 6, wherein the third indicator result includes a comprehensive score result, the comprehensive score result being determined based on one or more of the following component results: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; as well as Modality transition index results.

9. The computer-implemented method of claim 8, wherein the overall scoring result is a weighted average of one or more of the component results.

10. The computer-implemented method according to claim 6, wherein the method further comprises: Determine the duration of the actions in the entire site data; as well as The fourth indicator result is determined based on the duration of the action.

11. The computer-implemented method of claim 10, wherein the method further comprises: The number of people in the venue during at least a portion of the non-operational period is determined from the full venue data; as well as The results of the fifth indicator are determined based on the identified number of personnel.

12. The computer-implemented method of claim 11, wherein the method further comprises: Inefficiency can be detected at least in part by comparing the following: Clusters of historical indicator results; as well as A set of indicators including two or more of the following: The result of the first indicator; The result of the second indicator; The results of the third indicator; The results of the fourth indicator; and The results of the fifth indicator.

13. The computer-implemented method of claim 12, wherein detecting inefficiency comprises: Determine the distance in a space having dimensions corresponding to the indicator results in the set of indicator results, wherein the distance is between: At least a portion of the clusters of historical indicator results; and Points associated with the results of the aforementioned set of indicators.

14. The computer-implemented method of claim 12, wherein detecting inefficiency comprises: The clusters of points associated with the set of index results are found to be outside the standard deviation of the historical index results.

15. The computer-implemented method according to any one of claims 1-14, wherein determining the movement of the team member comprises: Determine the optical flow of at least a portion of the total field data; Determine that the optical flow of at least a portion of the total site data corresponds to the team member; as well as Determine the normalized motion associated with the optical flow.

16. The computer-implemented method according to any one of claims 1-14, wherein determining the movement of the team member comprises: The transformer-based attention model is applied to at least a portion of the entire field data.

17. The computer-implemented method according to any one of claims 1-14, wherein the method further comprises: A transformer configured to implement a gated loop unit method to detect non-operational periods in the full-area data is used to distinguish between the non-operational periods in the full-area data and the operation periods in the full-area data.

18. The computer-implemented method according to any one of claims 1-14, wherein detecting the team members comprises: The transformer-based spatial model is applied to at least a portion of the total field data.

19. The computer-implemented method according to any one of claims 4-14, wherein detecting the interval in the total site data comprises: The converter, configured to implement the gated loop unit method, will be applied to at least a portion of the total field data.

20. The computer-implemented method according to any one of claims 11-14, wherein the method further comprises: Determining the number of people in the venue during at least a portion of the non-operational period includes applying a transformer-based spatial model configured to process full-venue data.

21. A non-transitory computer-readable medium comprising instructions configured to cause at least one computer system to perform a method for evaluating team member performance during non-operational periods within a surgical setting, the method comprising: The team members inspected the entire site data collected during the aforementioned non-operational periods; Determine the movement of the team members; as well as The first indicator result is determined at least in part based on the movement of the team members.

22. The non-transitory computer-readable medium of claim 21, wherein the full-field data includes full-field visual image data and full-field depth data.

23. The non-transitory computer-readable medium according to claim 22, wherein, The team members are associated with the roles they play within the premises, and wherein, Detecting the team members includes determining that the team members are associated with the roles.

24. The non-transitory computer-readable medium of claim 23, further comprising: Detect intervals within the entire site data; Determine the duration of the interval; as well as The second indicator result is determined at least in part based on the duration of the interval.

25. The non-transitory computer-readable medium of claim 24, wherein the interval comprises one of the following: The skin closes until the patient leaves the area; The patient left the case-opening area; From the start of the case to the patient's entry into the interval; The patient enters the skin incision area; as well as Push out to push in the interval.

26. The non-transitory computer-readable medium of claim 24, wherein the method further comprises: The third indicator result is determined, and the third indicator result includes at least one of the following: Metrics results related to time-based workflows; Metrics results related to scheduling; as well as Results of indicators related to human resources.

27. The non-transitory computer-readable medium of claim 26, wherein determining the third indicator result includes determining one or more of the following: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; and Modality transition index results.

28. The non-transitory computer-readable medium of claim 26, wherein the third indicator result includes a comprehensive score result, the comprehensive score result being determined based on one or more of the following component results: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; as well as Modality transition index results.

29. The non-transitory computer-readable medium of claim 28, wherein the overall scoring result is a weighted average of one or more of the component results.

30. The non-transitory computer-readable medium of claim 26, wherein the method further comprises: Determine the duration of the actions in the entire site data; as well as The fourth indicator result is determined based on the duration of the action.

31. The non-transitory computer-readable medium of claim 30, wherein the method further comprises: The number of people in the venue during at least a portion of the non-operational period is determined from the full venue data; as well as The results of the fifth indicator are determined based on the identified number of personnel.

32. The non-transitory computer-readable medium of claim 31, wherein the method further comprises: Inefficiency can be detected at least in part by comparing the following: Clusters of historical indicator results; as well as A set of indicators including two or more of the following: The result of the first indicator; The result of the second indicator; The results of the third indicator; The results of the fourth indicator; and The results of the fifth indicator.

33. The non-transitory computer-readable medium of claim 32, wherein detecting inefficiency comprises: Determine the distance in a space having dimensions corresponding to the indicator results in the set of indicator results, wherein the distance is between: At least a portion of the clusters of historical indicator results; and Points associated with the results of the aforementioned set of indicators.

34. The non-transitory computer-readable medium of claim 32, wherein detecting inefficiency comprises: The clusters of points associated with the set of index results are found to be outside the standard deviation of the historical index results.

35. The non-transitory computer-readable medium according to any one of claims 21-34, wherein determining the movement of the team member comprises: Determine the optical flow of at least a portion of the total field data; Determine that the optical flow of at least a portion of the total site data corresponds to the team member; as well as Determine the normalized motion associated with the optical flow.

36. The non-transitory computer-readable medium according to any one of claims 21-34, wherein determining the movement of the team member comprises: The transformer-based attention model is applied to at least a portion of the entire field data.

37. The non-transitory computer-readable medium according to any one of claims 21-34, wherein the method further comprises: A transformer configured to implement a gated loop unit method to detect non-operational periods in the full-area data is used to distinguish between the non-operational periods in the full-area data and the operation periods in the full-area data.

38. The non-transitory computer-readable medium according to any one of claims 21-34, wherein detecting the team members comprises: The transformer-based spatial model is applied to at least a portion of the total field data.

39. The non-transitory computer-readable medium according to any one of claims 24-34, wherein detecting the interval in the full-site data comprises: The converter, configured to implement the gated loop unit method, will be applied to at least a portion of the total field data.

40. The non-transitory computer-readable medium according to any one of claims 31-34, wherein the method further comprises: Determining the number of people in the venue during at least a portion of the non-operational period includes applying a transformer-based spatial model configured to process full-venue data.

41. A computer system, the computer system comprising: At least one processor; as well as At least one memory, the at least one memory including instructions configured to cause the computer system to execute a method for evaluating team member performance during non-operational periods within a surgical setting, the method comprising: The team members inspected the entire site data collected during the aforementioned non-operational periods; Determine the movement of the team members; and The first indicator result is determined at least in part based on the movement of the team members.

42. The computer system of claim 41, wherein the full-area data includes full-area visual image data and full-area depth data.

43. The computer system according to claim 42, wherein, The team members are associated with the roles they play within the premises, and wherein, Detecting the team members includes determining that the team members are associated with the roles.

44. The computer system according to claim 43, the method further comprising: Detect intervals within the entire site data; Determine the duration of the interval; as well as The second indicator result is determined at least in part based on the duration of the interval.

45. The computer system of claim 44, wherein the interval includes one of the following: The skin closes until the patient leaves the area; The patient left the case-opening area; From the start of the case to the patient's entry into the interval; The patient enters the skin incision area; as well as Push out to push in the interval.

46. ​​The computer system of claim 44, wherein the method further comprises: The third indicator result is determined, and the third indicator result includes at least one of the following: Metrics results related to time-based workflows; Metrics results related to scheduling; as well as Results of indicators related to human resources.

47. The computer system of claim 46, wherein determining the third indicator result includes determining one or more of the following: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; and Modality transition index results.

48. The computer system of claim 46, wherein the third indicator result includes a comprehensive score result, the comprehensive score result being determined based on one or more of the following component results: Efficiency index results; Consistency index results; Adverse event indicator results; Case volume indicators; Results of the first case handover indicators; Delayed indicator results; OR flow rate metric results; Room layout index results; as well as Modality transition index results.

49. The computer system of claim 48, wherein the overall scoring result is a weighted average of one or more of the component results.

50. The computer system of claim 46, wherein the method further comprises: Determine the duration of the actions in the entire site data; as well as The fourth indicator result is determined based on the duration of the action.

51. The computer system of claim 50, wherein the method further comprises: The number of people in the venue during at least a portion of the non-operational period is determined from the full venue data; as well as The results of the fifth indicator are determined based on the identified number of personnel.

52. The computer system of claim 51, wherein the method further comprises: Inefficiency can be detected at least in part by comparing the following: Clusters of historical indicator results; as well as A set of indicators including two or more of the following: The result of the first indicator; The result of the second indicator; The results of the third indicator; The results of the fourth indicator; and The results of the fifth indicator.

53. The computer system of claim 52, wherein detecting inefficiency comprises: Determine the distance in a space having dimensions corresponding to the indicator results in the set of indicator results, wherein the distance is between: At least a portion of the clusters of historical indicator results; and Points associated with the results of the aforementioned set of indicators.

54. The computer system of claim 52, wherein detecting inefficiency comprises: The clusters of points associated with the set of index results are found to be outside the standard deviation of the historical index results.

55. The computer system according to any one of claims 41-54, wherein determining the movement of the team member comprises: Determine the optical flow of at least a portion of the total field data; Determine that the optical flow of at least a portion of the total site data corresponds to the team member; as well as Determine the normalized motion associated with the optical flow.

56. The computer system according to any one of claims 41-54, wherein determining the movement of the team member comprises: The transformer-based attention model is applied to at least a portion of the entire field data.

57. The computer system according to any one of claims 41-54, wherein the method further comprises: A transformer configured to implement a gated loop unit method to detect non-operational periods in the full-area data is used to distinguish between the non-operational periods in the full-area data and the operation periods in the full-area data.

58. The computer system according to any one of claims 41-54, wherein detecting the team members comprises: The transformer-based spatial model is applied to at least a portion of the total field data.

59. The computer system according to any one of claims 44-54, wherein detecting the interval in the total data comprises: The converter, configured to implement the gated loop unit method, will be applied to at least a portion of the total field data.

60. The computer system according to any one of claims 51-54, wherein the method further comprises: Determining the number of people in the venue during at least a portion of the non-operational period includes applying a transformer-based spatial model configured to process full-venue data.

Images