Latent space training for surgical theater data

Abstract

Various of the disclosed embodiments provide systems and methods for training a machine learning system to anticipate semantic features of surgical theater data, even in the absence of fully-labeled data. Specifically, embodiments may compress, then reconstruct, the available surgical theater data, even if unlabeled, and in doing so, teach an encoding and decoding machine learning system to recognize semantic features salient to downstream training and applications. In addition, many of the disclosed embodiments accommodate a wide variety of surgical theater data modalities, e.g., pairs of visual intensity video surgical theater data and depth frame video surgical theater data. Masking, losses, and other disclosed features may further improve the system's ability to infer semantic correlations. Using various of the disclosed embodiments, machine learning applications may be made possible that would be otherwise unfeasible in low-data surgical theater data regimes.

Description

LATENT SPACE TRAINING FOR SURGICAL THEATER DATACROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of, and priority to, United States Provisional Application No. 63 / 535,060, filed upon August 28, 2023, entitled “LATENT SPACE TRAINING FOR SURGICAL THEATER DATA”, and which is incorporated by reference herein in its entirety for all purposes.TECHNICAL FIELD

[0002] Various of the disclosed embodiments relate to systems and methods for improving the training of machine learning systems upon surgical theater data.BACKGROUND

[0003] Recent improvements in machine learning, and particularly in deep learning, hold great promise for a wide variety of surgical theater applications. Coupled with data acquisition from in-theater sensors, such applications may recognize or predict adverse configurations in the theater, recognize or predict adverse patient states, provide guidance for improving team workflow, recognize theater states within a preferred taxonomy of state characterizations, provide comparative analytics with other hospitals and surgical teams, provide guidance for teams transitioning from non-robotics surgical theaters to robotic surgical theaters, and vice versa, etc. Not only will these applications improve patient outcomes, but they will also improve surgical team efficiency, helping to reduce costs and make healthcare more predictable, consistent, and cost-effective.

[0004] Unfortunately, the unique conditions of the surgical theater and the copiousness of the collected data frequently render manual labeling and annotation of the data impractical. Lack of such labeled data may in turn complicate training of the machine learning system. For example, if one wishes for a machine learning system to recognize one of a plurality of surgical tasks from surgical theater data, such as surgical video, it may be incumbent upon a trained expert annotator, familiar with the particulars of the surgeries and of the tasks, to manually inspect the surgical theater data and to manually annotate each temporal segment as corresponding to one of the plurality of tasks.Naturally, such human-in-the-loop annotation risks subjective labeling variation by the annotators, is limited by annotator fatigue, and is bounded by the number of expertly trained annotators available to review and label such data.

[0005] Indeed, the situation is even further complicated by the multimodal character of much surgical theater data. Asking an expert human annotator to segment only surgical video data into discrete taxonomic states is difficult enough, as video, unlike static images, is dense with time-varying information. To additionally ask such an expert annotator to annotate disparate auditory data, kinematics data, depth data, etc. acquired in the theater, in parallel with the video annotation, while also recognizing features unique to those different types of data modalities, can be simply impossible. This is particularly unfortunate as such information dense data types may be especially useful for machine learning.

[0006] Even when annotators manage to accomplish the herculean task of annotating sufficient quantities of surgical theater data for the machine learning system’s training, many of the above-described applications require, or benefit greatly from, the ability to perform subsequent online training of the machine learning system as additional surgical theater data becomes available. Such new data may be specific to the particular conditions of the healthcare environment in which the machine learning system is now deployed (e.g., the particular hospital or surgical theater in which the system has been deployed). Accordingly, training upon this new data may greatly facilitate localization of the deployed machine learning system to the particular character of its local environment. Unfortunately, few hospitals have the resources to, yet again, annotate this newly acquired data for additional training of the machine learning system.

[0007] Accordingly, there exists a need for systems and methods to overcome challenges and difficulties such as those described above. For example, there exists a need for systems to facilitate a wide variety of downstream surgical theater analysis applications, without always requiring the onerous involvement of multitudinous human annotators.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various of the embodiments introduced herein may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements:

[0009] FIG. 1A is a schematic view of various elements appearing in a surgical theater during a surgical operation, as may occur in relation to some embodiments;

[0010] FIG. 1 B is a schematic view of various elements appearing in a surgical theater during a surgical operation employing a robotic surgical system, as may occur in relation to some embodiments;

[0011] FIG. 2A is a schematic depth map rendering from an example theater-wide sensor perspective, as may be used in some embodiments;

[0012] FIG. 2B is a schematic top-down view of objects in the theater of FIG. 2A, with corresponding sensor locations, as may be used in some embodiments;

[0013] FIG. 2C is a pair of images depicting a grid-like pattern of orthogonal rows and columns in perspective, as captured from a theater-wide visual image sensor having a rectilinear view and a theater-wide visual image sensor having a fisheye view, each of which may be used in connection with some embodiments;

[0014] FIG. 3 is a schematic depiction of data that may be acquired during a surgically operative period, as may occur in connection with some embodiments;

[0015] FIG. 4 is a schematic depiction of data that may be acquired during a surgically nonoperative period, as may occur in connection with some embodiments;

[0016] FIG. 5A is a schematic representation of a plurality of theater state recognitions, as may be determined in some embodiments;

[0017] FIG. 5B is a schematic representation of a plurality of object segmentation results, as may be determined in some embodiments;

[0018] FIG. 6A is a schematic representation of an example collection of temporal intervals, as may be used to assess activities in a surgical theater in some embodiments;

[0019] FIG. 6B is a schematic block diagram indicating example activity analysis class groupings, as may be used in connection with some embodiments;

[0020] FIG. 7A is a schematic block diagram illustrating components in an example training or inference data item, as may be used in some embodiments;

[0021] FIG. 7B is a schematic block diagram illustrating components in labeled, partially labeled, and unlabeled datasets, as may be used in various embodiments;

[0022] FIG. 8A is a flow diagram illustrating various operations in an example process for performing supervised training of a machine learning system, as may be performed in some embodiments;

[0023] FIG. 8B is a flow diagram illustrating various operations in an example process for performing unsupervised training, and then task-specific supervised training, of a machine learning system, as may be performed in some embodiments;

[0024] FIG. 8C is a schematic block diagram illustrating elements in an example deployed classifier application machine learning system, a supervised training process, an unsupervised to supervised structural adaptation, and an unsupervised training system, as may be implemented in connection with various embodiments;

[0025] FIG. 9 is a schematic block diagram illustrating an example loss determination iteration during unsupervised training of a reconstructive system, as may be implemented in some embodiments;

[0026] FIG. 10A is a schematic tensor breakout illustrating component masking values in accordance with a first masking strategy, as may be performed in some embodiments;

[0027] FIG. 10B is a schematic tensor breakout illustrating component masking values in accordance with a second masking strategy, as may be performed in some embodiments;

[0028] FIG. 10C is a schematic tensor breakout illustrating component masking values in accordance with a third masking strategy, as may be performed in some embodiments;

[0029] FIG. 10D is a schematic tensor breakout illustrating component masking values in accordance with a fourth masking strategy, as may be performed in some embodiments;

[0030] FIG. 10E is a pair of schematic intensity and depth tensors and their corresponding masked representations, as may be implemented in some embodiments;

[0031] FIG. 11 is a schematic processing block diagram illustrating data flow during the loss calculation for multimodal depth and intensity training of a machine learning system, as may be implemented in some embodiments;

[0032] FIG. 12 is a schematic processing block diagram illustrating data flow during a multimodal round of training, as may be implemented in some embodiments;

[0033] FIG. 13 is a flow diagram illustrating various operations in an example process for performing a round of loss determination during training, as may be implemented in some embodiments;

[0034] FIG. 14 is a flow diagram illustrating various operations in an example process for preparing an application-specific machine learning system, as may be performed in some embodiments;

[0035] FIG. 15 is a table depicting comparative results generated in connection with an example prototype implementation of an embodiment; and

[0036] FIG. 16 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments.

[0037] The specific examples depicted in the drawings have been selected to facilitate understanding. Consequently, the disclosed embodiments should not be restricted to the specific details in the drawings or the corresponding disclosure. For example, the drawings may not be drawn to scale, the dimensions of some elements in the figures may have been adjusted to facilitate understanding, and the operations of the embodiments associated with the flow diagrams may encompass additional, alternative, or fewer operations than those depicted here. Thus, some components and / or operations may be separated into different blocks or combined into a single block in a manner other than as depicted. The embodiments are intended to cover all modifications, equivalents,and alternatives falling within the scope of the disclosed examples, rather than limit the embodiments to the particular examples described or depicted.DETAILED DESCRIPTIONEmbodiments Overview

[0038] Various of the disclosed embodiments seek to train machine learning systems to recognize semantic relations within surgical theater data (e.g., relations within a single data modality or across two or more modalities). Machine learning systems trained to recognize these semantic relations may be then more amenable to training for a particular application with relatively little or no labeled data. More specifically, embodiments may compress, then reconstruct, the available surgical theater data, thereby learning a latent space representation of the surgical theater data capturing salient semantic relations. By further employing certain masking strategies, comparative loss determination methods, and multimodal surgical theater data groupings disclosed herein, embodiments may significantly improve downstream training for a wide variety of applications. Thus, by using various of the disclosed embodiments, it may be possible to create surgical theater machine learning applications that were previously unfeasible in low-labeled surgical theater data domains.Example Surgical Theaters Overview

[0039] FIG. 1A is a schematic view of various elements appearing in a surgical theater 100a during a surgical operation as may occur in relation to some embodiments. Particularly, FIG. 1A depicts a non-robotic surgical theater 100a, wherein a patient-side surgeon 105a performs an operation upon a patient 120 with the assistance of one or more assisting members 105b, who may themselves be surgeons, physician’s assistants, nurses, technicians, etc. The surgeon 105a may perform the operation using a variety of tools, e.g., a visualization tool 110b such as a laparoscopic ultrasound, visual image acquiring endoscope, etc., and a mechanical instrument 110a such as scissors, retractors, a dissector, etc.

[0040] The visualization tool 110b provides the surgeon 105a with an interior view of the patient 120, e.g., by displaying visualization output from an imaging devicemechanically and electrically coupled with the visualization tool 110b. The surgeon may view the visualization output, e.g., through an eyepiece coupled with visualization tool 110b or upon a display 125 configured to receive the visualization output. For example, where the visualization tool 110b is a visual image acquiring endoscope, the visualization output may be a color or grayscale image. Display 125 may allow assisting member 105b to monitor surgeon 105a’s progress during the surgery. The visualization output from visualization tool 110b may be recorded and stored for future review, e.g., using hardware or software on the visualization tool 110b itself, capturing the visualization output in parallel as it is provided to display 125, or capturing the output from display 125 once it appears on-screen, etc. While two-dimensional video capture with visualization tool 110b may be discussed extensively herein, as when visualization tool 110b is a visual image endoscope, one will appreciate that, in some embodiments, visualization tool 110b may capture depth data instead of, or in addition to, two-dimensional image data (e.g., with a laser rangefinder, stereoscopy, etc.).

[0041] A single surgery may include the performance of several groups of actions, each group of actions forming a discrete unit referred to herein as a task. For example, locating a tumor may constitute a first task, excising the tumor a second task, and closing the surgery site a third task. Each task may include multiple actions, e.g., a tumor excision task may require several cutting actions and several cauterization actions. While some surgeries require that tasks assume a specific order (e.g. , excision occurs before closure), the order and presence of some tasks in some surgeries may be allowed to vary (e.g., the elimination of a precautionary task or a reordering of excision tasks where the order has no effect). Transitioning between tasks may require the surgeon 105a to remove tools from the patient, replace tools with different tools, or introduce new tools. Some tasks may require that the visualization tool 110b be removed and repositioned relative to its position in a previous task. While some assisting members 105b may assist with surgery-related tasks, such as administering anesthesia 115 to the patient 120, assisting members 105b may also assist with these task transitions, e.g., anticipating the need for a new tool 110c.

[0042] Advances in technology have enabled procedures such as that depicted in FIG. 1A to also be performed with robotic systems, as well as the performance ofprocedures unable to be performed in non-robotic surgical theater 100a. Specifically, FIG. 1 B is a schematic view of various elements appearing in a surgical theater 100b during a surgical operation employing a robotic surgical system, such as a da Vinci™ surgical system, as may occur in relation to some embodiments. Here, patient side cart 130 having tools 140a, 140b, 140c, and 140d attached to each of a plurality of arms 135a, 135b, 135c, and 135d, respectively, may take the position of patient-side surgeon 105a. As before, one or more of tools 140a, 140b, 140c, and 140d may include a visualization tool (here visualization tool 140d), such as a visual image endoscope, laparoscopic ultrasound, etc. An operator 105c, who may be a surgeon, may view the output of visualization tool 140d through a display 160a upon a surgeon console 155. By manipulating a hand-held input mechanism 160b and pedals 160c, the operator 105c may remotely communicate with tools 140a-d on patient side cart 130 so as to perform the surgical procedure on patient 120. Indeed, the operator 105c may or may not be in the same physical location as patient side cart 130 and patient 120 since the communication between surgeon console 155 and patient side cart 130 may occur across a telecommunication network in some embodiments. An electronics / control console 145 may also include a display 150 depicting patient vitals and / or the output of visualization tool 140d.

[0043] Similar to the task transitions of non-robotic surgical theater 100a, the surgical operation of theater 100b may require that tools 140a-d, including the visualization tool 140d, be removed or replaced for various tasks as well as new tools, e.g., new tool 165, be introduced. As before, one or more assisting members 105d may now anticipate such changes, working with operator 105c to make any necessary adjustments as the surgery progresses.

[0044] Also similar to the non-robotic surgical theater 100a, the output from the visualization tool 140d may here be recorded, e.g., at patient side cart 130, surgeon console 155, from display 150, etc. While some tools 110a, 110b, 110c in non-robotic surgical theater 100a may record additional data, such as temperature, motion, conductivity, energy levels, etc., the presence of surgeon console 155 and patient side cart 130 in theater 100b may facilitate the recordation of considerably more data than is only output from the visualization tool 140d. For example, operator 105c’s manipulationof hand-held input mechanism 160b, activation of pedals 160c, eye movement with respect to display 160a, etc., may all be recorded. Similarly, patient side cart 130 may record tool activations (e.g., the application of radiative energy, closing of scissors, etc.), movement of instruments, etc., throughout the surgery. In some embodiments, the data may have been recorded using an in-theater recording device, which may capture and store sensor data locally or at a networked location (e.g., software, firmware, or hardware configured to record surgeon kinematics data, console kinematics data, instrument kinematics data, system events data, patient state data, etc., during the surgery).

[0045] Within each of theaters 100a, 100b, or in network communication with the theaters from an external location, may be computer systems 190a and 190b, respectively (in some embodiments, computer system 190b may be integrated with the robotic surgical system, rather than serving as a standalone workstation). As will be discussed in greater detail herein, the computer systems 190a and 190b may facilitate, e.g., data collection, data processing, etc.

[0046] Similarly, many of theaters 100a, 100b may include sensors placed around the theater, such as sensors 170a and 170c, respectively, configured to record activity within the surgical theater from the perspectives of their respective fields of view 170b and 170d. Sensors 170a and 170c may be, e.g., visual image sensors (e.g., color or grayscale image sensors), depth-acquiring sensors (e.g., via stereoscopically acquired visual image pairs, via time-of-flight with a laser rangefinder, structural light, etc.), or a combination of visual image and depth-acquiring sensors (e.g., red-green-blue-depth RGB-D sensors). In some embodiments, sensors 170a and 170c may also include audio acquisition sensors or sensors specifically dedicated to audio acquisition may be placed around the theater. A plurality of such sensors may be placed within theaters 100a, 100b, possibly with overlapping fields of view and sensing range, to achieve a more holistic assessment of the surgical theater. For example, depth-acquiring sensors may be strategically placed around the theater so that their resulting depth frames at each moment may be consolidated into a single three-dimensional virtual element model depicting objects in the surgical theater. Similarly, sensors may be strategically placed in the theater to focus upon regions of interest. For example, sensors may be attached to display 125, display 150, or patient side cart 130 with fields of view focusing upon thepatient 120’s surgical site, attached to the walls or ceiling, etc. Similarly, sensors may be placed upon console 155 to monitor the operator 105c. Sensors may likewise be placed upon movable platforms specifically designed to facilitate orienting of the sensors in various poses within the theater.

[0047] For clarity, as used herein, a “pose” refers to the translational position and rotational orientation of a body. For example, in a three-dimensional space, one may represent a pose with six total degrees of freedom. One will readily appreciate that poses may be represented using a variety of data structures, e.g., with matrices, with quaternions, with vectors, with combinations thereof, etc. Thus, in some situations, when there is no rotation, a pose may comprise only a translational component. Conversely, when there is no translation, a pose may comprise only a rotational component.

[0048] Similarly, for clarity, “theater-wide” sensor data refers herein to data acquired from one or more sensors configured to monitor a specific region of the theater (the region encompassing all, or a portion, of the theater) exterior to the patient, to personnel, to equipment, or to any other objects in the theater, such that the sensor can perceive the presence within, or passage through, at least a portion of the region of the patient, personnel, equipment, or other objects, throughout the surgery. Sensors so configured to collect such “theater-wide” data are referred to herein as “theater-wide sensors.” For clarity, one will appreciate that the specific region need not be rigidly fixed throughout the procedure, as, e.g., some sensors may cyclically pan their field of view so as to augment the size of the specific region, even though this may result in temporal lacunae for portions of the region in the sensor’s data (lacunae which may be remedied by the coordinated panning or fields of view of other nearby sensors). Similarly, in some cases, personnel or robotics systems may be able to relocate theater-wide sensors, changing the specific region, throughout the procedure, e.g., to better capture different tasks. Accordingly, sensors 170a and 170c are theater-wide sensors configured to produce theater-wide data. “Visualization data” refers herein to visual intensity image or depth image data captured from a sensor. Thus, visualization data may or may not be theater-wide data. For example, visualization data captured at sensors 170a and 170c is theater-wide data, whereas visualization data captured via visualization tool 140d would not be theater-wide data (for at least the reason that the data is not exterior to the patient).Example Theater-Wide Sensor Topologies

[0049] For further clarity regarding theater-wide sensor deployment, FIG. 2A is a schematic depth map rendering from an example theater-wide sensor perspective 205 as may be used in some embodiments. Specifically, this example depicts depth values corresponding to an electronics / control console 205a (e.g., the electronics / control console 145) and a nearby tray 205b, and cabinet 205c. Also within the field of view are depth values associated with a first technician 205d, presently adjusting a robotic arm (associated with depth values 205f) upon a robotic surgical system (associated with depth values 205e). Team members, with corresponding depth values 205g, 205h, and 205i, likewise appear in the field of view, as does a portion of the surgical table 205j. Depth values 205I corresponding to a movable dolly and a boom with a lighting system’s depth values 205k also appear within the field of view.

[0050] The theater-wide sensor capturing the perspective 205 may be only one of several sensors placed throughout the theater. For example, FIG. 2B is a schematic top- down view of objects in the theater at a given moment during the surgical operation. Specifically, the perspective 205 may have been captured via a theater-wide sensor 220a with corresponding field of view 225a. Thus, for clarity, cabinet depth values 205c may correspond to cabinet 210c, electronics / control console depth values 205a may correspond to electronics / control console 210a, and tray depth values 205b may correspond to tray 210b. Robotic system 21 Oe may correspond to depth values 205e, and each of the individual team members 210d, 210g, 210h, and 21 Oi may correspond to depth values 205d, 205g, 205h, and 205i, respectively. Similarly, dolly 2101 may correspond to depth values 2051. Depth values 205j may correspond to table 21 Oj (with an outline of a patient shown here for clarity, though the patient has not yet been placed upon the table corresponding to depth values 205j in the example perspective 205). A top-down representation of the boom corresponding to depth values 205k is not shown for clarity, though one will appreciate that the boom may likewise be considered in various embodiments.

[0051] As indicated, each of the sensors 220a, 220b, 220c is associated with different fields of view 225a, 225b, and 225c, respectively. The fields of view 225a-c maysometimes have complementary characters, providing different perspectives of the same object, or providing a view of an object from one perspective when it is outside, or occluded within, another perspective. Complementarity between the perspectives may be dynamic both spatially and temporally. Such dynamic character may result from movement of an object being tracked, but also from movement of intervening occluding objects (and, in some cases, movement of the sensors themselves). For example, at the moment depicted in FIGs. 2A and 2B, the field of view 225a has only a limited view of the table 210j, as the electronics / control console 210a substantially occludes that portion of the field of view 225a. Consequently, in the depicted moment, the field of view 225b is better able to view the surgical table 21 Oj. However, neither field of view 225b nor 225a has an adequate view of the operator 21 On in console 210k. To observe the operator 210n (e.g., when they remove their head in accordance with “head out” events), field of view 225c may be more suitable. However, over the course of the data capture, these complementary relationships may change. For example, before the procedure begins, electronics / control console 210a may be removed and the robotic system 21 Oe moved into the position 210m. In this configuration, field of view 225a may instead be much better suited for viewing the patient table 21 Oj than the field of view 225b. As another example, movement of the console 210k to the presently depicted pose of electronics / control console 210a may render field of view 225a more suitable for viewing operator 210n, than field of view 225c. Suitability of a field of view may thus depend upon the number and duration of occlusions, quality of the field of view (e.g., how close the object of interest is to the sensor), and movement of the object of interest within the theater. Such changes may be transitory and short in duration, as when a team member moving in the theater briefly occludes a sensor, or they may be chronic or sustained, as when equipment is moved into a fixed position throughout the duration of the procedure.

[0052] As mentioned, the theater-wide sensors may take a variety of forms and may, e.g., be configured to acquire visual image data, depth data, both visual and depth data, etc. One will appreciate that visual and depth image captures may likewise take on a variety of forms, e.g., to afford increased visibility of different portions of the theater. For example, FIG. 2C is a pair of images 250b, 255b depicting a grid-like pattern of orthogonal rows and columns in perspective, as captured from a theater-wide sensor having arectilinear view and a theater-wide sensor having a fisheye view, respectively. More specifically, some theater-wide sensors may capture rectilinear visual images or rectilinear depth frames, e.g., via appropriate lenses, post-processing, combinations of lenses and post-processing, etc. while other theater-wide sensors may instead, e.g., acquire fisheye or distorted visual images or rectilinear depth frames, via appropriate lenses, post-processing, combinations of lenses and post-processing, etc. For clarity, image 250b depicts a checkboard pattern in perspective from a rectilinear theater-wide sensor. Accordingly, the orthogonal rows and columns 250a shown here in perspective, retain linear relations with their vanishing points. In contrast, image 255b depicts the same checkboard pattern in the same perspective, but from a fish-eye theater-wise sensor perspective. Accordingly, the orthogonal rows and columns 255a, while in reality retaining a linear relationship with their vanishing points (as they appear in image 250b) appear here from the sensor data as having curved relations with their vanishing points. Thus, each type of sensor, and other sensor types, may be used alone, or in some instances, in combination, in connection with various embodiments.

[0053] Similarly, one will appreciate that not all sensors may acquire perfectly rectilinear, fisheye, or other desired mappings. Accordingly, checkered patterns, or other calibration fiducials (such as known shapes for depth systems), may facilitate determination of a given theater-wide sensor’s intrinsic parameters. For example, the focal point of the fisheye lens, and other details of the theater-wide sensor (principal points, distortion coefficients, etc.), may vary between devices and even across the same device over time. Thus, it may be necessary to recalibrate various processing methods for the particular device at issue, anticipating the device variation when training and configuring a system for machine learning tasks. Additionally, one will appreciate that the rectilinear view may be achieved by undistorting the fisheye view once the intrinsic parameters of the camera are known (which may be useful, e.g., to normalize disparate sensor systems to a similar form recognized by a machine learning architecture). Thus, while a fisheye view may allow the system and users to more readily perceive a wider field of view than in the case of the rectilinear perspective, when a processing system is considering data from some sensors acquiring undistorted perspectives and other sensors acquiring distorted perspectives, the differing perspectives may be normalized toa common perspective form (e.g., mapping all the rectilinear data to a fisheye representation or vice versa).Operational and Nonoperational Theater Data Overview

[0054] FIG. 3 is a schematic depiction of data that may be acquired during a surgically operative period, as may occur in connection with some embodiments. Specifically, over time 305, e.g., throughout the course of a day, an operating room, whether the robotic theater 100a or non-robotic theater 100b, may alternate between operative periods of surgical activity and nonoperative periods when surgery is not performed. In this example, during an initial preparatory pre-surgical period 310a at the beginning of the day, team members may prepare the room for the day’s surgeries. A first surgery may then be performed during the interval 315a by the same or different team members. An inter-surgical period 310b may then follow, during which the team (which may or may not include the same composition of team members as in the previous periods) prepares the theater for the next surgery, e.g., changing equipment configurations, wheeling patients in and out of the theater, etc. The alternating operative and nonoperative periods may then continue throughout the day, e.g., as reflected by surgical period 315b, the Nthsurgery 315c, and intervening ellipsis 310e (which may indicate the presence of additional surgical procedure intervals). At the day’s end, during a post-surgical period 31 Od, team members may, e.g., clean the theater, put away equipment, and ensure that requisite items are available and in order for the following day’s surgeries, etc.

[0055] During each of the surgical periods 315a-c, theater operations may generally produce data divided into two parallel groups: surgical equipment acquired data 375a and theater-wide data 375b. Each of these datasets may correspond to the performance of respective tasks. For example, the surgical equipment acquired data 375a, which, as mentioned need not necessarily be from a robotic system, may be acquired in connection with surgical tasks 370a, 370b, 370c, and 370d (intervening ellipsis 370e indicating the possibility of additional intervening tasks). As shown, each of tasks 370a-d may be associated with corresponding surgical data, such as visual image video data 320a-d (in some situations, depth video data may also, or alternatively, be available). Similarly,kinematics data 325a-d may be acquired from a robotic platform, a surgeon console, the instruments themselves, etc. Such data may indicate, e.g., the motion and poses of various instruments, end effectors, tools, etc., throughout each respective task. Similarly, system events data 335a-d may be acquired in connection with each task, indicating when instruments are activated (e.g., an endoscope position lock, a cauterization tool, surgical scissor activation, etc.). For clarity, ellipses 320e, 325e, and 335e again indicate the possibility of additional intervening tasks and corresponding data.

[0056] Thus, surgical data 375a generally corresponds to data generated by the actions of one or more surgeons. In contrast, the theater-wide data 375b may be acquired from one or more theater-wide sensors placed around the theater and may generally depict actions by team members within the theater, particularly in the performance of theater tasks 330a-d (ellipsis 330e indicating the possibility of additional tasks). As the tasks are different, though sometimes related, tasks 330a-d and 370a-d. In this example, there are three data streams 340a-d, 350a-d, and 360a-d (ellipses 380 indicating the possibility of additional data streams in some embodiments from other sensors, though one will appreciate that fewer than three streams may also appear in some embodiments) corresponding to three different theater-wide sensor systems having different corresponding poses within the theater (ellipses 340e, 350e, and 360e again indicating the possibility of additional datasets for each respective row of stream data). Though streams 340a-d, 350a-d, and 360a-d are shown here as visual image data, as previously discussed, one will appreciate that the streams may additionally, or alternatively, include, e.g., depth data.

[0057] While many of tasks 330a-d are performed in connection with or in anticipation of tasks 370a-d (though, naturally, not all are), tasks 330a-d performed in the theater may or may or not temporally correspond with the surgical tasks 370a-d, may be different in number, and may be different in their start and stop times. That is, though each of the surgical data 375a and theater-wide data 375b may be acquired relative to a common time keeping device, the start and end times of various tasks 330a-d and 370a- d may not correspond. For example, here, the surgical task 370c may require an imaging system. Accordingly, as indicated in the video data images 340b for task 330b, a team member has begun to move the imaging system into position for task 370c. Accordingly,despite their relation, the start and end times of the task 330b may occur well before task 370c. Thus, the start and end times, duration, and number of tasks 330a-d and tasks 370a-d, while sometimes correlated, may often differ. Finally, for clarity, note that, while shown here as linearly succeeding one another in time, one will appreciate that some tasks may proceed in parallel, e.g., where multiple team members are simultaneously performing their own role-specific tasks within the theater.

[0058] Thus, if one desires to recognize the tasks 330a-d or specific actions from the theater-wide sensor data streams 375b, and one has available the surgical data 375a, it may be possible for a reviewer or for a machine learning system to make partial inferences and to correspondingly identify and label at least certain collections of the theater-wide data 375b as being associated with one of tasks 330a-d (or other corresponding theater state, patient state, etc. of interest). Unfortunately, certainty in the annotations may require human review, and such manual inspection may, e.g., limit the annotated dataset size, slow such annotation, and introduce the risk of human error.

[0059] During the inter-surgical periods 310a-d, such annotation may be even more difficult, as the surgical data 375a may now be completely absent. For example, FIG. 4 is a schematic depiction of data that may be acquired during the surgically nonoperative period 310c. Here, the three data streams are still active, acquiring the respective datasets 425a-e, 430a-e, and 435a-e during the performance of various inter-surgical tasks 420a-e (again, each of ellipses 420f, 425f, 430f, and 435f indicating the possibility of additional intervening tasks and datasets). Because the corresponding surgical data 375a may not be available during these nonoperative periods, annotation may rely entirely upon human inspection or machine learning inspection of the theater-wide sensor datasets 425a-f, 430a-f, and 435a-f.Example Downstream Applications

[0060] One will appreciate a number of downstream machine learning applications, which may be performed upon surgical theater data, particularly in low data regimes, as enabled by various of the disclosed embodiments. For example, FIG. 5A is a schematic representation of a plurality of theater state recognitions, as may be performed in embodiments. A downstream machine learning system may receive theater-wide dataand be asked to identify a state of the theater from one or more frames. Here, for example, the system has been presented with six theater-wide visual intensity video frames 505a-f and asked to identify the corresponding state of the theater from a taxonomy (e.g., “Initial” setup, “Sterile Prep”, “Patient In”, etc.). Performing such taxonomic recognition, despite being in a low-data regime, may facilitate a wide variety of additional downstream applications and analyses (e.g., once the video data has been segmented into its respective taxonomic categories, assessment of the surgical team’s performance of various tasks relative to other hospitals, teams, configuration, etc. may be readily feasible)

[0061] As another example downstream application, FIG. 5B is a schematic representation of a plurality of object segmentation results, as may be performed in some embodiments. Specifically, analogous to a You Only Look Once (YOLO) object detection system (indeed, the neural network may be modified so as to resemble a YOLO architecture) FIG. 5B depicts a variety of theater-wide video depth frames wherein team members have been identified, and their corresponding depth values highlighted, in frames 550a-c. In depth frame 550d, the system has instead recognized a particular piece of equipment, here, the patient’s bed. As in the example of 5A, such a system may facilitate wide variety of downstream analyses. For example, knowing when team members are present, when equipment is in use, the relative orientations of the team members and equipment, the location and orientation of the patient, etc., may all readily facilitate more granular assessment of the surgical team’s performance.

[0062] Though the examples of FIGs. 5A and 5B refer to downstream classification applications, the reader will appreciate that applications other than classification may likewise be enabled by the disclosed embodiments. For example, once a neural network has been pre-trained to appreciate general semantic patterns, predictive applications, which seek to predict the course of the surgery, generative applications, which may seek to produce synthetic theater data, etc., may all be made more feasible.Example Downstream Application Theater Recognition Taxonomies

[0063] Thus, downstream surgical theater machine learning applications include, e.g.: segmenting images or video based upon the objects depicted therein; recognizingteam member’s roles; recognizing team member’s actions; determining the state of the patient; detecting adverse events within the theater, etc. Many of these downstream surgical theater applications may seek to classify data in accordance with a taxonomy. For example, surgical activity recognition may seek to classify the data as depicting one or more of a plurality of actions occurring within the theater.

[0064] As an example taxonomy (focusing mostly on nonoperative period activities), FIG 6A is a schematic representation of a collection of temporal intervals as may be used to assess activities in a surgical theater in some embodiments. The groupings depicted in FIGs. 6A-B can be especially useful for processing multi-modal surgical theater data. Specifically, the groupings of FIG. 6A provide an example “grammar” with which to describe workflow within the operating room. In this example, the activities do not generally overlap, though alternative grammars may accommodate such temporal overlap.

[0065] One will appreciate that the intervals may be applied cyclically in accordance with the alternating character of the operative and nonoperative periods in the theater described above in FIGs. 3 and 4. For example, initially, the surgical operation 315b may correspond to the interval 650e. Following the operation 315b’s completion, actions and corresponding data in the theater may be allocated to consecutive intervals 650a-d during the subsequent nonoperative period 310c. Data and actions in the next surgery, (e.g., surgery 315c if there are no intervening periods in ellipsis 31 Oe) may then be ascribed again to a second instance of the interval 650e, and so forth (consequently, data from each of the nonoperative periods 310b, 310b may be allocated to instances of intervals 650a-d). Intervals may also be grouped into larger intervals, as is the case here with the “wheels out to wheels in” interval 650f, which groups the intervals 650b and 650c, sharing the start time of interval 650b and the end time of interval 650c. The ability to automatically consolidate surgical theater data into this example taxonomy may facilitate a wide variety of downstream applications. Where such annotated data is readily available, supervised learning methods may suffice to enable such applications, but as will be discussed, such labeled data is often unavailable.

[0066] To provide further context, FIG. 6B depicts another more granular action taxonomy based upon the example taxonomy of FIG. 6A. Specifically, the durations of each of intervals 650a-e may be determined based upon respective start and end times of various tasks or actions within the theater. Naturally, when the intervals 650a-e are used consecutively, the end time for a preceding interval (e.g., the end of interval 650c) may be the start time of the succeeding interval (e.g., the beginning of interval 650d). When coupled with a task action grouping ontology, theater-wide data may be readily grouped into meaningful divisions for downstream analysis. This may facilitate, e.g., consistency in verifying that team members have been adhering to proposed feedback, as well as computer-based verification of the same, across disparate theaters, team configurations, etc. As will be explained, some task actions may occur over a period of time (e.g., cleaning), while others may occur at a specific moment (e.g., entrance of a team member).

[0067] Specifically, FIG. 6B depicts four high-level task action classes: post-surgery 620, turnover 625, pre-surgery 610, and surgery 615. Surgery 615 may include the tasks or actions 615a-i. Specifically, the task “first cut” 615a indicates a time when the first incision upon the patient occurs. The task “port placement” 615b indicates a time when a first port is placed into the patient. The task “rollup” 615c is the duration from which a team member begins moving a robotic system to a time when the robotic system assumes the pose it will use during at least an initial portion of the surgical procedure. The task “room prep” 615d begins with the first surgery preparation action specific to the surgery being performed and concludes with the last preparation action specific to the surgery being performed. The task “docking” 615e begins when a team member starts docking a robotic system and concludes when the robotic system is docked. The task “surgery” 615f begins with the first incision and ends with the final closure of the patient. Naturally, in many taxonomies, block 615f may be further broken down into considerably more action and task divisions (which may, e.g., be facilitated by the availability of surgical data 375a). The task “undocking” 615g begins when a team member starts to undock a robotic system and concludes when the robotic system is undocked. The task “rollback” 615h begins when a team member begins moving a robotic system away from a patient andconcludes when the robotic system assumes a pose it will retain until turnover begins. The task “patient close” 615a begins and ends with the final suturing of the patient.

[0068] Within the post-surgical class grouping 620, the task “robot undraping” 620a starts when a team member first begins undraping a robotic system and ends when the robotic system is undraped. The task “patient out” 620b starts and ends when the patient leaves the theater. The task “patient undraping” 620c starts when a team member first begins undraping the patient and ends when the patient is undraped.

[0069] Within the turnover class grouping 625, the task “clean” 625a starts when the first team member begins cleaning equipment in the theater and concludes when the last team member (which may be the same team member) completes the last cleaning of any equipment. The task “idle” 625b starts when team members are not performing any other task and concludes when they begin performing another task. The task “turnover” 605a starts when the first team member begins resetting the theater from the last procedure and concludes when the last team member (which may be the same team member) finishes the reset. The task “setup” 605b starts when the first team member begins changing the pose of equipment to be used in a surgery and concludes when the last team member (which may be the same team member) finishes the last equipment pose adjustment. The task “sterile prep” 605c starts when the first team member begins cleaning the surgical area and concludes when the last team member (which may be the same team member) finishes cleaning the surgical area. Additionally, while shown here in linear sequences, one will appreciate that task actions within the classes may proceed in orders other than that shown or, in some instances, may refer to temporal periods which may overlap.

[0070] Within pre-surgery class grouping 610, the task “patient in” 610a starts and ends when the patient first enters the theater. The task “robot draping” 610b starts when a team member begins draping the robotic system and concludes when draping is complete. The task “intubate” 610c starts when intubation of the patient begins and concludes when intubation is complete. The task “patient prep” 61 Od starts when a team member begins preparing the patient for surgery and concludes when preparations arecomplete. The task “patient draping” 61 Oe starts when a team member begins draping the patient and concludes when the patient is draped.

[0071] Thus, as indicated by the respective arrows in FIG. 6B, the intervals of FIG. 6A may be allocated as follows. “Skin-close to patient-out” 650a may begin at the last closing operation 615i of the previous surgery interval and concludes with the patient’s departure from the theater (e.g., from the end of the last suture at block 615i until the patient has departed at block 620b). Similarly, the interval “Patient-out to case-open” 650b may begin when the patient’s departure from the theater at block 620b and concludes with the start of sterile prep at block 605c for the next case.

[0072] The interval “case-open to patient-in” 650c may begin with the start of the sterile prep at block 605c and conclude with the start of the new patient entering the theater at block 610a. The interval “patient-in to skin cut” 650d may begin when the new patient enters the theater at block 610a and concludes at the start of the first cut at block 615a. The surgery itself may occur during the interval 650e as shown. As previously discussed, the “wheels out to wheels in” interval 650f is the interval from the start of “patient out to case open” 650b and concludes with the end of “case open to patient in” 650c

[0073] Like the more encompassing intervals of FIG. 6A, it would often be desirable to have a machine learning classifier that could recognize the more granular activities of FIG. 6B, e.g., from the theater-wide sensor data. Indeed, machine learning systems able to recognize personnel in the theater, equipment in the theater, adverse events the theater-wide video data, operating room configurations, etc., during operative periods, during nonoperative periods, and during both nonoperative and operative periods, would all facilitate many downstream applications. However, without adequately labeled data, preparing such machine learning systems through supervised methods alone may not be feasible. Again, while the above and other examples herein may focus upon theater-wide data to facilitate the reader’s understanding, various embodiments may be applied to nontheater-wide data (e.g., video of the patient interior from a visualization tool, kinematics data, etc.), or a combination of theater-wide and non-theater-wide surgical theater data.Example Dataset Properties

[0074] As discussed, many surgical theater data machine learning applications remain unfeasible due to their dependence upon large amounts of labeled surgical theater data. For clarity, FIGs. 7A and 7B schematically depict various types of labeled dataset instances.

[0075] Specifically, FIG. 7A schematically illustrates an example of an incoming surgical theater data item 730, in the context of an example classification system. Here, the data item 730 includes the theater data to be classified 730a (here, represented by a question mark as it arrives before the classifier with an unknown categorization), which, as mentioned, may be an individual visual image or depth frame, video of each, etc. A classifier machine learning system, for example, may seek to identify the probabilities 730b that the data 730a should be associated with one of a plurality of classifications, for example, one of the theater state categories of FIGs. 5A, 6A-B or perhaps one of the object classifications of FIG. 5B. Though only three categories A, B, C are shown here schematically for the reader’s understanding, the reader will appreciate that the probabilities may instead be spread across only two categories, or across many more categories. Similarly, though reference to classification applications will regularly be made herein, as a recurring contextual example to facilitate the reader’s understanding, as mentioned, the reader will appreciate that the downstream machine learning system may perform other applications than classification, e.g., synthetic data generation, data extrapolation and future prediction, etc.

[0076] To train a classifier in supervised manner to assign probabilities to categories like those discussed in FIGs. 5A-B and 6A-B, one would ideally have available an “entirely labeled dataset” 735 as shown in FIG. 7B, with which to perform supervised training. In such a dataset, each of the training instances 735a-c presents surgical theater data with a known classification (ellipsis 735d indicating the possibility of additional intervening instances). Thus, the data instance 735a includes surgical theater data known to be in class C, and consequently the known probability for this instance is 1 .0 for the class C and 0.0 for the other classes. Similarly, the instance 735b includes theater-wide data known to be in class B and consequently the known probability for this instances is 1.0 for the class B, etc.

[0077] Unfortunately, as discussed above with respect to FIGs. 3 and 4, it is often very difficult to acquire an “entirely labeled dataset” 735 for the surgical theater data 375a- b. Surgical theater datasets, even if including only theater-wide surgical data, often contain millions of images from various surgical settings, procedures, ORs, etc. rendering manual annotation by an expert difficult if not unfeasible.

[0078] As mentioned, with the assistance of the surgical data 375a, it may be possible to infer labels for some portions of the data 375a-b simply by inspection (e.g., some tools will only be activated during surgery or during certain surgical tasks). Such inferences may accommodate some automated classification, providing some labeled data instances, which may then be additionally supplemented by the expensive, and timeintensive, expert human annotation. However, as shown in FIG. 7B, this may result in only a “partially labeled dataset” 740 wherein only a few of the instances have proper labels. Here, for example, it is known that the instance 740a depicts the C class, and should receive a corresponding label for training. However, it is not known what class, and consequently what label, is appropriate for the instances 740b and 740c (ellipsis 740d indicating the possibility of additional intervening instances).

[0079] Indeed, given the difficulties in labeling surgical theater data, it may sometimes be the case that none of the available data has been labeled, precipitating an unlabeled dataset 745, wherein the nature of the theater-wide data for all the instances 745a-d is unknown, and consequently, it is not clear what classifications they should receive. Indeed, as explained above, expert annotation can be onerous and expensive, and so much of the acquired surgical theater data will be of the unlabeled dataset form 745.

[0080] Various of the disclosed embodiments reduce the amount of annotated data needed to achieve a desirable level of machine learning system performance in downstream applications. As will be discussed in the following section, various embodiments employ a coupled encoder and decoder, or an autoencoder, approach which leverages the copiousness of the available unlabeled data to infer semantic content applicable to downstream training. Such downstream training may then avail itself of whatever, albeit small amount, of labeled data is available. In this manner, thedownstream applications may achieve a superior performance than would otherwise be possible in such a low-labeled data regime.Example Unsupervised Pre-Training Machine Learning Methodologies

[0081] As much of the surgical theater data 375a-b will be of the unlabeled dataset form 745, various of the disclosed embodiments contemplate various unsupervised learning operations to acclimate the machine learning system to the desired grammar of various downstream applications. For intuition, one would expect diverse machine learning applications within the surgical theater to rely upon similar “semantic knowledge” unique to the theater context (e.g., “what a robotic surgical system looks like”, “where the surgeon is when the procedure begins”, “how people move throughout the theater”, “how patients appear within the OR and what orientations they usually assume”, etc.).

[0082] More specifically, various of the disclosed embodiments may train a machine learning system to reduce the surgical theater data to a parsimonious “compressed” form, and to then rederive the original surgical data from the parsimonious form. Training in this manner may force the system to recognize those most salient semantic characteristics of the surgical theater data, and such recognition may then be leveraged when performing downstream training for a more specific application task upon a much smaller labeled dataset than might otherwise be required. In a sense, the preliminary unsupervised training will have “primed” the machine learning system for its downstream task, such that less labeled data is needed for it to achieve a desired performance during training. Various clustering approaches to unlabeled semantic content recognition, such as Swapping Assignments between multiple Views of the same image (SwAV), as discussed in Caron, Mathilde, et al. "Unsupervised Learning of Visual Features by Contrasting Cluster Assignments." arXiv™ preprint arXivTM:2006.09882 (2020), require that the designer anticipate the number of prototype clusters ex ante. Various of the disclosed reconstructive approaches do not impose this restriction upon the designer, instead better allowing the system to organically adapt itself to whatever semantic content genuinely underlies the data. Accordingly, it has been found that various of the disclosed embodiments generalize more readily than certain counterpart SwAV implementations.

[0083] Again, though classification of specific classes is often presented herein for ease of the reader’s comprehension, one will appreciate that the disclosed approaches may be used for a wide variety of downstream application e.g., two-dimensional visual image or depth data segmentation, three-dimensional visual image video or depth video data segmentation, surgical activity recognition, semantic segmentation, etc. In addition, the reader will appreciate that various of the disclosed approaches also address privacy concerns, an important issue in the healthcare context, as the data and methods use preserve the anonymity of the individuals appearing within the theater.

[0084] Thus, where labeled dataset 735 is available, one may train a classifier in a supervised manner, as shown in the process 805 of FIG. 8A. In such a process 805, one can simply perform supervised training using the labeled data at block 805a and then deploy the classifier for classification at block 805b. While suitable when large amounts of labeled data are available, as mentioned, taking an initially untrained classifier to a trained state using only the process of 805 with only small amounts of labeled data is generally infeasible.

[0085] Where partially labeled data 740 is available, however, a combination of unsupervised and supervised training may sometimes be applied to produce a machine learning classifier of adequate performance. Specifically, as shown in process 810 of FIG. 8B, rather than proceed to the supervised classification of process 805 directly, one may instead first perform preliminary context training 810a, which may be unsupervised (or in some cases semi-supervised if some labeled data is available).

[0086] During preliminary context training 810a, at block 815a, the system may perform unsupervised training upon the classifier using the unlabeled data. Particularly, anticipating that the data will fall into one of the groupings of the downstream task (e.g., one of the actions or intervals of FIGs. 6A-B), unsupervised training may seek to first train the machine learning system to recognize general surgical theater semantic content, even though explicit identifiers for such semantic patterns are not known. For example, patient movement, a semantic pattern in the data, may occur only during “patient in” 610a and “patient out” 620b actions. Accordingly, a machine learning system trained to grossly distinguish between data where the patient is in motion, from data where the patient isnot in motion, will already be much better preprepared to distinguish actions “patient in” 610a and “patient out” 620b from other actions. Thus, following the pre-training of block 815a, the classifier may be adapted (e.g., as described in greater detail below) at block 815b to a form suitable for supervised training, and this modified architecture used for task specific training 810b, which, like process 805, performs supervised training with the available known labeled data at block 815c before deploying the classifier for its desired downstream purpose at block 815d.

[0087] Indeed, using some embodiments discussed herein, the unsupervised approach may be so successful for the purposes of some applications that the system may be deployed without performing task-specific training 810b. This may occur, e.g., when the unsupervised identified predictions sufficiently correspond to the desired downstream behavior that a human reviewer can simply annotate the predictions by inspection with their appropriate labels. Some de minimis architecture adjustments or post-processing may be used to finesse the unsupervised groupings to a desired form in these instances. For many applications, however, it will be necessary to perform additional training per task-specific training 810b.

[0088] Before describing various encoder I decoder embodiments for preliminary unsupervised context training in detail, FIG. 8C first provides the reader with orienting context by describing a general adaptation process, as may be employed in some embodiments. Specifically, FIG. 8C depicts a machine learning system during deployed 820 (corresponding, e.g., to the classifier after block 815d), supervised training 825 (corresponding, e.g., to the state of the machine learning system at block 815c), adaption to classification 830 (corresponding, e.g., to the state of the machine learning system at block 815b), and unsupervised 870 training states (corresponding, e.g. , to the state of the machine learning system at block 815a). The reader will appreciate that, in some implementations and embodiments, the training 870 may include some labeled data, and thus be “semi-supervised.” For example, rather than proceeding sequentially, as will be discussed in this example, one may alternate between the unsupervised and supervised training, producing a “semi-supervised” training approach (e.g., successively applying and removing the adaptations 830 during the training).

[0089] In this example, the training processes 870 and 825 will produce a machine learning system, e.g., classifier 820b, which, when presented with surgical theater data 820a (kinematics data, system event data, theater-wide visual intensity and depth data, combinations of theater-wide, kinematics, and system event data, etc.) of an unknown class (here, the “C” merely indicating the correct classification), is able to perform its desired task, here, a meaningful classification prediction. For example, the classifier 820b successfully identifies the C category as the most probable classification in its probability outputs 820c, with a probability of 0.7. As discussed with respect to process 805 and block 815c of process 810, such a classifier may be created with supervised training 825. Specifically, where known classified data instances are available, such as instance 825a, known to be associated with the class C, then during training the system may compare the classifier’s output 825c with the labeled data’s known label 825d and use the difference 825e to update the weights of the classifier 825b, e.g., using known backpropagation techniques. Completion of this training 825 may produce 840c the trained and deployed classifier 820b.

[0090] As mentioned, where inadequate amounts of labeled data are available, however, or where one desires to make the most advantage of what labeled data is available during the supervised training 825 (initially, or during subsequent online training), then unsupervised training 870 may be used to place the classifier into a state more amendable to supervised training upon a limited number of labeled instances. For example, from FIG. 6A one could recognize that the data will either be from one of the five classification’s 650a-e, from one of the finite activity tasks in FIG. 6B, etc., each of which may be associated with latent semantic characteristics. Some such latent semantic characteristics may be unique to a classification (e.g., patient absence during certain nonoperative periods), while others may be common to all the classifications, but with some variation (e.g., team member movement). Thus, during unsupervised training, it may be possible to train a machine learning system to identify a latent encoded representation, which may be compressed relative to (e.g., possess fewer dimensions than) the input tensor of the unlabeled data, suitable for reconstruction of the original unlabeled data, which indirectly captures various of these latent semantic features in theencoded representation (e g., motion of the patient, motion of equipment, lack of motion by a surgeon, instrument size and rigidity, robotic system pose patterns, etc.).

[0091] In this example, the unsupervised preliminary training 870 includes an encoder-decoder machine learning system 850, such as an autoencoder, or a distinct encoder and decoder. Here, the unlabeled data is shown as a plurality of tensors, represented by the tensor 855a, the tensor 855c, and ellipsis 855b indicate the possibility of additional tensors. For clarity, a “tensor” here simply refers to a data input having one or more dimensions and should not be confused, e.g., with the “metric tensor” of differential geometry (consequently, no corresponding metric or manifold is implied for interpreting the tensor). Thus a “matrix” is just a two-dimensional tensor. For example, the tensor 855c may comprise successive video image frames, thus constituting a three- dimensional tensor (two dimensions for each frame and a third temporal dimension corresponding to each frame’s relative time of acquisition).

[0092] Encoder 860a may be configured to receive the tensor 855c (e.g., where the input tensor is of three dimensions, receiving the tensor in a Conv3d layer, reformatting the tensor for receipt in a linear layer with input nodes corresponding to the product of all the tensor’s dimensions, etc.). The encoder 860a may then produce a compressed, encoded representation 860c. For example, as in an autoencoder, the number of layers or nodes in layers between the encoder 860a and decoder 860b may simply be fewer dimensions than those at the input to encoder 860a, thus compelling the input data to be represented in a “reduced latent space” form. Presented with this reduced form representation, the decoder 860b may then seek to recover the data of the original tensor 855c, here shown as the reconstructed tensor 855d.

[0093] By determining a difference 875a between the original tensor 855c and its recovered counterpart 855d, one may assess the quality of the encoder 860a and decoder 860b’s current operation (and by implication, the quality of the latent representation 860c). Initially during training, one will not expect the reconstructed counterpart tensor 855d to be very similar to the original tensor 855c, and so the difference 875a and associated loss 875b may be large, which is then used to update the encoder 860a and decoder 860b as indicated by the arrows 875c (e.g., viabackpropagation). Naturally, though only one reconstruction of a single instance is shown here, the reader will appreciate, as will be described in greater detail, that the loss 875b may consolidate the differences 875a across multiple tensors within a training batch.

[0094] Such iterative reconstruction, loss determination, and updating of the encoder 860a and decoder 860b (which, again, may be a single autoencoder neural network system, rather than two distinct systems), may eventually result in an encoder / decoder system 850 able to relatively consistently reconstruct an input tensor using its corresponding compressed latent space representation. Once in this satisfactory state, as shown by arrow 840a, the system 850 may then be adapted 830 for use in the downstream application, e.g., as part of the classifier supervised training 825. Such adaptation may be effected, e.g., via structural modification at block 830b (e.g., appending one or more layers to the decoder 860b consistent with the number of classes for classification or the type of desired output, removing decoder 860b and attaching a new neural network for interpretation of the encoded representation 860c, etc.). Such adaption may then produce a machine learning system 825b suitable for the supervised training on the available labeled data.Example Reconstruction Based Unsupervised Training

[0095] While some embodiments may employ encoding and decoding operations as described in FIG. 8C without masking, many embodiments may employ masking, or otherwise compel reconstruction from less than all of the input data, so as to better encourage the system to infer semantic relations between the data. Particularly, FIG. 9 is a schematic block diagram illustrating an example loss determination iteration during unsupervised training (e.g., as may occur within the system 850) of a reconstructive system (again, for a single instance to facilitate comprehension), as may be implemented in some embodiments. Specifically, in this example, a tensor 905a, denoted V, and corresponding to the input tensor 855c comprises surgical theater-wide data visual intensity video image frames, e.g., as captured by one of sensors 220a-c. Again, while visual intensity video is used in this example, one will appreciate that depth, or other data, including non-theater-wide surgical data, may be used instead. Here, the successive frames of video (possibly, e.g., down-sampled by decimation or up-sampled viainterpolation) appear successively in the temporal dimension of the tensor, with their respective two-dimensional intensity values provided at each temporal instance (in some embodiments each image may be associated with color pixels, themselves comprising three dimensions, and so the number of the tensor dimensions may be correspondingly increased).

[0096] Generally, the latent space feature optimization training system 905 corresponding to encoder-decoder machine learning system 850, may tokenize 910a the tensor 905a to produce N tokens 950a, denoted X in the figure (while self-evident, for clarity, like other boundary variables used herein to facilitate the reader’s comprehension, the reader will naturally appreciate that the N tokens referenced here need not be the same number N of surgeries referred to in FIG. 3). The tokens 950a may comprise, e.g., discrete portions of the tensor 905a when divided along the spatial dimensions of the frames, resulting, e.g., in columns with a length along the temporal dimension, but a height and width of less than all the height and width of the frames. These tokenized columns may or may not extend for the full temporal duration of the tensor. For example, if the spatial dimensions were equal and divided into four equal regions, and each columnar token were extended for only half of the total temporal length of the tensor 905a, then eight tokens would result (N=8).

[0097] Spatiotemporal sampling 910b of these tokens may then result in a selection 950b, denoted Xvof less than all of these tokens 950a. For example, the spatiotemporal sampling 910b may randomly select the tokens, select the tokens in accordance with a pre-chosen pattern or logic, select the tokens in accordance with a rotating selection of patterns, etc. Such sampling may be part of a masking operation, e.g., as will be described in greater detail herein with respect to FIGs. 10A-E. The system may then present the token selection 950b to an encoder system 910c (e.g., to a “cube embedding layer” or Conv3D layer, etc.), in this example, a Vision Transformer (ViT) model pretrained on, e.g., the Kinetics-400 dataset, or the lmageNet-21 k dataset, as was described in Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv™ preprint arXiv:2010.11929 (2020). This encoder may then produce the latent space embeddings of the sampled tokens 950c, here denoted Zv.

[0098] The system may then combine the sampled tokens’ latent space embeddings 950c with representations of the tokens not selected by the spatiotemporal sampling 91 Ob to produce a “complete” latent representation 950d for decoding by a decoder 910d. The representations of the tokens not selected by the spatiotemporal sampling, here denoted Zm, may simply be empty tokens (corresponding, e.g., to the masking of the corresponding inputs 950a, and may, e.g., be correspondingly “zeroed”). Thus, Zv+Zm here indicates the appending of the embeddings of unmasked patches (Zv) with the “masked token”, e.g., initialized with zeros, though they may be learnable parameters (z»).

[0099] Thus, for clarity, if there are 100 patches in an image, and the system masks 80%, then Zv corresponds to the embeddings of the 20 patches while the masked tokens initialized with zero zmcorrespond to the masked 80 patches. Thus, the appending operation may produce 100 patches corresponding to the dimensions of the original input, and be thus suitable, in this example, for the decoder’s creation of the reconstructed input.

[0100] The decoder 91 Od, like encoder 910c, may also be a transformer-based architecture. For example, the decoder 91 Od may be a vision transformer-based model with the decoder depth of four and six attention heads. As will be discussed with respect to FIG. 11 , where the system is considering multiple modalidites, decoder 91 Od may also have two fully connected layers for each modality for the reconstructed output (e.g., to project a latent representation into a two-dimensional pixel space). In some embodiments, the decoder 910d may have an embedding dimension of 384.

[0101] The decoder 91 Od, which may be series of neural network layers, may then generate the reconstructed tensor tokens 950e, denoted V ec, of the input tensor 905a. If not already in a form corresponding to the input tensor 905a, the system may reassemble 920 the output tokens 950e into an appropriate form 905b for comparison with the input 905a. As discussed, the difference 925b between the original input tensor 905a and the reconstructed output tensor 905b may then be used to determine a loss 925a with which to update the encoder 910c and decoder 910d, corresponding, e.g., to the difference 875a and loss 875b. Iterative determination of the loss for a variety of tensor inputs in this manner will eventually produce a system able to reconstruct the input tensors with adesired degree of fidelity, and by implication, prepare a compressed representation 950c able to capture salient semantic content of the data.Example Masking Operations

[0102] Various embodiments may employ a variety of strategies for performing the masking, e.g., as in the spatiotemporal sampling 910b. In some embodiments, when dealing with theater-wide data, the tokens or “tubes” of unmasked data in the tensor may comprise 16 frames after down-sampling to every fourth frame, as this was found effective for the system to infer desired semantic relations within the data in some situations. For example, in some implementations, the original framerate of surgical theater video (internal camera video, surgical theater-wide visual intensity video, surgical theater-wide depth frame video, etc.) is five frames-per-second. The reader will appreciate that the depth frame and visual intensity capture rates may or may not be the same (where they differ, downsampling or interpolation may be applied). However, while masks may be chosen specifically to construct local tubes of such a character, various embodiments may employ other masking strategies to accomplish a variety of objectives. Indeed, in some embodiments, different masking strategies may be employed throughout the training to fine-tune the system’s semantic awareness. For example, the desire to recognize more spatiotemporally local or more spatiotemporally global semantic relations via the masking may be informed by the anticipated downstream application. One may perform ablation studies for various choices of masks to determine which masking strategy, or combination of strategies, produces the best results for the anticipated application and low-data regime. For example, masking to detect the semantic character of robotic surgical system motion may not be the same as that focused upon team member motion, depending upon the nature of the collected data.

[0103] As one example masking strategy, FIG. 10A depicts a schematic tensor breakout illustrating component masking values. In this schematic three-dimensional tensor 1005, which may be, e.g., depth surgical theater-wide depth or visual intensity video (the reader will appreciate other suitable tensors, possibly with different dimensions, for other surgical data modalities), there are only four successive frames 1005a-d. In this example, a different mask is applied to each consecutive frame. Such a granular strategymay not be appropriate when seeking to have the system infer semantic temporal relations, but may be useful when seeking to recognize semantic spatial relations (e.g., if the downstream application will focus upon segmentation of individual video frames). Thus, depending upon the nature of the downstream application, emphasis upon spatial relations over temporal relations may be appropriate. The expected speed of movement, size of objects applicable to a downstream application, temporal correlations, expected number of personnel in the theater, etc. may thus serve as priors informing the masking choice. Thus, the masking strategy of FIG. 10A may be suitable for image-based data (e.g., where individual two-dimensional visual or depth frame images are being input).

[0104] As another example, in FIG. 10B the same mask has been applied to each of the frames 1010a-d of the tensor 1010. Such consecutive application of the same mask, for all or a portion of the temporal duration of the tensor 1010, may precipitate “tubes” of the nature referred to previously. Such temporal tubes may be useful for encouraging the system to recognize temporal relations between spatially local elements of the frames. For example, one would expect an individual pushing a cart to consistently assume a certain posture and for the cart to consistently assume a certain corresponding relatively linear motion. Encouraging the system to recognize the relations between depictions of the pushing team member and depictions of the cart, over time, without complete access to one or the other, via the masking strategy of FIG. 10B, may teach the system to appreciate the association between the two spatial and temporal semantic patterns (while this is example discusses a relation within one modality, one will appreciate that masking can be likewise tuned to recognize relations between modalities, e.g., as when depth and visual image data share relations with kinematics data). Thus, the masking strategy of FIG. 10A may be suitable for video-based data (e.g., three- dimensional visual or depth frame video is received as input).

[0105] Though discussed with respect to the example strategy of FIG. 10B, for clarity, FIG. 10C, provides an example explicitly indicating how a same mask may be applied for one set of consecutive frames within a tensor 1015 and a second mask for a second set of frames within the tensor 1015 (here, a first mask for frames 1015a and 1015b, then a second mask for frames 1015c and 1015d) This may naturally result in tubes of shorter duration than in the strategy of FIG. 10B.

[0106] In some situations, encouraging the system to perform exclusively temporal interpolation may facilitate the system’s representation of long temporal relations. For example, in the strategy of FIG. 10D, entire frames 1020b and 1020c have been masked from tensor 1020, compelling the system to infer the intervening values from the fames 1020a and 1020d. Such a strategy may be suitable for inferring certain types of motion. The reader will appreciate variations, as when only the last several frames, or first few frames are unmasked, and the system must predict the preceding, or succeeding, depicted frames, respectively. Temporally focused masking may thus encourage the system to recognize temporal relations between aggregate phenomenon (within a modality or across modalities), e.g., to characterize motion and momentum of common objects in the theater. In some embodiments, such temporal masking may be applied to training batches following application of one of the approaches in FIGs. 10A-C to preceding training batches, e.g., to encourage the system to recognize global temporal relations between the previously recognized local or spatial semantic patterns (naturally, such strategy variations may or may not be applied to peer modalities during the same training periods, depending upon the nature of the relation sought to be recognized).

[0107] For clarity, the reader will appreciate that semantic knowledge gained via the disclosed tensor maskings may readily facilitate downstream applications and architectures that (in accordance with adaptation 830) may receive and produce inputs and outputs of very different dimensions than the tensors applied during this latent space pretraining. For example, a two-dimensional visual intensity image or depth image segmentation application may receive and output two-dimensional tensors. As successive temporal frames may not be provided to the system of this example, but rather, only individual images, the system may avail itself of less than all the semantic knowledge gained during pretraining. However, as various of the masks, such as those of FIGs. 10A-D, may still emphasize the identification of local spatial relations, the latent space pre-training may still greatly improve the subsequent supervised training of the segmenting architecture.

[0108] As a specific masking example for theater-wide surgical theater data, FIG. 10E illustrates a pair of intensity 1025a and depth 1030a tensors and their corresponding masked representations 1025c and 1030c, respectively, as may be implemented in someembodiments. In some embodiments, the masking operations 1025b and 1030b may apply the same mask to each of the respective tensors or masks scaled in agreement with the different magnitudes of the tensors’ respective dimensions. As will be discussed in the next section, various embodiments have been found to achieve desirable results by employing architectures accommodating the consideration of different modalities of surgical theater data, such as the surgical theater-wide data of the intensity 1025a and depth 1030a tensors. For clarity, one will appreciate that masking strategies applied when only one modality is considered may likewise be applied for that modality when considered in tandem with other modalities (in contrast, e.g., to complementary masking operations between modalities as discussed elsewhere herein).

[0109] While, to facilitate the reader’s comprehension, three-dimensional tensors depicting visual image intensity video frames have been primarily discussed until now, various embodiments may also employ masking strategies upon other types of data and upon tensors of other than three dimensions, as will be discussed in greater detail herein. Indeed, different details of different modalities may be masked with separate and uncorrelated masks (so that semantic correlations may be stochastically inferred) or with masks corresponding by design (so as to emphasize specific semantic correlations).

[0110] Thus, the masking strategy for visual image video or for depth frame video may be chosen based upon temporal redundancies and temporal correlations found generally in the video modality. As mentioned, the “tube” masking of FIGs. 10B, or, more granularly, of FIG. 10C, may thus be suitable for these modalities. In some embodiments, entropy in the data may also be used to inform the masking strategy choice, in both these video modalities and other modalities (e.g., as will be discussed in FIG. 12). While the chosen masking strategy may not change over the course of training in some embodiments, in some embodiments “intelligent” masking may instead vary with the training data (e.g., as when motion in visual or depth video of a given training batch, as assessed via optical flow or temporal variation, informs the length of the tubes in FIG. 10C). For non-video modalities, random masking (as in FIG. 10A, albeit, as the reader will appreciate, the mask dimensions adjusted in accordance with the contemplated modality) may be suitable initially, and then varied intelligently if results are not as desired. In still further embodiments, where, e.g., the available depth data is more sparse than thevisual intensity data, and visual image video corresponding to the missing depth data is available, the sparse regions of the depth data may be masked while the corresponding region of the visual intensity modality may be deliberately unmasked (or vice versa). In these complementary situations, the reconstruction loss for the sparse modality may be likewise adjusted (e.g.., compared with an interpolated value for the missing region, scaled down, etc.). Thus, the masking strategy used for a modality having one set of dimensions may be used directly, or in a complementary fashion, with data of a different modality, or data availability, if the two modalities correspond temporally and spatially (as is the case with the visual image and depth frame video modalities of FIG. 10E).Example Intensity and Depth Multimodal Unsupervised Training

[0111] Many disclosed embodiments will consider various combinations of multimodal surgical theater data so as to capture more robust semantic representations, which may more greatly advance downstream application-specific training. Certain cross- modal loss determinations may also be considered to better ensure recognition of these relations. For example, FIG. 11 is a schematic processing block diagram illustrating data flow during the loss calculation for multimodal depth and intensity unsupervised (or semisupervised) latent space training of a machine learning system 1170, as may be implemented in some embodiments. Again, though a loss calculation will be discussed here when considering a single instance of the incoming data to facilitate the reader’s comprehension, the reader will appreciate that, in practice, the instances may be consolidated into blocks and the blocks considered in epochs of training. The consolidated loss of one or more batches may be then used to update the system, rather than at the instance level as depicted here.

[0112] Here, an instance comprising a pair of temporally correlated visual intensity 1105a and depth 1110a theater-wide data tensors is to be considered during latent space training (e.g., pretraining) of the system 1170. Respective masking logic 1105c and 1110c (e.g., as described herein with respect to FIGs. 10A-E) may be applied to each of the instance tensors to infer masked counterparts 1105d and 1110d, respectively. The masked counterparts 1105d and 1110d may then be formatted, e.g., tokenized (which, as previously discussed, may be part of the masking operation), to form inputs 1110e and1110f, respectively, for receipt by an encoder 1115, in this example a transformer encoder, so that they might be jointly encoded. While, in some embodiments, the encoder 1115 and decoders 1145a, 1145b may be the same or similar architectures as their counterparts in FIG. 9, in some embodiments, multimodal topologies may use, e.g.: a vision-transformer (ViT) base architecture for the encoder and decoders; fully convolutional neural networks, such as a ConvNext layer, in lieu of transformers, for each of the encoder and decoders; etc. In some implementations, the vision-transformer base may have 12 blocks (depth) and each block having 12 multi-head attention layers. The decoder(s) may have fewer layers and therefore be “shallower” than the encoder. For example, each decoder may have four blocks and each block may have six multi-head attention layers, with 384 dimensions. The last layer of each decoder may be a single fully connected layer, referred to as a prediction head. For example, the output dimension of the prediction head for the intensity decoder may be 1536 and for the depth decoder the output dimension may be 512.

[0113] As intensity and depth frames each may take a two-dimensional form (e.g., with each intensity value assuming a single grayscale, or a composite color, representation and each depth frame value associated with a depth determination), some embodiments may provide the raw values to the encoder, while others may scale the values to a common range (e.g., mapping each of the grayscale and depth values to a range between 0-1 , 0 being black and 1 white, 0 being the closest recognizable depth value and 1 the furthers possible recognizable depth value, etc.). Thus, some embodiments first transform the intensity and depth video into tokens using a cube embedding layer, such as a Conv3D layer. In some such implementations, each cube is of size 2 x 16 x 16 and corresponds to one token embedding. Masking may then be applied, as discussed, upon these tokens. As mentioned, the reader will appreciate tensor variations where, e.g., color pixel values are considered. Where the data is video, each modality may have its own patch embedding layer (e.g., a Conv3d layer). The system may then, e.g., concatenate the patch embeddings and pass them to the encoder.

[0114] A portion of the encoder 1115’s output 1120a may be designated to represent the compressed latent space visual intensity representation. For example, a portion of the encoder output may be specified for one modality (e.g., visual intensity video) andother portions for the other modalities (e.g., the remainder for depth frame video) and passed to respective head layers for reconstruction. Thus, in some embodiments, the latent space size may be the same for each modality (e.g., in some implementations, the latent space is 768 dimensions for both visual image and depth frame modalities). Another portion of the output 1120b is here designated to represent the compressed latent space depth modality representation. Analogous to the previously described embodiments, each of the latent space representations 1120a and 1120b may be passed to their respective decoders 1145a and 1145b, to produce 1145c, 1145d reconstructed visual intensity 1105b and reconstructed depth 1110b representations, respectively. Particularly, the latent representations from the encoder are here passed to the modalityspecific decoders to reconstruct the missing patches for both the intensity and depth. For clarity, as discussed elsewhere herein, not every embodiment may have separately designated decoder structures, as in this example, instead having, e.g., one single decoder with one prediction head for each modality and the various modality latent representations concatenated into a single tensor and passed to the single decoder structure. As mentioned above, the prediction heads may be fully connected. In some embodiments, the single decoder architecture may be the same as one of the decoders in the example of FIG. 11 (e.g., one of the transformer architectures discussed herein) save the adjustment to the prediction heads. These embodiments, which consolidate the decoders into one structure, may reduce the total model size and may also decrease the training time.

[0115] In the depicted embodiments, the total loss 1140e used to update the encoder 1115 and decoders 1145a, 1145b may include a variety of components. Specifically, as previously described, the difference 1130b between the original intensity tensor input 1105a and the reconstructed intensity tensor output 1105b may be used to determine a visual intensity modality reconstruction loss 1140c. Similarly, the difference 1130c between the original depth tensor input 1110a and the reconstructed depth tensor output 1110b may be used to determine a depth modality reconstruction loss 1140d. However, in addition to these reconstruction losses 1140c, 1140d, the total loss 1140e may also include one or both of, or may alternatively include one or both of, a matching loss 1140b and a contrastive loss 1140a. Again, while particular data instances arediscussed here to facilitate the reader’s understanding, the reader will appreciate that these operations (e.g., the loss determinations) may be performed at the batch level. The contrastive, reconstructive, and matching losses may be especially useful in enhancing cross-modal information (as occurs here in FIG. 11 , but even further in the multiple modalities of FIG. 12).

[0116] In more detail, as regards the visual intensity reconstruction loss 1140c, given a video clip of intensity modality V of size T x 3 x H x W where T is the number of frames in the clip, H and W are the height and the width of the frame, respectively, and 3 corresponds to the number of channels in the frame (e.g., where the frame depicts red, green, blue color pixel values), the loss 1140c may be determined, e.g., as £intin accordance with EQN. 1where p is the token index, and co is the set of masked tokens. V corresponds to the reconstructed representation prediction of the model.

[0117] Similarly, as regards the depth reconstruction loss 1140d, given a video clip of depth modality D of size T x 1 x H x W where T is the number of frames in the clip, H and W are height and width of the frame and 1 corresponds to the number of channels in the frame (here, the single distance depth value in each entry), the loss 1140d may be determined, e.g., as £depthin accordance with EQN. 2where m is the token index, A is the set of masked tokens, and D corresponds to the reconstructed representation prediction of the model.

[0118] If one applied only these reconstruction losses, then the system may discern only the semantics of each modality individually, rather than learn any cross-modal relations. In contrast, use of the contrastive and matching losses described herein, in lieu or in addition to, the reconstructive losses, may serve to “pull” modalities closer to one another when the data from the modalities corresponds semantically (e.g., if they belongto the same video clip), e.g., if they are treated as a pair in the embedding space (e.g., their data values correspond). For example, intensity-depth contrastive loss learning (or between additional or alternative modalities as discussed herein) may align the visual scene and the corresponding depth by pulling temporally paired intensity-depth data “closer” while “repelling” temporally unpaired intensity and depth data apart.

[0119] Regarding the contrastive loss 1140a, each of the compressed latent space representations 1120a and 1120b may be passed through a respective averaging pool network 1125a or 1125b, respectively, to produce respective cumulative outputs 1125c and 1125d (e.g., respective mean values). That is, each average pool layer may receive the encoder embeddings and output the mean of the embedding for each modality (in lieu of neural network layer, the reader will appreciate, e.g., that logic may instead be used to infer the mean of the features). As the average pool layer is used to determine the mean of the features and the means are then used to infer the contrastive loss, one could instead use a max pool layer or other neural network (consisting, e.g., of two or three fully connected layers mapping to a projection layer) to map the latent space features to a different dimensionality, so as to achieve a same or similar functional result.

[0120] Thus, given a batch of masked intensity-depth training data, various embodiments compute the corresponding features using the encoder followed by a global average pooling layer, and may then apply contrastive loss 1140a upon these features, determined, e.g., as £cin accordance with EQN. 3Where N is the number of instances in the batch (here, visual intensity and depth tensor pairs) sij =(e.g., represented schematically via difference 1130a) and T is the temperature. || || and ||x4|| correspond to the encoder features for intensity and depth, respectively, for a training instance i.

[0121] Regarding the matching loss 1140b, a fully connected multilayer perceptron (MLP) 1135a may receive each of the latent space representations 1120a and 1120b respectively to produce the matching loss 1140b. The MLP 1135a may be a fullyconnected single layer with an output dimension of two, followed by a SoftMax as the classifier upon the representation, so as to predict a two-class probability. That is, the matching loss may provide a binary classification loss, indicating “positive” and “negative” modality pairs. Like the encoder and decoder, the MLP may itself be updated by the total loss 1140b (e.g., as indicated by the arrow 1195). In some embodiments, the latent space representations for each modality (e.g., the latent space representation 1120a and the latent space representation 1120b) may be concatenated before passing them into MLP layer 1135a. As mentioned, in some implementations, the latent space may be 768 dimensions, and the MLP 1135a may accordingly receive a single fully connected layer with a dimension of 768 and output, e.g., a dimension of two.

[0122] Utilization of the matching loss, as described here, may better facilitate cross- modal training, as the matching loss may predict whether a pair of modalities is “matched” (in the example of EQN. 4, reflected by a positive value) or “not matched” (in the example of EQN. 4, reflected by a negative value). Specifically, these example embodiments reuse the features from the encoder, passing them to a linear layer followed by a SoftMax classifier to solve a 2-class classification problem to find the matching loss, e.g., as £min accordance with EQN. 4where y is sign function which outputs 1 if t is 1 , and otherwise outputs 0, qn(z ,zP) denotes the probability score of t from the SoftMax, z .z are the latent visual intensity and depth representations for the ithpair, respectively, and M is the total number of intensity-depth instance pairs in the batch. The matching loss may be particularly useful when two or more data modalities are considered, as will be discussed in greater detail herein with respect of FIG. 12.

[0123] Thus, the final total loss 1140a in these examples, may be as shown in EQN.5where a and [3 are hyper-parameters, which may be tuned during pre-training. In some implementations, the system may select a and |3 from the set of {0.1 , 0.2, 0.3, 0.4, 0.5} (e.g., then selecting the highest performing results at the end of training). Similarly, in some implementations, the system may select the masking ratio (the numerator being the size of the masked portion of the tensor, and the denominator the entire size of the tensor) from the set of {0.75, 0.85, 0.90, 0.95}. Optimal choices of the hyperparameters may be found by evaluating the pre-trained model under, e.g., a 5% labeled data setting. For further clarity, in some implementations the base learning rate is selected from {1 ,5e-4, 1.5e-5}.

[0124] The disclosed latent space training strategy is very data efficient, is not limited to only to visual and depth modalities, and may also be used in online learning settings, e.g. , as part of a progressive, iterative online approach, adjusting a deep learning system as more data becomes available. For example, the application neural network may be modified back to an encoder-decoder form (the same or different than was previously used), the disclosed unsupervised training performed with the newly available unlabeled data, and supervised training specific to the application then performed again with whatever new labeled data is available. Such online training, e.g., in deployed hospital settings, may be useful as more cases are performed, and consequently more data becomes available, facilitating increasingly improved “semantic background” understanding local to that unique deployment, while the supervised training increasingly improves the “application-specific foreground” adaptation, which may be likewise unique to that deployment environment.

[0125] As yet another variation on the above-described embodiments, in some embodiments multi-modal (such as the depth and intensity embodiments of FIG. 11 , as well as those involving additional or alternative data modalities as will be discussed herein) may also perform the pre-training (e.g., unsupervised 870 training) in two or more stages. For example, in a first stage, one may pre-train an encoder (e.g., encoder 1115, such as a vision transformer base) with the contrastive and matching losses discussed herein, but without applying any decoder or any masking (that is, the total loss will be derived from only the latent space representations). Following this first stage, in a second stage, one may then initialize the encoder with the weights determined from the first stage(if not already present), introduce the modality-specific decoders (or, as discussed herein, a single consolidated decoder with different prediction heads for the modalities), remove the MLP and average pool layers, then perform additional unsupervised training, but this time conversely using the reconstruction losses only for the total loss. Masking may be employed in this second stage. In this manner, reconstructive and cross-modal learning may proceed successively, rather than simultaneously, providing specific learning pressures upon their semantic content. While some embodiments may apply the first and second stage only once, the reader will appreciate that in some embodiments the first and second stages may be applied iteratively, e.g., where the trainer wants to monitor the training’s progress upon the semantic content of a particular dataset, possibly performing training of multiple copies of the training system in parallel, with variations in the number and durations of the first and second stages (which, the reader will appreciate, may thus themselves serve as training hyperparameters).Example Disparate Multimodal Unsupervised Latent Space Training

[0126] Various embodiments may further extend the embodiment of FIG. 11 using one or more of the reconstructive, matching, and contrastive losses in training the system, but with alternative or additional, and possibly quite disparate, surgical theater data modalities. Indeed, the losses and algorithm of and training strategy of FIG. 11 are not limited to the visual intensity and depth modalities of FIG. 11 . Modality versatility may be useful in a variety of surgical circumstances, e.g., when considering both robotic and nonrobotics surgical theaters (e.g., as data availability may vary when successively training with data from the different types of theaters).

[0127] For example, FIG. 12 is a schematic processing block diagram illustrating data flow during a multimodal round of training, as may be implemented in some embodiments. That is, the embodiments of FIG. 12 further configure the encoder 1245 and prepare corresponding decoders for receipt of additional types of peer instance surgical theater data modalities. Though not depicted in FIG. 12 for economy, the reader will appreciate that each of the previously discussed reconstructive, matching, and contrastive losses, previously discussed, as well as the various possible architectures, may be used here again, mutatis mutandis, with three or more data modalities. Thus,training may align the modalities by pulling semantically related pairs of modalities closer together within the latent feature space, thereby identifying cross-modal correspondence. For example, other modalities may, e.g., be aligned with the visual intensity modality by creating pairs, e.g., intensity+depth, intensity+text, intensity+audio, etc., using the same contrastive loss approaches discussed herein. While, as discussed herein, many embodiments may employ a single encoder architecture in their topology, in some embodiments, modality-specific or modality-agnostic encoders may likewise by used. However, use of only one encoder shared among multiple the modalities has been found sufficient for various applications during testing.

[0128] Here, as in FIG. 11 , a visual image intensity tensor 1205a and a temporally corresponding depth frame tensor 1230a may be each considered in this training instance, a masked counterpart 1205c, 1230c generated for each using respective masking logic 1205b, 1230b, latent space representations 1205d, 1230d determined via application to an encoder 1245, here, a transformer encoder, and then respective decoders 1205e, 1230e applied to produce reconstructed tensors 1205f, 1230f whose respective losses may be used to inform the total loss, as was the case in the embodiments of FIG. 11. Contrastive and matching losses, as were described in FIG. 11 , may likewise be determined between the intensity modality and each of the other modalities.

[0129] Similarly, a visual intensity video tensor 1210a (or mutatis mutandis, depth values acquired within the patient) may present video data captured from within the patient (e.g., as seen in whole or in part upon displays 125, 150 or within surgeon console 155). Appropriate corresponding masking logic 1210b may produce a masked representation 1210c, which may likewise be received by the encoder 1245, and then reproduced as reconstructed visual image 121 Of by applying a decoder 1210e specific to that purpose, to the latent representation 1210d. Thus, the encoder 1245 may learn to produce a joint representation derived from each of the multiple data modalities.

[0130] As another example, kinematics data likewise temporally corresponding to the other data modalities (e.g., captured in parallel with the theater-wide data 1205a, 1230a, with the internal data 1210a, etc.) may be presented in an appropriate tensor1215a. For example, where the kinematics data was captured as a plurality of waveforms over time (e.g., sampled poses for an end effector, rotation values for degrees of freedom in a manipulator, etc.) the tensor 1215a may, e.g., be two-dimensional, a first dimension associated with the number of waveforms, and a second dimension associated with the value (or values, with additional suitable dimensions) at each temporal point of each respective waveform value. The reader will appreciate that interpolation, smoothing, down-sampling, etc. may be applied to the kinematics data so as to correspond with the substantially contemporaneous values of other of the peer data modality tensors. As with the other modalities, the tensor 1215a may be likewise masked via logic 1215b to form masked representation 1215c, and the reconstructed counterpart 1215f produced by applying a kinematics specific decoder 1215e to the latent space representation 1215d.

[0131] Similar to the kinematics data, auditory data, such as auditory waveforms, may also be presented in a tensor 1220a, likewise masked via logic 1220b to form masked representation 1220c, and the reconstructed counterpart 1220f produced by applying an auditory specific decoder 1220e to the latent space representation 1220d. While contemporaneous audio in the form of a waveform may be presented in the tensor 1220a, and likewise down-sampled, smoothed, interpolated, etc. to match the other data modalities, in some embodiments, the audio and kinematics data may be presented in an alternative format, albeit, mapped to temporal points corresponding to the other of the multimodal peer data tensors. For example, waveforms may be converted to their frequency counterparts and a sliding window of the dominant frequencies presented instead within the waveform. Similarly, discrete counterparts presented over time, may also be determined from the waveforms and inserted into the tensor, as when natural language processing tools are used to infer spoken words, spoken phrases, non-verbal codebook values, such as code-excited linear prediction (CELP) codes, code-division multiple access (CDMA) values, etc. Text data, which may be part of the systems data, may also be included. For example, where theater configurations, surgical procedure types, and team member compositions are known from a healthcare scheduling roster, a dictionary of these values may likewise be included as a portion of a tensor (e.g., an additional tensor directed to surgical theater metadata). Similarly, medical codes, suchas International Classification of Diseases, Tenth Revision (ICD 10) codes may appear in such a metadata tensor, or in a “text” tensor, and may be likewise temporally masked.

[0132] By sustaining such discrete values over the temporal intervals of the other multimodal data peer tensors, such data may be rendered suitable for parallel consideration by the encoder system. Indeed, system event data (e.g., tool activations, operator head removals from a surgical console, states of a robotic surgical system, states of surgeon console, etc.), despite often assuming discrete values over time, may, if not presented in their original form, be likewise organized into a tensor 1225a by temporally extending their values. Similar to waveforms, the tensor 1225a may be, e.g., two-dimensional, with one dimension associated with each type of system event data, and the other dimension associated with the value of the event data. Corresponding masking logic 1225b may prepare a masked counterpart 1225c, from which the encoder 1245 may infer a latent space representation 1225d, suitable for decoder 1225e to form the reconstructed tensor 1225f.

[0133] The reader will appreciate that even though latent space training may avail itself of the disparate types of data during an operation, as in FIG. 3, such pretraining may still be suitable for applications directed to nonoperative periods, as in FIG. 4, and vice versa, so long as the relevant semantic associations to the considered application have been generally captured. For example, kinematics data may initially assist the system during pretraining to recognize semantic associations between movement of a cart and movement in the theater-wide surgical data. Once recognized, application specific training may then reinforce the association, but now as particularly considered by only the theater-wide surgical data. Conversely, a system able to recognize such associations when trained from theater-wide data alone, will be better positioned to anticipate the further association with kinematics data made available from the additional data modalities of FIG. 3. The reader will appreciate that less than all of the depicted modalities in FIG. 12, or additional, or alternative modalities may be used (as indicated by ellipses 1235a-f; e.g., text, as discussed, may be included, such as phrases “the operating room staffs are prepping or cleaning the room”, medical codes and descriptions, recognized spoken words, etc.). Similarly, again, while reconstruction of only oneinstance is shown in FIG. 12, the instances may be considered in batches so as to infer consolidated loss values.

[0134] As in FIG. 11 , embodiments like that of FIG. 12 may determine the loss based solely upon the difference in original and reconstructed tensors, solely upon the contrastive loss (e.g., summed between each modality pair), solely upon the matching loss (e.g. .summed between each modality pair), or upon combinations of these types of loss. Indeed, in some situations and for certain downstream applications, the presence of multiple modalities may render any one of the single types of component losses sufficient for training. Combining the types of losses, however, as in EQN. 5 (adjusted per the additional modalities), may facilitate more nuanced semantic determinations better facilitating the training for various downstream applications.

[0135] The disclosed distinct encoder and decoder architectures, or merged autoencoder embodiments, may thus leverage multiple modalities (visual intensity, depth values, text, kinematics data, etc.) to extract discriminative representations for semantic understanding. The resulting latent space representations may lessen the gap with subsequent supervised training, thus reducing the need to acquire vast quantities of labeled data. Unlike approaches limited to a single modality, various of the disclosed multimodal approaches are more amenable to holistic considerations of the theater state, so as to better appreciate semantic correlations appearing therein, within and between modalities. Thus, by employing multiple data modalities, some embodiments serve to pre-train the model for many different downstream tasks, such as scene understanding, audio classification, dense depth estimation, visual question-answering, cross-modal (e.g., video-text, audio-text) retrieval, etc., even in some zero-shot settings.Example Unsupervised Training Loss Determination

[0136] FIG. 13 is a flow diagram illustrating various operations in an example process 1300 for performing a round of loss determination for one or more data modalities during a round of training, as may be implemented in some embodiments. Specifically, at block 1305a the system may receive or select the data designated for this round of latent space training. As previously mentioned, the data may comprise one or more modalities from surgical theater data without theater-wide data, theater-wide dataexclusively, or surgical theater data which includes theater-wide data (naturally, one will appreciate that the disclosed approaches may be applied to robotics surgical theaters or non-robotic surgical theaters and, indeed, latent space training on data from one type of theater may be used in some situations for applications in the other).

[0137] At blocks 1305b and 1305c, the system may iterate over the various contemplated modalities, determining a corresponding masking strategy at block 1305d, and generating the masked representation of the instance at block 1305e (again, as discussed, as may be performed during tokenization). For example, as discussed with respect to FIG. 12, different masking logics (e.g., logics 1205b, 1210b, 1215b, 1220b, 1225b, and 1230b) may be applied to each of the respective modalities of the data received at block 1305a, and the masking strategy applied to a modality may change over the course of training (e.g., in a fashion complementary to changes in other modalities’ maskings).

[0138] At block 1310a, the system may construct an input tensor (e.g., the corresponding token representations) for receipt at the encoder (e.g., encoder 1115, encoder 1245, etc.) from the masked representations generated at block 1305e, and by applying the tensor to the encoder, acquire the reduced representation(s) at block 1310b. Each of the encoders 1115 and 1245 may, e.g., receive the input tensor via a 1-layer convolution.

[0139] As mentioned, while some embodiments may determine only some of the component losses, here, the system may determine each of the contrastive and matching losses as well as the reconstructive losses. In accordance with EQN. 3 and EQN. 4, the respective component losses may be determined at the batch level, using these latent space results for the specific instances of the modalities. Thus, at block 1310c the system may record the latent-space derived results for the considered multi-modal instances that will be used for the contrastive loss, and at block 131 Od the system may record the latent- space derived results for the considered multi-modal instances that will be used for the matching loss (again, for clarity, the reader will appreciate that the operations need not necessarily proceed in the depicted temporal order, nor be broken into the depicted organizational blocks).

[0140] At block 1310e, the system may generate the reconstructed representations for each of the modalities, e.g., as described in FIGs. 11 and 12. At blocks 1315a and 1315b, the system may again iterate over the contemplated modalities, determining the corresponding reconstruction loss at block 1315c. In the example process 1300, these results may be similarly recorded for use in determining the total loss at the batch level.

[0141] When all the instances (e.g., collections of temporally corresponding data of different data modalities) in the current batch have been considered at block 1320 (otherwise, the process may proceed again to the next instance), the total loss for the batch may be determined and applied to the various neural networks (e.g., as discussed above). Specifically, when all the batch instances have been considered at block 1320, the system may determine the total loss, e.g., in accordance with EQN. 5. The system may, e.g., sum the reconstructed losses determined at block 1315c for each modality and for each instance, to determine the collective reconstructive loss for the batch at block 1325a. Similarly, at block 1325b, the system may determine the contrastive loss for the batch using the constituent results determined at block 1310c for each instance. At block 1325c, the system may determine the matching loss for the batch using the constituent results determined at block 131 Od for each instance. Finally, e.g., per EQN. 5, the system may determine the total loss for the batch at block 1325d and then use the loss to update the various machine learning systems at block 1325e as described herein (e.g., encoder 1115; encoder 1245; each of the decoders 1205e, 1210e, 1215e, 1220e, 1225e, and 1230e; decoders 1145a and 1145b; MLP 1135a; etc )Example Downstream Application Training Process

[0142] FIG. 14 is a flow diagram illustrating various operations in an example process 1400 for preparing an application-specific machine learning system, as may be performed in some embodiments. At block 1405a, a training computer system may receive the partially labeled surgical theater data. At block 1405b, the system may determine which data to use for supervised or unsupervised training (if not specified by a human designer), e.g., distinguishing between labeled and unlabeled data. For example, the system may identify the data manually labeled by a human annotator, as well as make inferences from the surgical equipment acquired data 375a regarding the surgical stateand other contextual data or metadata to infer labels. Depending upon the heterogeneous character of the data, in some implementations, labels may be removed from the labeled data (or ignored) to provide more unsupervised training data, e.g., to create more balanced supervised and unsupervised training datasets.

[0143] The data to be used for unsupervised (or, mutatis mutandis, semisupervised) training may then be divided into training blocks at block 1405c. The system may perform the image processing for these blocks, or batches, at block 1405d. For example, as described herein, RGB and grayscale visual image pixel values may be mapped to a common range with depth distance values, corrupted data may be removed (and supervised and unsupervised training datasets adjusted accordingly), down sampling or interpolative up-sampling may be performed, etc. The system or human designer may choose suitable hyperparameters (e.g., weights of the total loss, masking strategies, epochs of training, backpropagation settings, etc.) for the unsupervised training (e.g., based upon empirically derived look-up tables, interpolations from historical choices based upon the nature of the provided surgical theater dataset, objectives or requirements of the downstream application, parallel running of distinct training systems, etc.) at block 1405e.

[0144] The system may then perform the rounds of initial latent space training at block 1410a to produce loss scores with which to update the encoder(s), decoder(s), and any other relevant machine learning systems. If all the rounds of training are complete, however, or if the performance of the encoder(s) I decoder(s) are satisfactory at block 1410b, the system may proceed to the adaptation process at block 1420a (e.g., in accordance with arrow 840a).

[0145] Until the initial latent space training is complete, however, during each training round the system may iterate over the data blocks (also referred to as batches) of, e.g., unsupervised data at blocks 1410c and 1410d, and in turn, each instance of the data block at blocks 141 Oe and 1415a. At block 1415b, appropriate masked representations for the data modalities may be determined in accordance with the currently applicable masking strategy for the respective modality. Using the masked representations, at block 1415c the system may determine the various constituentcomponents for the losses (e.g., the latent space average pool values for each modality, the MLP results for each modality pair in the instances, the reconstructive losses for each modality, etc.) as derived from the currently considered instance.

[0146] Once all the instances for the block have been considered, the total loss (e.g., total loss 1140e) for the block (e.g., from the cumulative reconstruction losses 1140c, 1140d, matching loss 1140b, and contrastive loss 1140a) may then be determined at block 1415d and used to update the various machine learning systems at block 1415e (e.g., the encoder(s), decoder(s), MLP, etc.).

[0147] Once the initial latent space training completes, portions of either or both of the encoder(s) and decoder(s) may be used at block 1420a to prepare an architecture suitable for the application, thus corresponding, e.g., to adaptation 830. For example, a new output layer may be appended to the combined encoder / decoder system (e.g., as part of the modifications 830b), to only the encoder, etc. This may produce a structure (e.g., the architecture of classifier 825b) suitable for use in downstream application training, e.g., supervised training (e.g., as shown by arrow 840b).

[0148] Accordingly, at block 1420b, the system may divide the application data (e.g., labeled data) into batches for application-specific training and select appropriate hyperparameters for the application-specific training at block 1420c (again, the reader will appreciate that application training may not proceed in exactly the manner as depicted here to facilitate understanding). Application-specific training may then be performed (e.g., recognizing the states of FIGs. 6A or 6B from labeled theater-wide training data) until all the training rounds are complete at block 1425a or until the machine learning system’s performance is acceptable at block 1425b, after which the application-specific machine learning system may be published for use at block 1435a. For each round of application-specific training, the system may consider the blocks of application data at blocks 1425c and 1425d, and for each block, consider the respective data instances at blocks 1425e and 1430a to determine the current loss for the machine learning system and adjust the application-specific machine learning system at block 1430b based upon those results, e.g., as discussed with respect to supervised training 825 (naturally, one will appreciate that different application-specific training methods may not proceed exactlyas shown here, but may have different loss determination methods, some updating by instance, some updating by batch, some updating only over the entire epoch, etc.). Thus, again, one will appreciate that the resultant machine learning system need not output probabilities for discrete classes, but may, e.g., be any suitable downstream application, such as a generative application, or an object recognition and visual image segmentation application, whose output may not be a single set of predictions, but, e.g., a tensor of probabilities, synthetic data values, projected data values, etc.Results for an Example Prototype Implementation of an Embodiment

[0149] FIG. 15 is a table depicting comparative results generated in connection with an example prototype implementation of an embodiment. Particularly, an example multimodal data implementation (using visual and depth data, as in FIG. 11 ) of a disclosed embodiment was compared with recent state of the art techniques. “MICCAI 2022-SwAV (Intensity + Depth)” here refers to the approach presented in Jamal, Muhammad Abdullah, and Omid Mohareri. "Multi-Modal Unsupervised Pre-Training for Surgical Operating Room Workflow Analysis." arXiv preprint arXiv: 2207.07894 (2022) (using both visual intensity image and depth frame modalities). “SurgMAE” here refers to the approach presented in Jamal, Muhammad Abdullah, and Omid Mohareri. "SurgMAE: Masked Autoencoders for Long Surgical Video Analysis." arXiv preprint arXiv:2305.11451 (2023). Results are shown in the table of FIG. 15, indicating the superior performance of the example embodiment implementation, as evidenced by the mean average precision values in each table cell of the three right-most columns. Specifically, as shown, the respective systems were evaluated upon low-data regime settings (e.g., 5%, 10%, 20% labeled data accessible to the downstream task). For all the experiments, the pre-trained model was fine-tuned for 75 epochs using a base learning rate of 6e-4. A cosine learning scheduler with end learning rate of 1e-5 was used, and the fine-tuning warmed up for 5 epochs with a learning rate of 1e-8. For models configured to use temporal data, a learning rate of 1e-3 for a total of 15 epochs was used.Computer System

[0150] FIG. 16 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments. The computing system 1600 may includean interconnect 1605, connecting several components, such as, e.g., one or more processors 1610, one or more memory components 1615, one or more input / output systems 1620, one or more storage systems 1625, one or more network adaptors 1630, etc. The interconnect 1605 may be, e.g., one or more bridges, traces, busses (e.g., an ISA, SCSI, PCI, I2C, Firewire bus, etc.), wires, adapters, or controllers.

[0151] The one or more processors 1610 may include, e.g., an Intel™ processor chip, a math coprocessor, a graphics processor, etc. The one or more memory components 1615 may include, e.g., a volatile memory (RAM, SRAM, DRAM, etc.), a non-volatile memory (EPROM, ROM, Flash memory, etc.), or similar devices. The one or more input / output devices 1620 may include, e.g., display devices, keyboards, pointing devices, touchscreen devices, etc. The one or more storage devices 1625 may include, e.g., cloud-based storages, removable Universal Serial Bus (USB) storage, disk drives, etc. In some systems memory components 1615 and storage devices 1625 may be the same components. Network adapters 1630 may include, e.g., wired network interfaces, wireless interfaces, Bluetooth™ adapters, line-of-sight interfaces, etc.

[0152] One will recognize that only some of the components, alternative components, or additional components than those depicted in FIG. 16 may be present in some embodiments. Similarly, the components may be combined or serve dual-purposes in some systems. The components may be implemented using special-purpose hardwired circuitry such as, for example, one or more ASICs, PLDs, FPGAs, etc. Thus, some embodiments may be implemented in, for example, programmable circuitry (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in special-purpose hardwired (non-programmable) circuitry, or in a combination of such forms.

[0153] In some embodiments, data structures and message structures may be stored or transmitted via a data transmission medium, e.g., a signal on a communications link, via the network adapters 1630. Transmission may occur across a variety of mediums, e.g., the Internet, a local area network, a wide area network, or a point-to-point dial-up connection, etc. Thus, “computer readable media” can include computer-readablestorage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0154] The one or more memory components 1615 and one or more storage devices 1625 may be computer-readable storage media. In some embodiments, the one or more memory components 1615 or one or more storage devices 1625 may store instructions, which may perform or cause to be performed various of the operations discussed herein. In some embodiments, the instructions stored in memory 1615 can be implemented as software and / or firmware. These instructions may be used to perform operations on the one or more processors 1610 to carry out processes described herein. In some embodiments, such instructions may be provided to the one or more processors 1610 by downloading the instructions from another system, e.g., via network adapter 1630.

[0155] For clarity, one will appreciate that while a computer system may be a single machine, residing at a single location, having one or more of the components of FIG. 16, this need not be the case. For example, distributed network computer systems may comprise multiple individual processing workstations, each workstation having some, or all, of the components depicted in FIG. 16. Processing and various operations described herein may accordingly be spread across the one or more workstations of such a computer system. For example, one will appreciate that a process amenable to being run in a single thread upon a single workstation may instead be separated into an arbitrary number of sub-threads across one or more workstations, such sub-threads then run in serial or in parallel to achieve a same, or substantially similar, result as the process run within the single thread. Similarly, one will appreciate that while a non-transitory computer readable medium may stand alone (e.g., in a single USB storage device), or reside within a single workstation (e.g., in the workstation’s random access memory or disk storage), such a medium need not reside at a single geographic location, but may include, e.g., multiple memory storage units residing across geographically separated workstations of a computer system in network communication with one another or across geographically separated storage devices.Remarks

[0156] The drawings and description herein are illustrative. Consequently, neither the description nor the drawings should be construed so as to limit the disclosure. For example, titles or subtitles have been provided simply for the reader’s convenience and to facilitate understanding. Thus, the titles or subtitles should not be construed so as to limit the scope of the disclosure, e.g., by grouping features which were presented in a particular order or together simply to facilitate understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, this document, including any definitions provided herein, will control. A recital of one or more synonyms herein does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any term discussed herein is illustrative only and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term.

[0157] Similarly, despite the particular presentation in the figures herein, one skilled in the art will appreciate that actual data structures used to store information may differ from what is shown. For example, the data structures may be organized in a different manner, may contain more or less information than shown, may be compressed and / or encrypted, etc. The drawings and disclosure may omit common or well-known details in order to avoid confusion. Similarly, the figures may depict a particular series of operations to facilitate understanding, which are simply exemplary of a wider class of such collection of operations. Accordingly, one will readily recognize that additional, alternative, or fewer operations may often be used to achieve the same purpose or effect depicted in some of the flow diagrams. For example, data may be encrypted, though not presented as such in the figures, items may be considered in different looping patterns (“for” loop, “while” loop, etc.), or sorted in a different manner, to achieve the same or similar effect, etc.

[0158] Reference herein to "an embodiment" or "one embodiment" means that at least one embodiment of the disclosure includes a particular feature, structure, or characteristic described in connection with the embodiment. Thus, the phrase "in one embodiment" in various places herein is not necessarily referring to the same embodiment in each of those various places. Separate or alternative embodiments maynot be mutually exclusive of other embodiments. One will recognize that various modifications may be made without deviating from the scope of the embodiments.

Claims

CLAIMSWe claim:1 . A method for preparing a machine learning system configured to receive surgical theater data, the method comprising: performing a first training session with a first set of surgical theater data upon the machine learning system, the machine learning system comprising: a first portion configured to create a latent space representation from surgical theater data input; and a second portion configured to create a reconstructed representation of the surgical theater data from the latent space representation.

2. The method of Claim 1 , wherein the surgical theater data input comprises: a first surgical theater data input of a first modality; and a second surgical theater data input of a second modality3. The method of Claim 2, the wherein the method further comprises: masking at least a portion of the first surgical theater data input; and masking at least a portion of the second surgical theater data input.

4. The method of Claim 3, wherein, the first modality is surgical theater-wide visual intensity image video data, and wherein, the second modality is surgical theater-wide depth frame video data.

5. The method of Claim 3, wherein, each of the first modality and the second modality is a different one of the following modalities: surgical theater-wide depth data; surgical theater-wide visual intensity data; surgical theater kinematics data;surgical theater auditory data; surgical theater system event data; surgical theater textual data; surgical theater visual intensity data, the surgical theater visual intensity data depicting an interior of a patient; and surgical theater depth data, the surgical theater depth data depicting an interior of a patient.

6. The method of Claim 3, wherein, masking the at least the portion of the first surgical theater data input produces a first masked representation, wherein, the first masked representation comprises a first tensor, at least one dimension of the first tensor corresponding to time, wherein, masking the at least the portion of the second surgical theater data input produces a second masked representation, and wherein, the second masked representation comprises a second tensor, at least one dimension of the second tensor corresponding to time.

7. The method of Claim 6, wherein performing a first training session comprises: providing the first masked representation as input to the machine learning system; and providing the second masked representation as input to the machine learning system.

8. The method of Claim 7, wherein, masking the at least the portion of the first surgical theater data input comprises masking at least one entire temporal data frame of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking at least one entire temporal data frame of the second tensor.

9. The method of Claim 7, wherein, masking the at least the portion of the first surgical theater data input comprises masking a same first spatial portion of the first tensor across successive temporal data frames of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking a same second spatial portion of the second tensor across successive temporal data frames of the second tensor.

10. The method of Claim 9, wherein, the first spatial portion and the second spatial portion are the same spatial portion.11 . The method of Claim 3, the method comprising: modifying the machine learning system with one or more neural network layers; and performing a second training session upon the modified machine learning system.

12. The method of Claim 11 , wherein, the second training session is directed to an application involving recognition of a situational state in a surgical theater.

13. The method of Claim 11 , wherein, the second training session is directed to an application involving detection of objects appearing in a surgical theater.

14. The method of one of Claims 2-13, wherein, the first portion of the machine learning system comprises an encoder neural network, and wherein, the second portion of the machine learning system comprises a decoder neural network.

15. The method of Claim 14, wherein the first portion of the machine learning system and the second portion of the machine learning system are portions of a same autoencoder neural network.

16. The method of Claim 14, wherein, the method further comprises: determining a total loss for updating the machine learning system, wherein determining the total loss comprises determining: a first difference between the first surgical theater input and a first reconstructed surgical theater data instance; and a second difference between the second surgical theater input and a second reconstructed surgical theater data instance.

17. The method of Claim 16, wherein determining the total loss further comprises: determining a contrastive loss based, at least in part, upon a difference determined from: a first latent space representation of the first surgical theater data input; and a second latent space representation of the second surgical theater data input.

18. The method of Claim 17, wherein determining the contrastive loss comprises: determining a first mean value for the first latent space representation; determining a second mean value for the second latent space representation; and determining the contrastive loss based upon the first mean value and the second mean value.

19. The method of Claim 18, wherein determining the total loss further comprises:determining a matching loss, wherein determining the matching loss comprises: determining a binary indication whether the first latent space representation and the second latent space representation correspond.

20. The method of Claim 19, wherein determining the binary indication comprises: applying a SoftMax classifier, the SoftMax classifier itself updated, at least in part, based upon the total loss.21 . A non-transitory computer-readable medium comprising instructions configured to cause one or more computer systems to perform a method for preparing a machine learning system configured to receive surgical theater data, the method comprising: performing a first training session with a first set of surgical theater data upon the machine learning system, the machine learning system comprising: a first portion configured to create a latent space representation from surgical theater data input; and a second portion configured to create a reconstructed representation of the surgical theater data from the latent space representation.

22. The non-transitory computer-readable medium of Claim 21 , wherein the surgical theater data input comprises: a first surgical theater data input of a first modality; and a second surgical theater data input of a second modality23. The non-transitory computer-readable medium of Claim 22, the wherein the method further comprises: masking at least a portion of the first surgical theater data input; and masking at least a portion of the second surgical theater data input.

24. The non-transitory computer-readable medium of Claim 23, wherein,the first modality is surgical theater-wide visual intensity image video data, and wherein, the second modality is surgical theater-wide depth frame video data.

25. The non-transitory computer-readable medium of Claim 23, wherein, each of the first modality and the second modality is a different one of the following modalities: surgical theater-wide depth data; surgical theater-wide visual intensity data; surgical theater kinematics data; surgical theater auditory data; surgical theater system event data; surgical theater textual data; surgical theater visual intensity data, the surgical theater visual intensity data depicting an interior of a patient; and surgical theater depth data, the surgical theater depth data depicting an interior of a patient.

26. The non-transitory computer-readable medium of Claim 23, wherein, masking the at least the portion of the first surgical theater data input produces a first masked representation, wherein, the first masked representation comprises a first tensor, at least one dimension of the first tensor corresponding to time, wherein, masking the at least the portion of the second surgical theater data input produces a second masked representation, and wherein, the second masked representation comprises a second tensor, at least one dimension of the second tensor corresponding to time.

27. The non-transitory computer-readable medium of Claim 26, wherein performing a first training session comprises:providing the first masked representation as input to the machine learning system; and providing the second masked representation as input to the machine learning system.

28. The non-transitory computer-readable medium of Claim 27, wherein, masking the at least the portion of the first surgical theater data input comprises masking at least one entire temporal data frame of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking at least one entire temporal data frame of the second tensor.

29. The non-transitory computer-readable medium of Claim 27, wherein, masking the at least the portion of the first surgical theater data input comprises masking a same first spatial portion of the first tensor across successive temporal data frames of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking a same second spatial portion of the second tensor across successive temporal data frames of the second tensor.

30. The non-transitory computer-readable medium of Claim 29, wherein, the first spatial portion and the second spatial portion are the same spatial portion.31 . The non-transitory computer-readable medium of Claim 23, the method comprising: modifying the machine learning system with one or more neural network layers; and performing a second training session upon the modified machine learning system.

32. The non-transitory computer-readable medium of Claim 31 , wherein, the second training session is directed to an application involving recognition of a situational state in a surgical theater.

33. The non-transitory computer-readable medium of Claim 31 , wherein, the second training session is directed to an application involving detection of objects appearing in a surgical theater.

34. The non-transitory computer-readable medium of one of Claims 22-33, wherein, the first portion of the machine learning system comprises an encoder neural network, and wherein, the second portion of the machine learning system comprises a decoder neural network.

35. The non-transitory computer-readable medium of Claim 34, wherein the first portion of the machine learning system and the second portion of the machine learning system are portions of a same autoencoder neural network.

36. The non-transitory computer-readable medium of Claim 34, wherein, the method further comprises: determining a total loss for updating the machine learning system, wherein determining the total loss comprises determining: a first difference between the first surgical theater input and a first reconstructed surgical theater data instance; and a second difference between the second surgical theater input and a second reconstructed surgical theater data instance.

37. The non-transitory computer-readable medium of Claim 36, wherein determining the total loss further comprises:determining a contrastive loss based, at least in part, upon a difference determined from: a first latent space representation of the first surgical theater data input; and a second latent space representation of the second surgical theater data input.

38. The non-transitory computer-readable medium of Claim 37, wherein determining the contrastive loss comprises: determining a first mean value for the first latent space representation; determining a second mean value for the second latent space representation; and determining the contrastive loss based upon the first mean value and the second mean value.

39. The non-transitory computer-readable medium of Claim 38, wherein determining the total loss further comprises: determining a matching loss, wherein determining the matching loss comprises: determining a binary indication whether the first latent space representation and the second latent space representation correspond.

40. The non-transitory computer-readable medium of Claim 39, wherein determining the binary indication comprises: applying a SoftMax classifier, the SoftMax classifier itself updated, at least in part, based upon the total loss.41 . A computer system, the computer system comprising: at least one processor; and at least one memory, the at least one memory comprising instructions configured to cause the computer system to perform a method for preparing a machine learning system configured to receive surgical theater data, the method comprising:performing a first training session with a first set of surgical theater data upon the machine learning system, the machine learning system comprising: a first portion configured to create a latent space representation from surgical theater data input; and a second portion configured to create a reconstructed representation of the surgical theater data from the latent space representation.

42. The computer system of Claim 41 , wherein the surgical theater data input comprises: a first surgical theater data input of a first modality; and a second surgical theater data input of a second modality43. The computer system of Claim 42, the wherein the method further comprises: masking at least a portion of the first surgical theater data input; and masking at least a portion of the second surgical theater data input.

44. The computer system of Claim 43, wherein, the first modality is surgical theater-wide visual intensity image video data, and wherein, the second modality is surgical theater-wide depth frame video data.

45. The computer system of Claim 43, wherein, each of the first modality and the second modality is a different one of the following modalities: surgical theater-wide depth data; surgical theater-wide visual intensity data; surgical theater kinematics data; surgical theater auditory data; surgical theater system event data;surgical theater textual data; surgical theater visual intensity data, the surgical theater visual intensity data depicting an interior of a patient; and surgical theater depth data, the surgical theater depth data depicting an interior of a patient.

46. The computer system of Claim 43, wherein, masking the at least the portion of the first surgical theater data input produces a first masked representation, wherein, the first masked representation comprises a first tensor, at least one dimension of the first tensor corresponding to time, wherein, masking the at least the portion of the second surgical theater data input produces a second masked representation, and wherein, the second masked representation comprises a second tensor, at least one dimension of the second tensor corresponding to time.

47. The computer system of Claim 46, wherein performing a first training session comprises: providing the first masked representation as input to the machine learning system; and providing the second masked representation as input to the machine learning system.

48. The computer system of Claim 47, wherein, masking the at least the portion of the first surgical theater data input comprises masking at least one entire temporal data frame of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking at least one entire temporal data frame of the second tensor.

49. The computer system of Claim 47, wherein,masking the at least the portion of the first surgical theater data input comprises masking a same first spatial portion of the first tensor across successive temporal data frames of the first tensor, and wherein, masking the at least the portion of the second surgical theater data input comprises masking a same second spatial portion of the second tensor across successive temporal data frames of the second tensor.

50. The computer system of Claim 49, wherein, the first spatial portion and the second spatial portion are the same spatial portion.51 . The computer system of Claim 43, the method comprising: modifying the machine learning system with one or more neural network layers; and performing a second training session upon the modified machine learning system.

52. The computer system of Claim 51 , wherein, the second training session is directed to an application involving recognition of a situational state in a surgical theater.

53. The computer system of Claim 51 , wherein, the second training session is directed to an application involving detection of objects appearing in a surgical theater.

54. The computer system of one of Claims 42-53, wherein, the first portion of the machine learning system comprises an encoder neural network, and wherein, the second portion of the machine learning system comprises a decoder neural network.

55. The computer system of Claim 54, wherein the first portion of the machine learning system and the second portion of the machine learning system are portions of a same autoencoder neural network.

56. The computer system of Claim 54, wherein, the method further comprises: determining a total loss for updating the machine learning system, wherein determining the total loss comprises determining: a first difference between the first surgical theater input and a first reconstructed surgical theater data instance; and a second difference between the second surgical theater input and a second reconstructed surgical theater data instance.

57. The computer system of Claim 56, wherein determining the total loss further comprises: determining a contrastive loss based, at least in part, upon a difference determined from: a first latent space representation of the first surgical theater data input; and a second latent space representation of the second surgical theater data input.

58. The computer system of Claim 57, wherein determining the contrastive loss comprises: determining a first mean value for the first latent space representation; determining a second mean value for the second latent space representation; and determining the contrastive loss based upon the first mean value and the second mean value.

59. The computer system of Claim 58, wherein determining the total loss further comprises:determining a matching loss, wherein determining the matching loss comprises: determining a binary indication whether the first latent space representation and the second latent space representation correspond.

60. The computer system of Claim 59, wherein determining the binary indication comprises: applying a SoftMax classifier, the SoftMax classifier itself updated, at least in part, based upon the total loss.

Images