SYSTEMS AND METHODS FOR THE AUTOMATIC COMPLETION OF A POST-OPERATIVE REPORT ON A SURGICAL PROCEDURE

Description

Cross References to Related Applications

[0001] This application is based on and claims benefit of priority of U.S. Provisional Patent Application No. 62 / 808,500, filed February 21, 2019, U.S. Provisional Patent Application No. 62 / 808,512, filed February 21, 2019, U.S. Provisional Patent Application No. 62 / 838,066, filed April 24, 2019, U.S. Provisional Patent Application No. 62 / 960,466, filed January 13, 2020, and U.S. Provisional Patent Application No. 62 / 967,283, filed January 29, 2020.BACKGROUND Technical Field

[0002] The disclosed embodiments generally relate to systems and methods for analysis of videos of surgical procedures.Background Information

[0003] When preparing for a surgical procedure, it may be beneficial for a surgeon to view video footage depicting certain surgical events, including events that may have certain characteristics. In addition, during a surgical procedure, it may be helpful to capture and analyze videos to provide various types of decision support to surgeons. Further, it may be helpful analyze surgical videos to facilitate postoperative activity.

[0004] Therefore, there is a need for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and / or facilitate postoperative activity.

[0005] WO 2016 / 149794 A1 relates to the recording and collection of surgical data received as real-time data streams, and the use of machine-learning approaches to analyze the data streams for medical and / or clinical events and subsequent decision making.SUMMARY

[0006] Embodiments consistent with the present disclosure provide systems and methods for analysis of surgical videos. Exemplary systems and methods may be implemented using a combination of conventional hardware and software as well as specialized hardware and software, such as a machine constructed and / or programmed specifically for performing functions associated with the disclosed method steps. Consistent with other disclosed embodiments, non-transitory computer-readable storage media may store program instructions, which are executable by at least one processing device and perform any of the steps and / or methods described herein.

[0007] According to the claimed embodiments, a system for generating decision support data for surgical videos as set forth in independent claim 1 is provided.

[0008] According to further embodiments of the invention, a non-transitory computer readable medium as defined in independent claim 14 is provided.

[0009] Disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example systems, methods, and computer readable media related to reviewing surgical video are disclosed. These examples may include accessing at least one video of a surgical procedure and causing the at least one video to be output for display. These examples may further include overlaying, on the at least one video outputted for display, a surgical timeline. The surgical timeline may include markers identifying at least one of a surgical phase, an intraoperative surgical event, and a decision making junction. The surgical timeline may enable a surgeon, while viewing playback of the at least one video to select one or more markers on the surgical timeline, and thereby cause a display of the video to skip to a location associated with the selected marker.

[0010] In one example, the one or more markers may include a decision making junction marker corresponding to a decision making junction of the surgical procedure. The selection of the decision making junction marker may enable the surgeon to view two or more alternative video clips from two or more corresponding other surgical procedures. Further, the two or more video clips may present differing conduct. In another example, the selection of the decision making junction marker may cause a display of one or more alternative possible decisions related to the selected decision making junction marker.

[0011] Further disclosure not forming part of the claimed invention , but which may be useful for its implementation, relates to for example systems, methods, and computer readable media related to video indexing are disclosed. The video indexing may include accessing video footage to be indexed, including footage of a particular surgical procedure, which may be analyzed to identify a video footage location associated with a surgical phase of the particular surgical procedure. A phase tag may be generated and may be associated with the video footage location. The video indexing may include analyzing the video footage to identify an event location of a particular intraoperative surgical event within the surgical phase and associating an event tag with the event location of the particular intraoperative surgical event. Further, an event characteristic associated with the particular intraoperative surgical event may be stored.

[0012] The video indexing may further include associating at least a portion of the video footage of the particular surgical procedure with the phase tag, the event tag, and the event characteristic in a data structure that contains additional video footage of other surgical procedures. The data structure may also include respective phase tags, respective event tags, and respective event characteristics associated with one or more of the other surgical procedures. A user may be enabled to access the data structure through selection of a selected phase tag, a selected event tag, and a selected event characteristic of video footage for display. Then, a lookup in the data structure of surgical video footage matching the at least one selected phase tag, selected event tag, and selected event characteristic may be performed to identify a matching subset of stored video footage. The matching subset of stored video footage may be displayed to the user, thereby enabling the user to view surgical footage of at least one intraoperative surgical event sharing the selected event characteristic, while omitting playback of video footage lacking the selected event characteristic.

[0013] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example systems, methods, and computer readable media related to generating surgical summary footage are disclosed. These examples may include accessing particular surgical footage containing a first group of frames associated with at least one intraoperative surgical event and a second group of frames not associated with surgical activity. These examples may further include accessing historical data associated with historical surgical footage of prior surgical procedures, wherein the historical data includes information that distinguishes portions of the historical surgical footage into frames associated with intraoperative surgical events and frames not associated with surgical activity. The first group of frames in the particular surgical footage may be distinguished from the second group of frames based on the information of the historical data. Upon request of a user, an aggregate of the first group of frames of the particular surgical footage may be presented to the user, whereas the second group of frames may be omitted from presentation to the user.

[0014] Some examples may further include analyzing the particular surgical footage to identify a surgical outcome and a respective cause of the surgical outcome. The identifying may be based on the historical outcome data and respective historical cause data. An outcome set of frames in the particular surgical footage may be detected based on the analyzing. The outcome set of frames may be within an outcome phase of the surgical procedure. Further, based on the analyzing, a cause set of frames in the particular surgical footage may be detected. The cause set of frames may be within a cause phase of the surgical procedure remote in time from the outcome phase, while an intermediate set of frames may be within an intermediate phase interposed between the cause set of frames and the outcome set of frames. A cause-effect summary of the surgical footage may then be generated, wherein the cause-effect summary includes the cause set of frames and the outcome set of frames and omits the intermediate set of frames. The aggregate of the first group of frames presented to the user may include the cause-effect summary.

[0015] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media related to surgical preparation are disclosed. These examples may include accessing a repository of a plurality of sets of surgical video footage reflecting a plurality of surgical procedures performed on differing patients and including intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics. The methods may further include enabling a surgeon preparing for a contemplated surgical procedure to input case-specific information corresponding to the contemplated surgical procedure. The case-specific information may be compared with data associated with the plurality of sets of surgical video footage to identify a group of intraoperative events likely to be encountered during the contemplated surgical procedure. Further, the case-specific information and the identified group of intraoperative events likely to be encountered may be used to identify specific frames in specific sets of the plurality of sets of surgical video footage corresponding to the identified group of intraoperative events. The identified specific frames may include frames from the plurality of surgical procedures performed on differing patients.

[0016] These examples may further include determining that a first set and a second set of video footage from differing patients contain frames associated with intraoperative events sharing a common characteristic and omitting an inclusion of the second set from a compilation to be presented to the surgeon and including the first set in the compilation to be presented to the surgeon. Finally, these examples may include enabling the surgeon to view a presentation including the compilation containing frames from the differing surgical procedures performed on differing patients.

[0017] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media related to analyzing complexity of surgical footage are disclosed. These examples may include analyzing frames of the surgical footage to identify in a first set of frames an anatomical structure. These examples may further include accessing first historical data. The first historical data may be based on an analysis of first frame data captured from a first group of prior surgical procedures. The first set of frames may be analyzed using the first historical data and using the identified anatomical structure to determine a first surgical complexity level associated with the first set of frames.

[0018] Some examples may further include analyzing frames of the surgical footage to identify in a second set of frames a medical tool, the anatomical structure, and an interaction between the medical tool and the anatomical structure. These examples may include accessing second historical data, the second historical data being based on an analysis of a second frame data captured from a second group of prior surgical procedures. The second set of frames may be analyzed using the second historical data and using the identified interaction to determine a second surgical complexity level associated with the second set of frames.

[0019] These examples may further include tagging the first set of frames with the first surgical complexity level, tagging the second set of frames with the second surgical complexity level; and generating a data structure including the first set of frames with the first tag and the second set of frames with the second tag. The generated data structure may enable a surgeon to select the second surgical complexity level, and thereby cause the second set of frames to be displayed, while omitting a display of the first set of frames.

[0020] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer-readable media for enabling adjustments of an operating room schedule are disclosed. Adjusting the operating room schedule may include receiving from an image sensor positioned in a surgical operating room, visual data tracking an ongoing surgical procedure, accessing a data structure containing historical surgical data, and analyzing the visual data of the ongoing surgical procedure and the historical surgical data to determine an estimated time of completion of the ongoing surgical procedure. Adjusting the operating room schedule may further include accessing a schedule for the surgical operating room. The schedule may include a scheduled time associated with completion of the ongoing surgical procedure. Further, adjusting the operating room schedule may include calculating, based on the estimated time of completion of the ongoing surgical procedure, whether an expected time of completion is likely to result in a variance from the scheduled time associated with the completion, and outputting a notification upon calculation of the variance, to thereby enable subsequent users of the surgical operating room to adjust their schedules accordingly.

[0021] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media for analyzing surgical images to determine insurance reimbursement are disclosed. These operations for analyzing surgical images to determine insurance reimbursement may include accessing video frames captured during a surgical procedure on a patient, analyzing the video frames captured during the surgical procedure to identify in the video frames at least one medical instrument, at least one anatomical structure, and at least one interaction between the at least one medical instrument and the at least one anatomical structure, and accessing a database of reimbursement codes correlated to medical instruments, anatomical structures, and interactions between medical instruments and anatomical structures. These operations may further include comparing the identified at least one interaction between the at least one medical instrument and the at least one anatomical structure with information in the database of reimbursement codes to determine at least one reimbursement code associated with the surgical procedure.

[0022] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media for populating a post-operative report of a surgical procedure are disclosed. The operations for populating a post-operative report of a surgical procedure may include receiving an input of a patient identifier, receiving an input of an identifier of a health care provider, and receiving an input of surgical footage of a surgical procedure performed on the patient by the health care provider. These operations may further include analyzing a plurality of frames of the surgical footage to derive image-based information for populating a post-operative report of the surgical procedure, and causing the derived image-based information to populate the post-operative report of the surgical procedure.

[0023] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media for enabling determination and notification of an omitted event in a surgical procedure are disclosed. The operations for enabling determination and notification of an omitted event may include accessing frames of video captured during a specific surgical procedure, accessing stored data identifying a recommended sequence of events for the surgical procedure, comparing the accessed frames with the recommended sequence of events to identify an indication of a deviation between the specific surgical procedure and the recommended sequence of events for the surgical procedure, determining a name of an intraoperative surgical event associated with the deviation, and providing a notification of the deviation including the name of the intraoperative surgical event associated with the deviation.

[0024] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media for estimating contact force on an anatomical structure during a surgical procedure disclosed. These examples may involve receiving, from at least one image sensor in an operating room, image data of a surgical procedure, and analyzing the received image data to determine an identity of an anatomical structure and to determine a condition of the anatomical structure as reflected in the image data. A contact force threshold associated with the anatomical structure may be selected based on the determined condition of the anatomical structure. An actual contact force on the anatomical structure may be determined and compared with the selected contact force threshold. Thereafter, a notification may be output based on a determination that the indication of actual contact force exceeds the selected contact force threshold.

[0025] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods and computer readable media for updating a predicted outcome during a surgical procedure. These examples may involve receiving, from at least one image sensor arranged to capture images of a surgical procedure, image data associated with a first event during the surgical procedure. These examples may determine, based on the received image data associated with the first event, a predicted outcome associated with the surgical procedure, and may receive, from at least one image sensor arranged to capture images of a surgical procedure, image data associated with a second event during the surgical procedure. These examples may then determine, based on the received image data associated with the second event, a change in the predicted outcome, causing the predicted outcome to drop below a threshold. A recommended remedial action may be identified and recommended based on image-related data on prior surgical procedures contained in a data structure.

[0026] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems methods, and computer readable media for enabling fluid leak detection during surgery. These examples may involve receiving, in real time, intracavitary video of a surgical procedure. The processor may be configured to analyze frames of the intracavitary video to determine an abnormal fluid leakage situation in the intracavitary video. These examples may institute a remedial action when the abnormal fluid leakage situation is determined.

[0027] Further disclosure not forming part of the claimed invention, but which may be useful for its implementation, relates to, for example, systems, methods, and computer readable media for predicting post discharge risk are disclosed. The operations for predicting post discharge risk may include accessing frames of video captured during a specific surgical procedure on a patient, accessing stored historical data identifying intraoperative events and associated outcomes, analyzing the accessed frames, and based on information obtained from the historical data, identifying in the accessed frames at least one specific intraoperative event, determining, based on information obtained from the historical data and the identified at least one intraoperative event, a predicted outcome associated with the specific surgical procedure, and outputting the predicted outcome in a manner associating the predicted outcome with the patient.

[0028] The forgoing summary provides just a few examples of disclosed embodiments to provide a flavor for this disclosure and is not intended to summarize all aspects of the disclosed embodiments. Moreover, the following detailed description is exemplary and explanatory only and is not restrictive of the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. Fig. 29 relates to the claimed embodiments, while the subject matter of Figs. 1-28 and Figs. 30-36 relate to unclaimed information that may be useful for implementing the claimed embodiments. In the drawings: Fig. 1 is a perspective view of an example operating room, consistent with disclosed embodiments. Fig. 2 is a perspective view of cameras, consistent with disclosed embodiments. Fig. 3 is a perspective view of an example of a surgical instrument, consistent with disclosed embodiments. Fig. 4 illustrates an example timeline overlaid on a video of a surgical procedure consistent with the disclosed embodiments. Fig. 5 is a flowchart illustrating an example process for reviewing surgical video, consistent with the disclosed embodiments. Fig. 6 is a schematic illustration of an example data structure consistent with the disclosed embodiments. Fig. 7 is a schematic illustration of an example user interface for selecting indexed video footage for display consistent with the disclosed embodiments. Figs. 8A and 8B are flowcharts illustrating an example process for video indexing consistent with the disclosed embodiments. Fig. 9 is a flowchart illustrating an example process for distinguishing a first group of frames from a second group of frames, consistent with the disclosed embodiments. Fig. 10 is a flowchart illustrating an example process for generating a cause-effect summary, consistent with the disclosed embodiments. Fig. 11 is a flowchart illustrating an example process for generating surgical summary footage, consistent with the disclosed embodiments. Fig. 12 is a flowchart illustrating an exemplary process for surgical preparation, consistent with the disclosed embodiments. Fig. 13 is a flowchart illustrating an exemplary process for analyzing complexity of surgical footage, consistent with the disclosed embodiments. Fig. 14 is a schematic illustration of an exemplary system for managing various data collected during a surgical procedure, and for controlling various sensors consistent with disclosed embodiments. Fig. 15 is an exemplary schedule consistent with disclosed embodiments. Fig. 16 is an exemplary form for entering information for a schedule consistent with disclosed embodiments. Fig. 17A shows an exemplary data structure consistent with disclosed embodiments. Fig. 17B shows an exemplary plot of data of historic completion times consistent with disclosed embodiments. Fig. 18 shows an example of a machine-learning model consistent with disclosed embodiments. Fig. 19 shows an exemplary process for adjusting an operating room schedule consistent with disclosed embodiments. Fig. 20 is an exemplary data structure for storing correlations between reimbursement codes and information obtained from surgical footage, consistent with disclosed embodiments. Fig. 21 is block diagram of an exemplary machine learning method consistent with disclosed embodiments. Fig. 22 is a flow chart of an exemplary process for analyzing surgical images to determine insurance reimbursement, consistent with disclosed embodiments. Fig. 23 is an example post-operative report containing fields, consistent with disclosed embodiments. Fig. 24A is an example of a process, including structure, for populating a post-operative report, consistent with disclosed embodiments. Fig. 24B is another example of a process, including structure, for populating a post-operative report, consistent with disclosed embodiments. Fig. 25 is a flow diagram of an exemplary process for populating a post-operative report, consistent with disclosed embodiments. Fig. 26 is a schematic illustration of an exemplary sequence of events, consistent with disclosed embodiments. Fig. 27 shows an exemplary comparison of a sequence of events, consistent with disclosed embodiments. Fig. 28 shows an exemplary process of enabling determination and notification of an omitted event, consistent with disclosed embodiments. Fig. 29 is a flowchart illustrating an exemplary process for decision support for surgical procedures, consistent with the disclosed embodiments. Fig. 30 is a flowchart illustrating an exemplary process for estimating contact force on an anatomical structure during a surgical procedure, consistent with the disclosed embodiments Fig. 31 is a flowchart illustrating an exemplary process for updating a predicted outcome during a surgical procedure, consistent with the disclosed embodiments. Fig. 32 is a flowchart illustrating an exemplary process for enabling fluid leak detection during surgery, consistent with the disclosed embodiments. Fig. 32A is an exemplary graph showing a relationship between intraoperative events and outcomes, consistent with disclosed embodiments. Fig. 32B is an exemplary probability distribution graph for different events with and without the presence of an intraoperative event, consistent with disclosed embodiments. Fig. 33 shows exemplary probability distribution graphs for different events, consistent with disclosed embodiments. Fig. 34 shows exemplary probability distribution graphs for different events, as a function of event characteristics, consistent with disclosed embodiments. Fig. 35A shows an exemplary machine-learning model, consistent with disclosed embodiments. Fig. 35B shows an exemplary input for a machine-learning model, consistent with disclosed embodiments. Fig. 36 shows an exemplary process for predicting post discharge risk, consistent with disclosed embodiments. DETAILED DESCRIPTION

[0030] Unless specifically stated otherwise, as apparent from the following description, throughout the specification discussions utilizing terms such as "processing", "calculating", "computing", "determining", "generating", "setting", "configuring", "selecting", "defining", "applying", "obtaining", "monitoring", "providing", "identifying", "segmenting", "classifying", "analyzing", "associating", "extracting", "storing", "receiving", "transmitting", or the like, include actions and / or processes of a computer that manipulate and / or transform data into other data, the data represented as physical quantities, for example such as electronic quantities, and / or the data representing physical objects. The terms "computer", "processor", "controller", "processing unit", "computing unit", and " processing module" should be expansively construed to cover any kind of electronic device, component or unit with data processing capabilities, including, by way of non-limiting example, a personal computer, a wearable computer, smart glasses, a tablet, a smartphone, a server, a computing system, a cloud computing platform, a communication device, a processor (for example, digital signal processor (DSP), an image signal processor (ISR), a microcontroller, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPA), a graphics processing unit (GPU), a visual processing unit (VPU), and so on), possibly with embedded memory, a single core processor, a multi core processor, a core within a processor, any other electronic computing device, or any combination of the above.

[0031] The operations in accordance with the teachings herein may be performed by a computer specially constructed or programmed to perform the described functions.

[0032] As used herein, the phrase "for example," "such as", "for instance" and variants thereof describe non-limiting embodiments of the presently disclosed subject matter. Reference in the specification to features of "embodiments" "one case", "some cases", "other cases" or variants thereof means that a particular feature, structure or characteristic described may be included in at least one embodiment of the presently disclosed subject matter. Thus the appearance of such terms does not necessarily refer to the same embodiment(s). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] Features of the presently disclosed subject matter, are, for brevity, described in the context of particular embodiments. However, it is to be understood that features described in connection with one embodiment are also applicable to other embodiments. Likewise, features described in the context of a specific combination may be considered separate embodiments, either alone or in a context other than the specific combination.

[0034] In embodiments of the presently disclosed subject matter, one or more stages illustrated in the figures may be executed in a different order and / or one or more groups of stages may be executed simultaneously and vice versa. The figures illustrate a general schematic of the system architecture in accordance embodiments of the presently disclosed subject matter. Each module in the figures can be made up of any combination of software, hardware and / or firmware that performs the functions as defined and explained herein. The modules in the figures may be centralized in one location or dispersed over more than one location.

[0035] Examples of the presently disclosed subject matter are not limited in application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The subject matter may be practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

[0036] In this document, an element of a drawing that is not described within the scope of the drawing and is labeled with a numeral that has been described in a previous drawing may have the same use and description as in the previous drawings.

[0037] The drawings in this document may not be to any scale. Different figures may use different scales and different scales can be used even within the same drawing, for example different scales for different views of the same object or different scales for the two adjacent objects.

[0038] Consistent with disclosed embodiments, "at least one processor" may constitute any physical device or group of devices having electric circuitry that performs a logic operation on an input or inputs. For example, the at least one processor may include one or more integrated circuits (IC), including application-specific integrated circuit (ASIC), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), server, virtual server, or other circuits suitable for executing instructions or performing logic operations. The instructions executed by at least one processor may, for example, be pre-loaded into a memory integrated with or embedded into the controller or may be stored in a separate memory. The memory may include a Random Access Memory (RAM), a Read-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions. In some embodiments, the at least one processor may include more than one processor. Each processor may have a similar construction or the processors may be of differing constructions that are electrically connected or disconnected from each other. For example, the processors may be separate circuits or integrated in a single circuit. When more than one processor is used, the processors may be configured to operate independently or collaboratively. The processors may be coupled electrically, magnetically, optically, acoustically, mechanically or by other means that permit them to interact.

[0039] Disclosed embodiments may include and / or access a data structure. A data structure consistent with the present disclosure may include any collection of data values and relationships among them. The data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, uni-dimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, or in any manner enabling data access. By way of non-limiting examples, data structures may include an array, an associative array, a linked list, a binary tree, a balanced tree, a heap, a stack, a queue, a set, a hash table, a record, a tagged union, ER model, and a graph. For example, a data structure may include an XML database, an RDBMS database, an SQL database or NoSQL alternatives for data storage / search such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data in the data structure may be stored in contiguous or non-contiguous memory. Moreover, a data structure, as used herein, does not require information to be co-located. It may be distributed across multiple servers, for example, that may be owned or operated by the same or different entities. Thus, the term "data structure" as used herein in the singular is inclusive of plural data structures.

[0040] In some embodiments, machine learning algorithms (also referred to as machine learning models in the present disclosure) may be trained using training examples, for example in the cases described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regressions algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbors algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recursive neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and so forth. For example, a trained machine learning algorithm may comprise an inference model, such as a predictive model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, and so forth. In some examples, the training examples may include example inputs together with the desired outputs corresponding to the example inputs. Further, in some examples, training machine learning algorithms using the training examples may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate outputs for inputs not included in the training examples. In some examples, engineers, scientists, processes and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, validation examples and / or test examples may include example inputs together with the desired outputs corresponding to the example inputs, a trained machine learning algorithm and / or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and / or test examples, the estimated outputs may be compared to the corresponding desired outputs, and the trained machine learning algorithm and / or the intermediately trained machine learning algorithm may be evaluated based on a result of the comparison. In some examples, a machine learning algorithm may have parameters and hyper parameters, where the hyper parameters are set manually by a person or automatically by an process external to the machine learning algorithm (such as a hyper parameter search algorithm), and the parameters of the machine learning algorithm are set by the machine learning algorithm according to the training examples. In some implementations, the hyper-parameters are set according to the training examples and the validation examples, and the parameters are set according to the training examples and the selected hyper-parameters.

[0041] In some embodiments, trained machine learning algorithms (also referred to as trained machine learning models in the present disclosure) may be used to analyze inputs and generate outputs, for example in the cases described below. In some examples, a trained machine learning algorithm may be used as an inference model that when provided with an input generates an inferred output. For example, a trained machine learning algorithm may include a classification algorithm, the input may include a sample, and the inferred output may include a classification of the sample (such as an inferred label, an inferred tag, and so forth). In another example, a trained machine learning algorithm may include a regression model, the input may include a sample, and the inferred output may include an inferred value for the sample. In yet another example, a trained machine learning algorithm may include a clustering model, the input may include a sample, and the inferred output may include an assignment of the sample to at least one cluster. In an additional example, a trained machine learning algorithm may include a classification algorithm, the input may include an image, and the inferred output may include a classification of an item depicted in the image. In yet another example, a trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value for an item depicted in the image (such as an estimated property of the item, such as size, volume, age of a person depicted in the image, cost of a product depicted in the image, and so forth). In an additional example, a trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, a trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of objects within the image. In some examples, the trained machine learning algorithm may include one or more formulas and / or one or more functions and / or one or more rules and / or one or more procedures, the input may be used as input to the formulas and / or functions and / or rules and / or procedures, and the inferred output may be based on the outputs of the formulas and / or functions and / or rules and / or procedures (for example, selecting one of the outputs of the formulas and / or functions and / or rules and / or procedures, using a statistical measure of the outputs of the formulas and / or functions and / or rules and / or procedures, and so forth).

[0042] In some embodiments, artificial neural networks may be configured to analyze inputs and generate corresponding outputs. Some non-limiting examples of such artificial neural networks may comprise shallow artificial neural networks, deep artificial neural networks, feedback artificial neural networks, feed forward artificial neural networks, autoencoder artificial neural networks, probabilistic artificial neural networks, time delay artificial neural networks, convolutional artificial neural networks, recurrent artificial neural networks, long short term memory artificial neural networks, and so forth. In some examples, an artificial neural network may be configured manually. For example, a structure of the artificial neural network may be selected manually, a type of an artificial neuron of the artificial neural network may be selected manually, a parameter of the artificial neural network (such as a parameter of an artificial neuron of the artificial neural network) may be selected manually, and so forth. In some examples, an artificial neural network may be configured using a machine learning algorithm. For example, a user may select hyper-parameters for the an artificial neural network and / or the machine learning algorithm, and the machine learning algorithm may use the hyper-parameters and training examples to determine the parameters of the artificial neural network, for example using back propagation, using gradient descent, using stochastic gradient descent, using mini-batch gradient descent, and so forth. In some examples, an artificial neural network may be created from two or more other artificial neural networks by combining the two or more other artificial neural networks into a single artificial neural network.

[0043] In some embodiments, analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data to obtain a preprocessed image data, and subsequently analyzing the image data and / or the preprocessed image data to obtain the desired outcome. Some non-limiting examples of such image data may include one or more images, videos, frames, footages, 2D image data, 3D image data, and so forth. One of ordinary skill in the art will recognize that the followings are examples, and that the image data may be preprocessed using other kinds of preprocessing methods. In some examples, the image data may be preprocessed by transforming the image data using a transformation function to obtain a transformed image data, and the preprocessed image data may comprise the transformed image data. For example, the transformed image data may comprise one or more convolutions of the image data. For example, the transformation function may comprise one or more image filters, such as low-pass filters, high-pass filters, band-pass filters, all-pass filters, and so forth. In some examples, the transformation function may comprise a nonlinear function. In some examples, the image data may be preprocessed by smoothing at least parts of the image data, for example using Gaussian convolution, using a median filter, and so forth. In some examples, the image data may be preprocessed to obtain a different representation of the image data. For example, the preprocessed image data may comprise: a representation of at least part of the image data in a frequency domain; a Discrete Fourier Transform of at least part of the image data; a Discrete Wavelet Transform of at least part of the image data; a time / frequency representation of at least part of the image data; a representation of at least part of the image data in a lower dimension; a lossy representation of at least part of the image data; a lossless representation of at least part of the image data; a time ordered series of any of the above; any combination of the above; and so forth. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may comprise information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data. Some non-limiting examples of such image features may comprise information based on and / or related to edges; corners; blobs; ridges; Scale Invariant Feature Transform (SIFT) features; temporal features; and so forth.

[0044] In some embodiments, analyzing image data (for example, by the methods, steps and modules described herein) may comprise analyzing the image data and / or the preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, and so forth. Some non-limiting examples of such inference models may include: an inference model preprogrammed manually; a classification model; a regression model; a result of training algorithms, such as machine learning algorithms and / or deep learning algorithms, on training examples, where the training examples may include examples of data instances, and in some cases, a data instance may be labeled with a corresponding desired label and / or result; and so forth.

[0045] In some embodiments, analyzing image data (for example, by the methods, steps and modules described herein) may comprise analyzing pixels, voxels, point cloud, range data, etc. included in the image data.

[0046] Fig. 1 shows an example operating room 101, consistent with disclosed embodiments. A patient 143 is illustrated on an operating table 141. Room 101 may include audio sensors, video / image sensors, chemical sensors, and other sensors, as well as various light sources (e.g., light source 119 is shown in Fig. 1) for facilitating the capture of video and audio data, as well as data from other sensors, during the surgical procedure. For example, room 101 may include one or more microphones (e.g., audio sensor 111, as shown in Fig. 1), several cameras (e.g., overhead cameras 115, 121, and 123, and a tableside camera 125) for capturing video / image data during surgery. While some of the cameras (e.g., cameras 115, 123 and 125) may capture video / image data of operating table 141 (e.g., the cameras may capture the video / image data at a location 127 of a body of patient 143 on which a surgical procedure is performed), camera 121 may capture video / image data of other parts of operating room 101. For instance, camera 121 may capture video / image data of a surgeon 131 performing the surgery. In some cases, cameras may capture video / image data associated with surgical team personnel, such as an anesthesiologist, nurses, surgical tech and the like located in operating room 101. Additionally, operating room cameras may capture video / image data associated with medical equipment located in the room.

[0047] In various embodiments, one or more of cameras 115, 121, 123 and 125 may be movable. For example, as shown in Fig. 1, camera 115 may be rotated as indicated by arrows 135A showing a pitch direction, and arrows 135B showing a yaw direction for camera 115. In various embodiments, pitch and yaw angles of cameras (e.g., camera 115) may be electronically controlled such that camera 115 points at a region-of-interest (ROI), of which video / image data needs to be captured. For example, camera 115 may be configured to track a surgical instrument (also referred to as a surgical tool) within location 127, an anatomical structure, a hand of surgeon 131, an incision, a movement of anatomical structure, and the like. In various embodiments, camera 115 may be equipped with a laser 137 (e.g., an infrared laser) for precision tracking. In some cases, camera 115 may be tracked automatically via a computer-based camera control application that uses an image recognition algorithm for positioning the camera to capture video / image data of a ROI. For example, the camera control application may identify an anatomical structure, identify a surgical tool, hand of a surgeon, bleeding, motion, and the like at a particular location within the anatomical structure, and track that location with camera 115 by rotating camera 115 by appropriate yaw and pitch angles. In some embodiments, the camera control application may control positions (i.e., yaw and pitch angles) of various cameras 115, 121, 123 and 125 to capture video / image date from different ROIs during a surgical procedure. Additionally or alternatively, a human operator may control the position of various cameras 115, 121, 123 and 125, and / or the human operator may supervise the camera control application in controlling the position of the cameras.

[0048] Cameras 115, 121, 123 and 125 may further include zoom lenses for focusing in on and magnifying one or more ROIs. In an example embodiment, camera 115 may include a zoom lens 138 for zooming closely to a ROI (e.g., a surgical tool in the proximity of an anatomical structure). Camera 121 may include a zoom lens 139 for capturing video / image data from a larger area around the ROI. For example, camera 121 may capture video / image data for the entire location 127. In some embodiments, video / image data obtained from camera 121 may be analyzed to identify a ROI during the surgical procedure, and the camera control application may be configured to cause camera 115 to zoom towards the ROI identified by camera 121.

[0049] In various embodiments, the camera control application may be configured to coordinate the position, focus, and magnification of various cameras during a surgical procedure. For example, the camera control application may direct camera 115 to track an anatomical structure and may direct camera 121 and 125 to track a surgical instrument. Cameras 121 and 125 may track the same ROI (e.g., a surgical instrument) from different view angles. For example, video / image data obtained from different view angles may be used to determine the position of the surgical instrument relative to a surface of the anatomical structure, to determine a condition of an anatomical structure, to determine pressure applied to an anatomical structure, or to determine any other information where multiple viewing angles may be beneficial. By way of another example, bleeding may be detected by one camera, and one or more other cameras may be used to identify the source of the bleeding.

[0050] In various embodiments, control of position, orientation, settings, and / or zoom of cameras 115, 121, 123 and 125 may be rule-based and follow an algorithm developed for a given surgical procedure. For example, the camera control application may be configured to direct camera 115 to track a surgical instrument, to direct camera 121 to location 127, to direct camera 123 to track the motion of the surgeon's hands, and to direct camera 125 to an anatomical structure. The algorithm may include any suitable logical statements determining position, orientation, settings and / or zoom for cameras 115, 121, 123 and 125 depending on various events during the surgical procedure. For example, the algorithm may direct at least one camera to a region of an anatomical structure that develops bleeding during the procedure. Some non-limiting examples of settings of cameras 115, 121, 123 and 125 that may be controlled (for example by the camera control application) may include image pixel resolution, frame rate, image and / or color correction and / or enhancement algorithms, zoom, position, orientation, aspect ratio, shutter speed, aperture, focus, and so forth.

[0051] In various cases, when a camera (e.g., camera 115) tracks a moving or deforming object (e.g., when camera 115 tracks a moving surgical instrument, or a moving / pulsating anatomical structure), a camera control application may determine a maximum allowable zoom for camera 115, such that the moving or deforming object does not escape a field of view of the camera. In an example embodiment, the camera control application may initially select the first zoom for camera 115, evaluate whether the moving or deforming object escapes the field of view of the camera, and adjust the zoom of the camera as necessary to prevent the moving or deforming object from escaping the field of view of the camera. In various embodiments, the camera zoom may be readjusted based on a direction and a speed of the moving or deforming object.

[0052] In various embodiments, one or more image sensors may include moving cameras 115, 121, 123 and 125. Cameras 115, 121, 123 and 125 may be used for determining sizes of anatomical structures and determining distances between different ROIs, for example using triangulation. For example, Fig. 2 shows exemplary cameras 115 (115 View 1, as shown in Fig. 2) and 121 supported by movable elements such that the distance between the two cameras is D 1 , as shown in Fig. 2. Both cameras point at ROI 223. By knowing the positions of cameras 115 and 121 and the direction of an object relative to the cameras (e.g., by knowing angles A 1 and A 2 , as shown in Fig. 2, for example based on correspondences between pixels depicting the same object or the same real-world point in the images captured by 115 and 121), distances D 2 and D 3 may be calculated using, for example, the law of sines and the known distance between the two cameras D 1 . In an example embodiment, when camera 115 (115, View 2) rotates by a small angle A 3 (measured in radians), to point at ROI 225, the distance between ROI 223 and ROI 225 may be approximated (for small angles A 3 ) by A 3 D 2 More accuracy may be obtained using another triangulation process. Knowing distances between ROI 223 and 225 allows determining a length scale for an anatomical structure. Further, distances between various points of the anatomical structure, and distances from the various points to one or more cameras may be measured to determine a point-cloud representing a surface of the anatomical structure. Such a point-cloud may be used to reconstruct a three-dimensional model of the anatomical structure. Further, distances between one or more surgical instruments and different points of the anatomical structure may be measured to determine proper locations of the one or more surgical instruments in the proximity of the anatomical structure. In some other examples, one or more of cameras 115, 121, 123 and 125 may include a 3D camera (such as a stereo camera, an active stereo camera, a Time of Flight camera, a Light Detector and Ranging camera, etc.), and actual and / or relative locations and / or sizes of objects within operating room 101, and / or actual distances between objects, may be determined based on the 3D information captured by the 3D camera.

[0053] Returning to Fig. 1, light sources (e.g., light source 119) may also be movable to track one or more ROIs. In an example embodiment, light source 119 may be rotated by yaw and pitch angles, and in some cases, may extend towards to or away from a ROI (e.g., location 127). In some cases, light source 119 may include one or more optical elements (e.g., lenses, flat or curved mirrors, and the like) to focus light on the ROI. In some cases, light source 119 may be configured to control the color of the light (e.g., the color of the light may include different types of white light, a light with a selected spectrum, and the like). In an example embodiment, light 119 may be configured such that the spectrum and intensity of the light may vary over a surface of an anatomic structure illuminated by the light. For example, in some cases, light 119 may include infrared wavelengths which may result in warming of at least some portions of the surface of the anatomic structure.

[0054] In some embodiments, the operating room may include sensors embedded in various components depicted or not depicted in Fig. 1. Examples of such sensors may include: audio sensors; image sensors; motion sensors; positioning sensors: chemical sensors; temperature sensors; barometers; pressure sensors; proximity sensors; electrical impedance sensors; electrical voltage sensors; electrical current sensors; or any other detector capable of providing feedback on the environment or a surgical procedure, including, for example, any kind of medical or physiological sensor configured to monitor patient 143.

[0055] In some embodiments, audio sensor 111 may include one or more audio sensors configured to capture audio by converting sounds to digital information (e.g., audio sensors 121).

[0056] In various embodiments, temperature sensors may include infrared cameras (e.g., an infrared camera 117 is shown in Fig. 1) for thermal imaging. Infrared camera 117 may allow measurements of the surface temperature of an anatomic structure at different points of the structure. Similar to visible cameras D115, 121, 123 and 125, infrared camera 117 may be rotated using yaw or pitch angles. Additionally or alternatively, camera 117 may include an image sensor configured to capture image from any light spectrum, include infrared image sensor, hyper-spectral image sensors, and so forth.

[0057] Fig. 1 includes a display screen 113 that may show views from different cameras 115, 121, 123 and 125, as well as other information. For example, display screen 113 may show a zoomed-in image of a tip of a surgical instrument and a surrounding tissue of an anatomical structure in proximity to the surgical instrument.

[0058] Fig. 3 shows an example embodiment of a surgical instrument 301 that may include multiple sensors and light-emitting sources. Consistent with the present embodiments, a surgical instrument may refer to a medical device, a medical instrument, an electrical or mechanical tool, a surgical tool, a diagnostic tool, and / or any other instrumentality that may be used during a surgery. As shown, instrument 301 may include cameras 311A and 311B, light sources 313A and 313B as well as tips 323A and 323B for contacting tissue 331. Cameras 311A and 311B may be connected via data connection 319A and 319B to a data transmitting device 321. In an example embodiment, device 321 may transmit data to a data-receiving device using a wireless communication or using a wired communication. In an example embodiment, device 321 may use WiFi, Bluetooth, NFC communication, inductive communication, or any other suitable wireless communication for transmitting data to a data-receiving device. The data-receiving device may include any form of receiver capable of receiving data transmissions. Additionally or alternatively, device 321 may use optical signals to transmit data to the data-receiving device (e.g., device 321 may use optical signals transmitted through the air or via optical fiber). In some embodiments, device 301 may include local memory for storing at least some of the data received from sensors 311A and 311B. Additionally, device 301 may include a processor for compressing video / image data before transmitting the data to the data-receiving device.

[0059] In various embodiments, for example when device 301 is wireless, it may include an internal power source (e.g., a battery, a rechargeable battery, and the like) and / or a port for recharging the battery, an indicator for indicating the amount of power remaining for the power source, and one or more input controls (e.g., buttons) for controlling the operation of device 301. In some embodiments, control of device 301 may be accomplished using an external device (e.g., a smartphone, tablet, smart glasses) communicating with device 301 via any suitable connection (e.g., WiFi, Bluetooth, and the like). In an example embodiment, input controls for device 301 may be used to control various parameters of sensors or light sources. For example, input controls may be used to dim / brighten light sources 313A and 313B, move the light sources for cases when the light sources may be moved (e.g., the light sources may be rotated using yaw and pitch angles), control the color of the light sources, control the focusing of the light sources, control the motion of cameras 311A and 311B for cases when the cameras may be moved (e.g., the cameras may be rotated using yaw and pitch angles), control the zoom and / or capturing parameters for cameras 311A and 311B, or change any other suitable parameters of cameras 311A-311B and light sources 313A-313B. It should be noted camera 311A may have a first set of parameters and camera 311B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls. Similarly, light source 313A may have a first set of parameters and light source 313B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls.

[0060] Additionally, instrument 301 may be configured to measure data related to various properties of tissue 331 via tips 323A and 323B and transmit the measured data to device 321. For example, tips 323A and 323B may be used to measure the electrical resistance and / or impedance of tissue 331, the temperature of tissue 331, mechanical properties of tissue 331 and the like. To determine elastic properties of tissue 331, for example, tips 323A and 323B may be first separated by an angle 317 and applied to tissue 331. The tips may be configured to move such as to reduce angle 317, and the motion of tips may result in pressure on tissue 331. Such pressure may be measured (e.g., via a piezoelectric element 327 that may be located between a first branch 312A and a second branch 312B of instrument 301), and based on the change in angle 317 (i.e., strain) and the measured pressure (i.e., stress), the elastic properties of tissue 331 may be measured. Furthermore, based on angle 317 distance between tips 323A and 323B may be measured, and this distance may be transmitted to device 321. Such distance measurements may be used as a length scale for various video / image data that may be captured by various cameras 115, 121, 123 and 125, as shown in Fig. 1.

[0061] Instrument 301 is only one example of possible surgical instrument, and other surgical instruments such as scalpels, graspers (e.g., forceps), clamps and occluders, needles, retractors, cutters, dilators, suction tips, and tubes, sealing devices, irrigation and injection needles, scopes and probes, and the like, may include any suitable sensors and light-emitting sources. In various cases, the type of sensors and light-emitting sources may depend on a type of surgical instrument used for a surgical procedure. In various cases, these other surgical instruments may include a device similar to device 301, as shown in Fig. 3, for collecting and transmitting data to any suitable data-receiving device.

[0062] When preparing for a surgical procedure, it may be beneficial for a surgeon to review video footage of surgical procedures having similar surgical events. It may be too time consuming, however, for a surgeon to view the entire video or to skip around to find relevant portions of the surgical footage. Therefore, there is a need for unconventional approaches that efficiently and effectively enable a surgeon to view a surgical video summary that aggregates footage of relevant surgical events while omitting other irrelevant footage.

[0063] Aspects of this disclosure may relate to reviewing surgical video, including methods, systems, devices, and computer readable media. An interface may allow a surgeon to review surgical video (of their own surgeries, other's surgeries, or compilations) with a surgical timeline simultaneously displayed. The timeline may include markers keyed to activities or events that occur during a surgical procedure. These markers may allow the surgeon to skip to particular activities to thereby streamline review of the surgical procedure. In some embodiments, key decision making junction points may be marked, and the surgeon may be permitted to view alternative actions taken at those decision making junction points.

[0064] For ease of discussion, a method is described below, with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Other aspects of such a method may occur using non-electronic means. In a broadest sense, the method is not limited to particular physical and / or electronic instrumentalities, but rather may be accomplished using many differing instrumentalities.

[0065] Consistent with disclosed embodiments, a method may involve accessing at least one video of a surgical procedure. As described in greater detail above, video may include any form of recorded visual media including recorded images and / or sound. The video may be stored as a video file such as an Audio Video Interleave (AVI) file, a Flash Video Format (FLV) file, QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), a Windows Media Video (WMV) file, a Material Exchange Format (MXF) file, or any other suitable video file formats, for example as described above.

[0066] A surgical procedure may include any medical procedure associated with or involving manual or operative procedures on a patient's body. Surgical procedures may include cutting, abrading, suturing, or other techniques that involve physically changing body tissues and organs. Examples of such surgical procedures are provided above. A video of a surgical procedure may include any series of still images that were captured during and are associated with the surgical procedure. In some embodiments, at least a portion of the surgical procedure may be depicted in one or more of the still images included in the video. For example, the video of the surgical procedure may be recorded by an image capture device, such as a camera, in an operating room or in a cavity of a patient. Accessing the video of the surgical procedure may include retrieving the video from a storage device (such as one or more memory units, a video server, a cloud storage platform, or any other storage platform), receiving the video from another device through a communication device, capturing the video using image sensors, or any other means for electronically accessing data or files.

[0067] Some aspects of the present disclosure may involve causing the at least one video to be output for display. Outputting the at least one video may include any process by which the video is produced, delivered, or supplied using a computer or at least one processor. As used herein, "display" may refer to any manner in which a video may be presented to a user for playback. In some embodiments, outputting the video may include presenting the video using a display device, such as a screen (e.g., an OLED, QLED LCD, plasma, CRT, DLPT, electronic paper, or similar display technology), a light projector (e.g., a movie projector, a slide projector), a 3D display, screen of a mobile device, electronic glasses or any other form of visual and / or audio presentation. In other embodiments, outputting the video for display may include storing the video in a location that is accessible by one or more other computing devices. Such storage locations may include a local storage (such as a hard drive of flash memory), a network location (such as a server or database), a cloud computing platform, or any other accessible storage location. The video may be accessed from a separate computing device for display on the separate computing device. In some embodiments, outputting the video may include transmitting the video to an external device. For example, outputting the video for display may include transmitting the video through a network to a user device for playback on the user device.

[0068] Embodiments of the present disclosure may further include overlaying on the at least one video outputted for display a surgical timeline. As used herein, a "timeline" may refer to any depiction from which a sequence of events may be tracked or demarcated. In some embodiments, a timeline may be a graphical representation of events, for example, using an elongated bar or line representing time with markers or other indicators of events along the bar. A timeline may also be a text-based list of events arranged in chronological order. A surgical timeline may be a timeline representing events associated with a surgery. As one example, a surgical timeline may be a timeline of events or actions that occur during a surgical procedure, as described in detail above. In some embodiments, the surgical timeline may include textual information identifying portions of the surgical procedure. For example, the surgical timeline may be a list of descriptions of intraoperative surgical events or surgical phases within a surgical procedure. In other embodiments, by hovering over or otherwise actuating graphical markers on a timeline, a descriptor associated with the marker may appear.

[0069] Overlaying the surgical timeline on the at least one video may include any manner of displaying the surgical timeline such that it can be viewed simultaneously with the at least one video. In some embodiments, overlaying the video may include displaying the surgical timeline such that it at least partially overlaps the video. For example, the surgical timeline may be presented as a horizontal bar along a top or bottom of the video or a vertical bar along a side of the video. In other embodiments, overlaying may include presenting the surgical timeline alongside the video. For example, the video may be presented on a display with the surgical timeline presented above, below, and / or to the side of the video. The surgical timeline may be overlaid on the video while the video is being played. Thus, "overlaying" as used herein refers more generally to simultaneous display. The simultaneous display may or may not be constant. For example, the overlay may appear with the video output before the end of the surgical procedure depicted in the displayed video. Or, the overlay may appear during substantially all of the video procedure.

[0070] Fig. 4 illustrates an example timeline 420 overlaid on a video of a surgical procedure consistent with the disclosed embodiments. The video may be presented in a video playback region 410, which may sequentially display one or more frames of the video. In the example shown in Fig. 4, timeline 420 may be displayed as a horizontal bar representing time, with the leftmost portion of the bar representing a beginning time of the video and the rightmost portion of the bar representing an end time. Timeline 420 may include a position indicator 424 indicating the current playback position of the video relative to the timeline. Colored region 422 of timeline 420 may represent the progress within timeline 420 (e.g., corresponding to video that has already been viewed by the user, or to video coming before the currently presented frame). In some embodiments, position indicator 424 may be interactive, such that the user can move to different positions within the video by moving position indicator 424. In some embodiments, the surgical timeline may include markers identifying at least one of a surgical phase, an intraoperative surgical event, and a decision making junction. For example, timeline 420 may further include one or more markers 432, 434, and / or 436. Such markers are described in greater detail below.

[0071] In the example shown in Fig. 4, timeline 420 may be displayed such that it overlaps video playback region 410, either physically, temporally, or both. In some embodiments, timeline 420 may not be displayed at all times. As one example, timeline 420 may automatically switch to a collapsed or hidden view while a user is viewing the video and may return to the expanded view shown in Fig. 4 when the user takes an action to interact with timeline 420. For example, user may move a mouse pointer while viewing the video, move the mouse pointer over the collapsed timeline, move the mouse pointer to a particular region, click or tap the video playback region, or perform any other actions that may indicate an intent to interact with timeline 420. As discussed above, timeline 420 may be displayed in various other locations relative to video playback region 410, including on a top portion of video playback region 410, above or below video playback region 410, or within control bar 612. In some embodiments, timeline 420 may be displayed separately from a video progress bar. For example, a separate video progress bar, including position indicator 424 and colored region 422, may be displayed in control bar 412 and timeline 420 may be a separate timeline of events associated with a surgical procedure. In such embodiments, timeline 420 may not have the same scale or range of time as the video or the video progress bar. For example, the video progress bar may represent the time scale and range of the video, whereas timeline 420 may represent the timeframe of the surgical procedure, which may not be the same (e.g., where the video comprises a surgical summary, as discussed in detail above). In some embodiments, video playback region 410 may include a search icon 440, which may allow a user to search for video footage, for example, through user interface 700, as described above in reference to Fig. 7. The surgical timeline shown in Fig. 4 is provided by way of example only, and one skilled in the art would appreciate various other configurations that may be used.

[0072] Embodiments of the present disclosure may further include enabling a surgeon, while viewing playback of the at least one video to select one or more markers on the surgical timeline, and thereby cause a display of the video to skip to a location associated with the selected marker. As used herein, "playback" may include any presentation of a video in which one or more frames of the video are displayed to the user. Typically, playback will include sequentially displaying the images to reproduce moving images and / or sounds, however playback may also include the display of individual frames.

[0073] Consistent with the disclosed embodiments, a "marker" may include any visual indicator associated with location within the surgical timeline. As described above, the location may refer to any particular position within a video. For example, the location may be a particular frame or range of frames in the video, a particular timestamp, or any other indicator of position within the video. Markers may be represented on the timeline in various ways. In some embodiments, the markers may be icons or other graphic representations displayed along the timeline at various locations. The markers may be displayed as lines, bands, dots, geometric shapes (such as diamonds, squares, triangles, or any other shape), bubbles, or any other graphical or visual representation. In some embodiments, the markers may be text-based. For example, the markers may include textual information, such as a name, a description, a code, a timestamp, and so forth. In another example, the surgical timeline may be displayed as a list, as described above. Accordingly, the markers may include text-based titles or descriptions referring to a particular location of the video. Markers 432, 434, and 436 are shown by way of example in Fig. 4. The markers may be represented as callout bubbles, including an icon indicating the type of marker associated with the location. The markers may point to a particular point along timeline 420 indicating the location in the video.

[0074] Selection of the marker may include any action by a user directed towards a particular marker. In some embodiments, selecting the marker may include clicking on or tapping the marker through a user interface, touching the marker on a touch sensitive screen, glancing at the marker through smart glasses, indicating the marker through a voice interface, indicating the marker with a gesture, or undertaking any other action that causes the marker to be selected. Selection of the marker may thereby cause a display of the video to skip to a location associated with the selected marker. As used herein, skipping may include selectively displaying a particular frame within a video. This may include stopping display of a frame at a current location in the video (for example, if the video is currently playing) and displaying a frame at the location associated with the selected marker. For example, if a user clicks on or otherwise selects marker 432, as shown in Fig. 4, a frame at the location associated with marker 432 may be displayed in video playback region 410. In some embodiments, the video may continue playing from that location. Position indicator 424 may move to a position within timeline 420 associated with marker 432 and colored region 422 may be updated accordingly. While the present embodiment is described as enabling a surgeon to select the one or more markers, it is understood that this is an example only, and the present disclosure is not limited to any form of user. Various other users may view and interact with the overlaid timeline, including a surgical technician, a nurse, a physician's assistant, an anesthesiologist, a doctor, or any other healthcare professional, as well as a patient, an insurer, a medical student, and so forth. Other examples of users are provided herein.

[0075] In accordance with embodiments of the present disclosure, the markers may be automatically generated and included in the timeline based on information in the video at a given location. In some embodiments, computer analysis may be used to analyze frames of the video footage and identify markers to include at various locations in the timeline. Computer analysis may include any form of electronic analysis using a computing device. In some embodiments, computer analysis may include using one or more image recognition algorithms to identify features of one or more frames of the video footage. Computer analysis may be performed on individual frames, or may be performed across multiple frames, for example, to detect motion or other changes between frames. In some embodiments computer analysis may include object detection algorithms, such as Viola-Jones object detection, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG) features, convolutional neural networks (CNN), or any other forms of object detection algorithms. Other example algorithms may include video tracking algorithms, motion detection algorithms, feature detection algorithms, color-based detection algorithms, texture based detection algorithms, shape based detection algorithms, boosting based detection algorithms, face detection algorithms, or any other suitable algorithm for analyzing video frames. In one example, a machine learning model may be trained using training examples to generate markers for videos, and the trained machine learning model may be used to analyze the video and generate markers for that video. Such generated markers may include locations within the video for the marker, type of the marker, properties of the marker, and so forth. An example of such training example may include a video clip depicting at least part of a surgical procedure, together with a list of desired markers to be generated, possibly together with information for each desired marker, such as a location within the video for the marker, a type of the marker, properties of the marker, and so forth.

[0076] This computer analysis may be used to identify surgical phases, intraoperative events, event characteristics, and / or other features appearing in the video footage. For example, in some embodiments, computer analysis may be used to identify one or more medical instruments used in a surgical procedure, for example as described above. Based on identification of the medical instrument, a particular intraoperative event may be identified at a location in the video footage associated with the medical instrument. For example, a scalpel or other instrument may indicate that an incision is being made and a marker identifying the incision may be included in the timeline at this location. In some embodiments, anatomical structures may be identified in the video footage using the computer analysis, for example as described above. For example, the disclosed methods may include identifying organs, tissues, fluids or other structures of the patient to determine markers to include in the timeline and their respective locations. In some embodiments, locations for video markers may be determined based on an interaction between a medical instrument and the anatomical structure, which may indicate a particular intraoperative event, type of surgical procedure, event characteristic, or other information useful in identifying marker locations. For example, visual action recognition algorithms may be used to analyze the video and detect the interactions between the medical instrument and the anatomical structure. Other examples of features that may be detected in video footage for placing markers may include, motions of a surgeon or other medical professional, patient characteristics, surgeon characteristics or characteristics of other medical professionals, sequences of operations being performed, timings of operations or events, characteristics of anatomical structures, medical conditions, or any other information that may be used to identify particular surgical procedures, surgical phases, intraoperative events, and / or event characteristics appearing in the video footage.

[0077] In some embodiments, marker locations may be identified using a trained machine learning model. For example, a machine learning model may be trained using training examples, each training example may include video footage known to be associated with surgical procedures, surgical phases, intraoperative events, and / or event characteristics, together with labels indicating locations within the video footage. Using the trained machine learning model, similar phases and events may be identified in other video footage for the determining marker locations. Various machine learning models may be used, including a logistic regression model, a linear regression model, a regression model, a random forest model, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naive Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, artificial neural networks (such as deep neural networks, convolutional neural networks, etc.) or any other form of machine learning model or algorithm.

[0078] In some embodiments, video markers may be identified in conjunction with the video indexing techniques discussed above. As described above, video footage may be indexed based on surgical phases, intraoperative events, and / or event characteristics identified in the video footage. This information may be stored in a data structure, such as data structure 600, as described in reference to Fig. 6. The data structure may include footage locations and / or event locations associated with phases and events within the video footage. In some embodiments, the markers displayed in the timeline may correspond to these locations in the video. Accordingly any of the techniques or processes described above for indexing video footage may similarly apply to determining marker locations for presenting in a timeline.

[0079] According to various exemplary embodiments of the present disclosure, the markers may be coded by at least one of a color or a criticality level. The coding of a marker may be any indicator of a type, property, or characteristic of the marker. The coding may be useful for a user in visually determining which locations of the video may be of interest. Where the marker is coded by color, the color of the marker displayed on the surgical timeline may indicate the property or characteristic of the marker based on a predefined color scheme. For example, the marker may have a different color depending on what type of intraoperative surgical event the marker represents. In some example embodiments, markers associated with an incision, an excision, a resection, a ligation, a graft, or various other events may each be displayed with a different color. In other embodiments, intraoperative adverse events may be associated with one color (e.g., red), where planned events may be associated with another color (e.g., green). In some embodiments, color scales may be used. For example, the severity of an adverse event may be represented by on a color scale ranging from yellow to red, or other suitable color scales.

[0080] In some embodiments, the location and / or size of the marker may be associated with a criticality level. The criticality level may represent the relative importance of an event, action, technique, phase or other occurrence identified by the marker. Accordingly, as used herein, the term "criticality level" refers to any measure of an immediate need for an action to prevent hazardous result within a surgical procedure. For example, criticality level may include a numerical measure (such as "1.12", "3.84", "7", "-4.01", etc.), for example within a particular range of values. In another example, criticality level may include finite number of discrete levels (such as "Level 0", "Level 1", "Level 2", "High Criticality", "Low Criticality", "Non Critical", etc.).

[0081] While color is provided as one example for distinguishing marker appearance to represent information, various other techniques may be used. For example, markers may have varying sizes, shapes, positions, orientations, font size, font types, font colors, marker animations, or other visual properties. In some embodiments, markers may be associated with different icons depending on the type of event, action, or phase with which they are associated. For example, as shown in Fig. 4, marker 432, which may be associated with a decision junction, may have a different icon than marker 434, which may be associated with another type of event, such as a complication. The icon may represent the type of intraoperative event associated with that location. For example, marker 436 may indicate that an incision occurs at this location in the video. The icons (or other visual properties) may be used to distinguish between unplanned events and planned events, types of errors (e.g., miscommunication errors, judgment errors, or other forms of errors), specific adverse events that occurred, types of techniques being performed, the surgical phase being performed, locations of intraoperative surgical events (e.g., in the abdominal wall, etc.), a surgeon performing the procedure, an outcome of the surgical procedure, or various other information.

[0082] In some exemplary embodiments, the one or more markers may include a decision making junction marker corresponding to a decision making junction of the surgical procedure. In some embodiments, such decision making junction markers maybe visually distinct from other forms or types of markers. As an illustrative example, the decision making junction marker may have an icon indicating the location is associated with a decision making junction, as shown in Fig. 4 by marker 432. As used herein, a decision making junction may refer to any part of a procedure in which a decision is made, or in which a decision of a selected type of decisions or of a plurality of selected types of decisions is made. For example, the decision making junction marker may indicate a location of a video depicting a surgical procedure where multiple courses of action are possible, and a surgeon opts to follow one course over another. For example, the surgeon may decide whether to depart from a planned surgical procedure, to take a preventative action, to remove an organ or tissue, to use a particular instrument, to use a particular surgical technique, or any other intraoperative decisions a surgeon may encounter. In one example, a decision making junction may refer to a part of a procedure in which a decision that has significant effect on an outcome of the procedure is made. In another example, decision making junction may refer to a part of a procedure in which a decision that has no clear decision making guidelines has to be made. In yet another example, a decision making junction may refer to a part of a procedure in which a surgeon is faced with two or more viable alternatives, and where choosing the better alternative of the two or more viable alternatives (for example, the alternative that is predicted to reduce a particular risk, the alternative that is predicted to improve outcome, the alternative that is predicted to reduce cost, etc.) is based on at least a particular number of factors (for example, is based on at least two factors, on at least five factors, on at least ten factors, on at least one hundred factors, and so forth). In an additional example, decision making junction may refer to a part of a procedure in which a surgeon is faced with a decision of a particular type, and where the particular type is included in a group of selected decision types.

[0083] The decision making junction may be detected using the computer analysis described above. In some embodiments, video footage may be analyzed to identify particular actions or sequences of actions performed by a surgeon that may indicate a decision has been made. For example, if the surgeon pauses during a procedure, begins to use a different medical device, or changes to a different course of action, this may indicate a decision has been made. In some embodiments, the decision making junction may be identified based on a surgical phase or intraoperative event identified in the video footage at that location. For example, an adverse event, such as a bleed, may be detected which may indicate a decision must be made on how to address the adverse event. As another example, a particular phase of a surgical procedure may be associated with multiple possible courses of action. Accordingly, detecting this surgical phase in the video footage may indicate a decision making junction. In the claimed invention, a trained machine learning model is used to identify the decision making junction. In the claimed invention, the machine learning model is trained using training examples to detect decision making junctions in videos, and the trained machine learning model is used to analyze the video and detect the decision making junction. An example of such training example may include a video clip, together with a label indicating locations of decision making junctions within the video clip, or together with a label indicating an absent of decision making junctions in the video clip.

[0084] The selection of the decision making junction marker may enable the surgeon to view two or more alternative video clips from two or more corresponding other surgical procedures, thereby enabling the viewer to compare alternative approaches. Alternative video clips may be any video clips illustrating a procedure other than one currently being displayed to the user. Such an alternative may be drawn from other video footage not included in the current video being output for display. Alternatively, if the current video footage includes a compilation of differing procedures, the alternative footage may be drawn from a differing location of the current video footage being displayed. The other surgical procedures may be any surgical procedure other than the specific procedure depicted in the current video being output for display. In some embodiments, the other surgical procedures may be the same type of surgical procedure depicted in the video being output for display, but performed at different times, on different patients, and / or by different surgeons. In some embodiments, the other surgical procedures may not be the same type of procedure but may share the same or similar decision making junctions as the one identified by the decision making junction marker. In some embodiments, the two or more video clips may present differing conduct. For example, the two or more video clips may represent an alternate choice of action than the one taken in the current video, as represented by the decision making junction marker.

[0085] The alternative video clips may be presented in various ways. In some embodiments, selecting the decision making junction marker may automatically cause display of the two or more alternative video clips. For example, one or more of the alternative video clips may be displayed in video playback region 410. In some embodiments, the video playback region may be split or divided to show one or more of the alternative video clips and / or the current video. In some embodiments, the alternative video clips may be displayed in another region, such as above, below, or to the side of video playback region 410. In some embodiments, the alternative video clips may be displayed in a second window, on another screen, or in any other space other than playback region 410. According to other embodiments, selecting the decision marker may open a menu or otherwise display options for viewing the alternative video clips. For example, selecting the decision naming marker may pop up an alternative video menu containing depictions of the conduct in the associated alternative video clips. The alternative video clips may be presented as thumbnails, text-based descriptions, video previews (e.g., playing a smaller resolution version or shortened clip), or the like. The menu may be overlaid on the video, may be displayed in conjunction with the video, or may be displayed in a separate area.

[0086] In accordance with embodiments of the present disclosure, the selection of the decision making junction marker may cause a display of one or more alternative possible decisions related to the selected decision making junction marker. Similar to the alternative videos, the alternative possible decisions may be overlaid on the timeline and / or video, or may be displayed in a separate region, such as above, below and / or to the side of the video, in a separate window, on a separate screen, or in any other suitable manner. The alternative possible decisions may be a list of alternative decisions the surgeon could have made at the decision making junction. The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback. Such a list may be interactive, enabling the viewer to select an alternative course of action from the list and thereby cause video footage of the alternative course of action to be displayed.

[0087] Further, in some embodiments, one or more estimated outcomes associated with the one or more alternative possible decisions may be displayed in conjunction with the display of the one or more alternative possible decisions. For example, the list of alternative possible decisions may include estimated outcomes of each of the alternative possible decisions. The estimated outcomes may include an outcome that is predicted to occur were the surgeon to have taken the alternative possible decision. Such information may be helpful for training purposes. For example, the surgeon may be able to determine that a more appropriate action could have been taken than the one in the video and may plan future procedures accordingly. In some embodiments, each of the alternative possible decisions may be associated with multiple estimated outcomes and a probability of each may be provided. The one or more estimated outcomes may be determined in various ways. In some embodiments, the estimated outcomes may be based on known probabilities associated with the alternative possible decisions. For example, aggregated data from previous surgical procedures with similar decision making junctions may be used to predict the outcome of the alternative possible decisions associated with the marker. In some embodiments, the probabilities and / or data may be tailored to one or more characteristics or properties of the current surgical procedure. For example, patient characteristics (such as a patient's medical condition, age, weight, medical history, or other characteristics), surgeon skill level, difficulty of the procedure, type of procedure, or other factors may be considered in determining the estimated outcomes. Other characteristics may also be analyzed, including the event characteristics described above with respect to video indexing.

[0088] In accordance with the present disclosure, the decision making junction of the surgical procedure may be associated with a first patient, and the respective similar decision making junctions may be selected from past surgical procedures associated with patients with similar characteristics to the first patient. The past surgical procedures may be preselected or automatically selected based on similar estimated outcomes as the respective similar decision making junctions, or because of similarities between the patient in the current video with the patient's in the past surgical procedures. These similarities or characteristics may include a patient's gender, age, weight, height, physical fitness, heart rate, blood pressure, temperature, whether the patient exhibits a particular medical condition or disease, medical treatment history, or any other traits or conditions that may be relevant.

[0089] Similarly, in some embodiments, the decision making junction of the surgical procedure may be associated with a first medical professional, and the respective similar past decision making junctions may be selected from past surgical procedures associated with medical professionals with similar characteristics to the first medical professional. These characteristics may include, but are not limited to, the medical professional's age, medical background, experience level (e.g., the number of times the surgeon has performed this or similar surgical procedures, the total number of surgical procedures the surgeon has performed, etc.), skill level, training history, success rate for this or other surgical procedures, or other characteristics that may be relevant.

[0090] In some exemplary embodiments, the decision making junction of the surgical procedure is associated with a first prior event in the surgical procedure, and the similar past decision making junctions are selected from past surgical procedures including prior events similar to the first prior event. In one example, prior events may be determined to be similar to the first prior event based on, for example, the type of the prior events, characteristics of the prior events, and so forth. For example, a prior event may be determined as similar to the first prior event when a similarity measure between the two is above a selected threshold. Some non-limiting examples of such similarity measures are described above. The occurrence and / or characteristics of the prior event may be relevant for determining estimated outcomes for the alternative possible decisions. For example, if the surgeon runs into complications with a patient, the complications may at least partially be determinative of the most appropriate outcome, whereas a different outcome may be appropriate in absence of the complications. The first prior event may include, but is not limited to, any of the intraoperative events described in detail above. Some non-limiting characteristics of the first prior may include any of the event characteristics described above. For example, the first prior event may include an adverse event or complication, such as bleeding, mesenteric emphysema, injury, conversion to unplanned open, incision significantly larger than planned, hypertension, hypotension, bradycardia, hypoxemia, adhesions, hemias, atypical anatomy, dural tears, periorator injury, arterial occlusions, and so forth. The first prior event may also include positive or planned events, such as a successful incision, administration of a drug, usage of a surgical instrument, an excision, a resection, a ligation, a graft, suturing, stitching, or any other event.

[0091] In accordance with the present disclosure, the decision making junction of the surgical procedure may be associated with a medical condition, and the respective similar decision making junctions may be selected from past surgical procedures associated with patients with similar medical conditions. The medical conditions may include any condition of the patient related to the patient's health or well-being. In some embodiments, the medical condition may be the condition being treated by the surgical procedure. In other embodiments, the medical condition may be a separate medical condition. The medical condition may be determined in various ways. In some embodiments, the medical condition may be determined based on data associated with the plurality of videos. For example, the video may be tagged with information including the medical condition. In other embodiments, the medical condition may be determined by an analysis of the at least one video and may be based on an appearance of an anatomical structure appearing in the at least one video. For example, the color of a tissue, the relative color of one tissue with respect to the color of another tissue, size of an organ, relative size of one organ with respect to a size of another organ, appearance of a gallbladder or other organ, presence of lacerations or other marks, or any other visual indicators associated with an anatomical structure, may be analyzed to determine the medical condition. In one example, a machine learning model may be trained using training examples to determine medical conditions from videos, and the trained machine learning model may be used to analyze the at least one video footage and determine the medical condition. An example of such training example may include a video clip of a surgical procedure, together with a label indicating one or more medical conditions.

[0092] In some aspects of the present disclosure, information related to a distribution of past decisions made in respective similar past decision making junctions may be displayed in conjunction with the display of the alternative possible decisions. For example, as described above, a particular decision making junction may be associated with multiple possible decisions for a course of action. The past decisions may include decisions that were made by surgeons in previous surgical procedures when faced with the same or similar decision making junction. For example, each of the past decisions may correspond to one of the alternate possible decisions described above. Accordingly, as used herein, respective similar past decision making junctions refers to the decision making junction that occurred in the past surgical procedure when the past decision was made. In some embodiments, the respective similar past decision making junctions may be the same as the decision making junction identified by the marker. For example, if the decision making junction is an adverse event, such as a bleed, the past decisions may correspond to how other surgeons have addressed the bleed in previous surgical procedures. In other embodiments, the decision making junction may not be identical, but may be similar. For example, the possible decisions made by surgeons encountering a dural tear may be similar to other forms of tears and, accordingly, a distribution of past decisions associated with a dural tear may be relevant to the other forms of tears. The past decisions may be identified by analyzing video footage, for example, using the computer analysis techniques described above. In some embodiments, the past decisions may be indexed using the video indexing techniques described above, such that they can be readily accessed for displaying a distribution of past decisions. In one example, the distribution may include a conditional distribution, for example presenting a distribution of past decisions made in respective similar past decision making junctions that has a common property. In another example, the distribution may include an unconditional distribution, for example presenting a distribution of past decisions made in all respective similar past decision making junctions.

[0093] The displayed distribution may indicate how common each of the possible decisions were among the other alternative possible decisions associated with the respective similar past decision making junctions. In some embodiments, the displayed distribution may include a number of times each of the decisions was made. For example, a particular decision making junction may have three alternative possible decisions: decision A, decision B, and decision C. Based on the past decisions made in similar decision making junctions, the number of times each of these alternative possible decisions has been performed may be determined. For example, decision A may have been performed 167 times, decision B may have been performed 47 times, and decision C may have been performed 13 times. The distribution may be displayed as a list of each of the alternative possible decisions, along with the number of times they have been performed. The displayed distribution may also indicate the relative frequency of each of the decisions, for example, by displaying ratios, percentages, or other statistical information. For example, the distribution may indicate that decisions A, B and C have been performed in 73.6%, 20.7% and 5.7% of past decisions, respectively. In some embodiments, the distribution may be displayed as a graphical representation of the distribution, such as a bar graph, a histogram, a pie chart, a distribution curve, or any other graphical representation that may be used to show distribution.

[0094] In some embodiments, only a subset of the decisions may be displayed. For example, only the most common decisions may be displayed based on the number of times the decision was made (e.g., exceeding a threshold number of times, etc.). Various methods described above for identifying the similar past decision making junctions may be used, including identifying surgical procedures associated with similar medical conditions, patient characteristics, medical professional characteristics, and / or prior events.

[0095] In some embodiments, the one or more estimated outcomes may be a result of an analysis of a plurality of videos of past surgical procedures including respective similar decision making junctions. For example, a repository of video footage may be analyzed using various computer analysis techniques, such as the object and / or motion detection algorithms described above, to identify videos including decision making junctions that are the same as or share similar characteristics with the decision making junction identified by the marker. This may include identifying other video footage having the same or similar surgical phases, intraoperative surgical events, and / or event characteristics as those that were used to identify the decision making junction in the video presented in the timeline. The outcomes of the alternative possible decisions may be estimated based on the outcomes in the past surgical procedures. For example, if a particular method of performing a suture consistently results in a full recovery by the patient, this outcome may be estimated for this possible decision and may be displayed on the timeline.

[0096] In some exemplary embodiments, the analysis may include usage of an implementation of a computer vision algorithm. The computer vision algorithm may be the same as or similar to any of the computer vision algorithms described above. One example of such computer algorithm may include the object detection and tracking algorithms described above. Another example of such computer vision algorithm may include usage of a trained machine learning model. Other non-limiting examples of such computer vision algorithm are described above. For example, if the decision making junction marker was identified based on a particular adverse event occurring in the video, other video footage having the same or similar adverse events may be identified. The video footage may further be analyzed to determine an outcome of the decision made in past surgical video. This may include the same or similar computer analysis techniques described above. In some embodiments, this may include analyzing the video to identify the result of the decision. For example, if the decision making junction is associated with an adverse event associated with an anatomical structure, such as a tear, the anatomical structure may be assessed at various frames after the decision to determine whether the adverse event was remediated, how quickly it was remediated, whether additional adverse events occurred, whether the patient survived, or other indicators of the outcome.

[0097] In some embodiments, additional information may also be used to determine the outcome. For example, the analysis may be based on one or more electronic medical records associated with the plurality of videos of past surgical procedures. For example, determining the outcome may include referencing an electronic medical record associated with the video in which a particular decision was made to determine whether the patient recovered, how quickly the patient recovered, whether there were additional complications, or the like. Such information may be useful in predicting the outcome that may result at a later time, outside of the scope of the video footage. For example, the outcome may be several days, weeks, or months after the surgical procedure. In some embodiments, the additional information may be used to inform the analysis of which videos to include in the analysis. For example, using information gleaned from the medical records, videos sharing similar patient medical history, disease type, diagnosis type, treatment history (including past surgical procedures), healthcare professional identities, healthcare professional skill levels, or any other relevant data may be identified. Videos sharing these or other characteristics may provide a more accurate idea of what outcome can be expected for each alternative possible decision.

[0098] The similar decision making junctions may be identified based on how closely they correlate to the current decision making junction. In some embodiments, the respective similar decision making junctions may be similar to the decision making junction of the surgical procedure according to a similarity metric. The metric may be any value, classification, or other indicator of how closely the decision making junctions are related. Such a metric may be determined based on the computer vision analysis in order to determine how closely the procedures or techniques match. The metric may also be determined based on the number of characteristics the decision making junctions have in common and the degree to which the characteristics match. For example, two decision making junctions with patients having similar medical conditions and physical characteristics may be assigned a higher similarity based on the similarity metric than two more distinctive patients. Various other characteristics and / or considerations may also be used. Additionally or alternatively, the similarity metric may be based on any similarity measure, such as the similarity measures described above. For example, the similarity metric may be identical to the similarity measure, may be a function of the similarity measure, and so forth.

[0099] Various other marker types may be used in addition to or instead of decision making junction markers. In some embodiments, the markers may include intraoperative surgical event markers, which may be associated with locations in the video associated with the occurrence of an interoperative event. Examples of various intraoperative surgical events that may be identified by the markers are provided throughout the present disclosure, including in relation to the video indexing described above. In some embodiments, the intraoperative surgical event markers may be generic markers, indicating that an intraoperative surgical event occurred at that location. In other embodiments, the intraoperative surgical event markers may identify a property of the intraoperative surgical event, including the type of the event, whether the event was an adverse event, or any other characteristic. Example markers are shown in Fig. 4. As an illustrative example, the icon shown for marker 434 may be used to represent a generic intraoperative surgical event marker. Marker 436 on the other hand, may represent a more specific intraoperative surgical event marker, such as identifying that an incision occurred at that location. The markers shown in Fig. 4 are provided by way of example, and various other forms of markers may be used.

[0100] These intraoperative surgical event markers may be identified automatically, as described above. Using the computer analysis methods described above, medical instruments, anatomical structures, surgeon characteristics, patient characteristics, event characteristics, or other features may be identified in the video footage. For example, the interaction between an identified medical instrument and an anatomical structure may indicate that an incision, a suturing, or other intraoperative event is being performed. In some embodiments, the intraoperative surgical event markers may be identified based on information provided in a data structure, such as data structure 600 described above in reference to Fig. 6.

[0101] Consistent with the disclosed embodiments, selection of an intraoperative surgical event marker may enable the surgeon to view alternative video clips from differing surgical procedures. In some embodiments, the alternative video clips may present differing ways in which a selected intraoperative surgical event was handled. For example, in the current video the surgeon may perform an incision or other action according to one technique. Selecting the intraoperative surgical event markers may allow the surgeon to view alternative techniques that may be used to perform the incision or other action. In another example, the intraoperative surgical event may be an adverse event, such as a bleed, and the alternative video clips may depict other ways surgeons have handled the adverse event. In some embodiments, where the markers relate to intraoperative surgical events, the selection of an intraoperative surgical event marker may enable the surgeon to view alternative video clips from differing surgical procedures. For example, the differing surgical procedures may be of a different type (such as a laparoscopic surgery versus thoracoscopic surgery) but may still include the same or similar intraoperative surgical events. The surgical procedures may also differ in other ways, including differing medical conditions, differing patient characteristics, differing medical professionals, or other distinctions. Selecting the intraoperative surgical event marker may allow the surgeon to view alternative video clips from the differing surgical procedures.

[0102] The alternative video clips may be displayed in various ways, similar to other embodiments described herein. For example, selecting the intraoperative surgical event markers may cause a menu to be displayed, from which the surgeon may select the alternative video clips. The menu may include descriptions of the differing ways in which the selected intraoperative surgical event was handled, thumbnails of the video clips, previews of the video clips, and / or other information associated with the video clips, such as the dates they were recorded, the type of surgical procedure, a name or identity of a surgeon performing the surgical procedure, or any other relevant information.

[0103] In accordance with some embodiments of the present disclosure, the at least one video may include a compilation of footage from a plurality of surgical procedures, arranged in procedural chronological order. Procedural chronological order may refer to the order events occur relative to a surgical procedure. Accordingly, arranging a compilation of footage in procedural chronological order may include arranging the different events from differing patients in the order in which they would have occurred if the procedure had been conducted on a single patient. In other words, although compiled from various surgeries on differing patients, playback of the compilation will display the footage in the order the footage would appear within the surgical procedure. In some embodiments, the compilation of footage may depict complications from the plurality of surgical procedures. In such embodiments, the one or more markers may be associated with the plurality of surgical procedures and may be displayed on a common timeline. Thus, although a viewer interacts with a single timeline, the video footage presented along the timeline may be derived from differing procedures and / or differing patients. Example complications that may be displayed are described above with respect to video indexing.

[0104] Fig. 5 is a flowchart illustrating an example process 500 for reviewing surgical videos, consistent with the disclosed embodiments. Process 500 may be performed by at least one processor, such as one or more microprocessors. In some embodiments, process 500 is not necessarily limited to the steps illustrated, and any of the various embodiments described herein may also be included in process 500. At step 510, process 500 may include accessing at least one video of a surgical procedure, for example as described above. The at least one video may include video footage from a single surgical procedure or may be a compilation of footage from a plurality of procedures, as previously discussed. Process 500 may include causing the at least one video to be output for display in step 520. As described above, causing the at least one video to be output for display may include sending a signal for causing display of the at least one video on a screen or other display device, storing the at least one video in a location accessible to another computing device, transmitting the at least one video, or any other process or steps that may cause the video to be displayed.

[0105] At step 530, process 500 may include overlaying on the at least one video outputted for display a surgical timeline, wherein the surgical timeline includes markers identifying at least one of a surgical phase, an intraoperative surgical event, and a decision making junction. In some embodiments, the surgical timeline may be represented as a horizontal bar displayed along with the video. The markers may be represented as shapes, icons, or other graphical representations along the timeline. Fig. 4 provides an example of such an embodiment. In other embodiments, the timeline may be a text-based list of phases, events, and / or decision making junctions in chronological order. The markers may similarly be text-based and may be included in the list.

[0106] Step 540 may include enabling a surgeon, while viewing playback of the at least one video, to select one or more markers on the surgical timeline, and thereby cause a display of the video to skip to a location associated with the selected marker. In some embodiments, the surgeon may be able to view additional information about the event or occurrence associated with the marker, which may include information from past surgical procedures. For example, the markers may be associated with an intraoperative surgical event and selecting the marker may enable the surgeon to view alternative video clips of past surgical procedures associated with the intraoperative surgical event. For example, the surgeon may be enabled to view clips from other surgeries where a similar intraoperative surgical event was handled differently, where a different technique was used, or where an outcome varied. In some embodiments, the marker may be a decision making junction marker, representing a decision that was made during the surgical procedure. Selecting the decision making junction marker may enable the surgeon to view information about the decision, including alternative decisions. Such information may include videos of past surgical procedures including similar decision making junctions, a list or distribution of alternate possible decisions, estimated outcomes of the alternate possible decisions, or any other relevant information. Based on the steps described in process 500, the surgeon or other users may be able to more effectively and more efficiently review surgical videos using the timeline interface.

[0107] In preparing for a surgical procedure, it is often beneficial for surgeons to review videos of similar surgical procedures that have been performed. It may be too cumbersome and time consuming, however, for a surgeon to identify relevant videos or portions of videos in preparing for a surgical procedure. Therefore, there is a need for unconventional approaches that efficiently, effectively index surgical video footage based on contents of the footage such that it may be easily accessed and reviewed by a surgeon or other medical professional.

[0108] Aspects of this disclosure may relate to video indexing, including methods, systems, devices, and computer readable media. For example, surgical events within surgical phases may be automatically detected in surgical footage. Viewers may be enabled to skip directly to an event, to view only events with specified characteristics, and so forth. In some embodiments, a user may specify within a surgical phase (e.g., a dissection) an event (e.g., inadvertent injury to an organ) having a characteristic (e.g., a particular complication), so that the user may be presented with video clips of one or more events sharing that characteristic.

[0109] For ease of discussion, a method is described below, with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Other aspects of such a method may occur using non-electronic means. In a broadest sense, the method is not limited to particular physical and / or electronic instrumentalities, but rather may be accomplished using many differing instrumentalities.

[0110] Consistent with disclosed embodiments, a method may involve accessing video footage to be indexed, the video footage to be indexed including footage of a particular surgical procedure. As used herein, video may include any form of recorded visual media including recorded images and / or sound. For example, a video may include a sequence of one or more images captured by an image capture device, such as cameras 115, 121, 123, and / or 125, as described above in connection with Fig. 1. The images may be stored as individual files or may be stored in a combined format, such as a video file, which may include corresponding audio data. In some embodiments, video may be stored as raw data and / or images output from an image capture device. In other embodiments the video may be processed. For example, video files may include Audio Video Interleave (AVI), Flash Video Format (FLV), QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), Windows Media Video (WMV), Material Exchange Format (MXF), uncompressed format, lossy compressed format, lossless compressed format, or any other suitable video file formats.

[0111] Video footage may refer to a length of video that has been captured by an image capture device. In some embodiments, video footage may refer to a length of video that includes a sequence of images in the order they were originally captured in. For example, video footage may include video that has not been edited to form a video compilation. In other embodiments, video footage may be edited in one or more ways, such as to remove frames associated with inactivity, or to otherwise compile frames not originally captured sequentially. Accessing the video footage may include retrieving video footage from a storage location, such as a memory device. The video footage may be accessed from a local memory, such as a local hard drive, or may be accessed from a remote source, for example, through a network connection. Consistent with the present disclosure, indexing may refer to a process for storing data such that it may be retrieved more efficiently and / or effectively. Indexing video footage may include associating one or more properties or indicators with the video footage such that the video footage may be identified based on the properties or indicators.

[0112] A surgical procedure may include any medical procedure associated with or involving manual or operative procedures on a patient's body. Surgical procedures may include cutting, abrading, suturing, or other techniques that involve physically changing body tissues and organs. Some examples of such surgical procedures may include a laparoscopic surgery, a thoracoscopic procedure, a bronchoscopic procedure, a microscopic procedure, an open surgery, a robotic surgery, an appendectomy, a carotid endarterectomy, a carpal tunnel release, a cataract surgery, a cesarean section, a cholecystectomy, a colectomy (such as a partial colectomy, a total colectomy, etc.), a coronary angioplasty, a coronary artery bypass, a debridement (for example of a wound, a burn, an infection, etc.), a free skin graft, a hemorrhoidectomy, a hip replacement, a hysterectomy, a hysteroscopy, an inguinal hernia repair, a knee arthroscopy, a knee replacement, a mastectomy (such as a partial mastectomy, a total mastectomy, a modified radical mastectomy, etc.), a prostate resection, a prostate removal, a shoulder arthroscopy, a spine surgery (such as a spinal fusion, a laminectomy, a foraminotomy, a discectomy, a disk replacement, an interlaminar implant, etc.), a tonsillectomy, a cochlear implant procedure, brain tumor (for example meningioma, etc.) resection, interventional procedures such as percutaneous transluminal coronary angioplasty, transcatheter aortic valve replacement, minimally Invasive surgery for intracerebral hemorrhage evacuation, or any other medical procedure involving some form of incision. While the present disclosure is described in reference to surgical procedures, it is to be understood that it may also apply to other forms of medical procedures, or procedures generally.

[0113] In some exemplary embodiments, the accessed video footage may include video footage captured via at least one image sensor located in at least one of a position above an operating table, in a surgical cavity of a patient, within an organ of a patient or within vasculature of a patient. An image sensor may be any sensor capable of recording video. An image sensor located in a position above an operating table may include any image sensor placed external to a patient configured to capture images from above the patient. For example, the image sensor may include cameras 115 and / or 121, as shown in Fig. 1. In other embodiments, the image sensor may be placed internal to the patient, such as, for example, in a cavity. As used herein, a cavity may include any relatively empty space within an object. Accordingly, a surgical cavity may refer to a space within the body of a patient where a surgical procedure or operation is being performed, or where surgical tools are present and / or used. It is understood that the surgical cavity may not be completely empty but may include tissue, organs, blood or other fluids present within the body. An organ may refer to any self-contained region or part of an organism. Some examples of organs in a human patient may include a heart or liver. A vasculature may refer to a system or grouping of blood vessels within an organism. An image sensor located in a surgical cavity, an organ, and / or a vasculature may include a camera included on a surgical tool inserted into the patient.

[0114] Aspects of this disclosure may include analyzing the video footage to identify a video footage location associated with a surgical phase of the particular surgical procedure. As used herein with respect to video footages, a location may refer any particular position or range within the video footage. In some embodiments the location may include a particular frame or range of frames of a video. Accordingly, video footage locations may be represented as one or more frame numbers or other identifiers of a video footage file. In other embodiments, the location may refer to a particular time associated with the video footage. For example, a video footage location may refer to a time index or timestamp, a time range, a particular starting time and / or ending time, or any other indicator of position within the video footage. In other embodiments, the location may refer to at least one particular position within at least one frame. Accordingly, video footage locations may be represented as one or more pixels, voxels, bounding boxes, bounding polygons, bounding shapes, coordinates, and so forth.

[0115] For the purposes of the present disclosure, a phase may refer to a particular period or stage of a process or series of events. Accordingly, a surgical phase may refer to a particular period or stage of a surgical procedure, as described above. For example, surgical phases of a laparoscopic cholecystectomy surgery may include trocar placement, preparation, calot's triangle dissection, clipping and cutting of cystic duct and artery, gallbladder dissection, gallbladder packaging, cleaning and coagulation of liver bed, gallbladder retraction, and so forth. In another example, surgical phases of a cataract surgery may include preparation, povidone-iodine injection, corneal incision, capsulorhexis, phaco-emulsification, cortical aspiration, intraocular lens implantation, intraocular-lens adjustment, wound sealing, and so forth. In yet another example, surgical phases of a pituitary surgery may include preparation, nasal incision, nose retractor installation, access to the tumor, tumor removal, column of nose replacement, suturing, nose compress installation, and so forth. Some other examples of surgical phases may include preparation, incision, laparoscope positioning, suturing, and so forth.

[0116] In some embodiments, identifying the video footage location may be based on user input. User input may include any information provided by a user. As used with respect to video indexing, the user input may include information relevant to identifying the video footage location. For example, a user may input a particular frame number, timestamp, range of times, start times and / or stop times, or any other information that may identify a video footage location. Alternatively, the user input might include entry or selection of a phase, event, procedure, or device used, which input may be associated with particular video footage (e.g., for example through a lookup table or other data structure). The user input may be received through a user interface of a user device, such as a desktop computer, a laptop, a table, a mobile phone, a wearable device, an internet of things (IoT) device, or any other means for receiving input from a user. The interface may include, for example, one or more drop down menus with one or more pick lists of phase names; a data entry field that permits the user to enter the phase name and / or that suggests phase names once a few letters are entered; a pick list from which phase names may be chosen; a group of selectable icons each associated with a differing phase, or any other mechanism that allows users to identify or select a phase. For example, a user may input the phase name through a user interface similar to user interface 700, as described in greater detail below with respect to Fig. 7. In another example, the user input may be received through voice commands and / or voice inputs, and the user input may be processed using speech recognition algorithms. In yet another example, the user input may be received through gestures (such as hand gestures), and the user input may be processed using gesture recognition algorithms.

[0117] In some embodiments, identifying the video footage location may include using computer analysis to analyze frames of the video footage. Computer analysis may include any form of electronic analysis using a computing device. In some embodiments, computer analysis may include using one or more image recognition algorithms to identify features of one or more frames of the video footage. Computer analysis may be performed on individual frames, or may be performed across multiple frames, for example, to detect motion or other changes between frames. In some embodiments computer analysis may include object detection algorithms, such as Viola-Jones object detection, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG) features, convolutional neural networks (CNN), or any other forms of object detection algorithms. Other example algorithms may include video tracking algorithms, motion detection algorithms, feature detection algorithms, color-based detection algorithms, texture based detection algorithms, shape based detection algorithms, boosting based detection algorithms, face detection algorithms, or any other suitable algorithm for analyzing video frames. In one example, a machine learning model may be trained using training examples to identify particular locations within videos, and the trained machine learning model may be used to analyze the video footage and identify the video footage location. An example of such training example may include a video clip together with a label indicating a location within a video clip, or together with a label indicating that no corresponding location is included within the video clip.

[0118] In some embodiments, the computer image analysis may include using a neural network model trained using example video frames including previously-identified surgical phases to thereby identify at least one of a video footage location or a phase tag. In other words, frames of one or more videos that are known to be associated with a particular surgical phase may be used to train a neural network model, for example using a machine learning algorithm, using back propagation, using gradient descent optimization, and so forth. The trained neural network model may therefore be used to identify whether one or more video frames are also associated with the surgical phase. Some non-limiting examples of such artificial neural networks may comprise shallow artificial neural networks, deep artificial neural networks, feedback artificial neural networks, feed forward artificial neural networks, autoencoder artificial neural networks, probabilistic artificial neural networks, time delay artificial neural networks, convolutional artificial neural networks, recurrent artificial neural networks, long short term memory artificial neural networks, and so forth. In some embodiments, the disclosed methods may further include updating the trained neural network model based on at least one of the analyzed frames.

[0119] In some aspects of the present disclosure, analyzing the video footage to identify the video footage location associated with at least one of the surgical event or the surgical phase may include performing computer image analysis on the video footage to identify at least one of a beginning location of the surgical phase for playback or a beginning of a surgical event for playback. In other words, using the computer analysis techniques discussed above, the disclosed methods may include identifying a location within the video footage where a surgical phase or event begins. For example, the beginning of a surgical event, such as an incision, may be detected using the object and / or motion detection algorithms described above. In other embodiments, the beginning of the incision may be detected based on machine learning techniques. For example, a machine learning model may be trained using video footage and corresponding label indicating known beginning points of an incision or other surgical events and / or procedures. The trained model may be used to identify similar procedure and / or event beginning locations within other surgical video footage.

[0120] Some aspects of this disclosure may include generating a phase tag associated with the surgical phase. As used herein, a "tag" may refer to any process or marker by which information is associated with or linked to a set of data. In some embodiments, a tag may be a property of a data file, such as a video file. Accordingly, generating the tag may include writing or overwriting properties within a video file. In some embodiments, generating a tag may include writing information to a file other than the video file itself, for example, by associating the video file with the tag in a separate database. The tag may be expressed as textual information, a numerical identifier, or any other suitable means for tagging. A phase tag may be a tag that identifies a phase of a surgical phase, as described above. In one embodiment, a phase tag may be a marker indicating a location in video where a surgical phase begins, a marker indicating a location in video where a surgical phase ends, a marker indicating a location in video in the middle of a surgical phase, or indicating a range of video encompassing the surgical phase. The tag may be a pointer in the video data itself or may be located in a data structure to permit a lookup of a phase location. The phase tag may include computer readable information for causing display of the phase and may also include human-readable information for identifying the phase to a user. For example, generating a phase tag associated with the surgical phase may include generating a tag including text such as "laparoscope positioning" to indicate the tagged data is associated with that phase of the surgical procedure. In another example, generating a phase tag associated with the surgical phase may include generating a tag including binary encoding of a surgical phase identifier. In some embodiments, generating the phase tag may be based on a computer analysis of video footage depicting the surgical phase. For example, the disclosed methods may include analyzing footage of the surgical phase using the object and motion detection analysis methods described above to determine the phase tag. For example, if it is known that a phase begins or ends using a particular type of medical device or other instrumentality used in a unique way or in a unique order, image recognition may be performed on the video footage to identify a particular phase through image recognition performed to identify the unique use of the instrumentality to identify a particular phase. Generating the phase tag may also include using a trained machine learning model or a neural network model (such as deep neural network, convolutional neural networks, etc.), which may be trained to associate one or more video frames with one or more phase tags. For example, training examples may be fed to a machine learning algorithm to develop a model configured to associate other video footage data with one or more phase tags. An example of such training example may include a video footage together with a label indicating the desired tags or the absent of desired tags corresponding to the video footage. Such label may include an indication of one or more locations within the video footage corresponding to the surgical phase, an indication of a type of the surgical phase, an indication of properties of the surgical phase, and so forth.

[0121] A method in accordance with the present disclosure may include associating the phase tag with the video footage location. Any suitable means may be used to associate the phase tag with the video footage location. Such tag may include an indication of one or more locations within the video footage corresponding to the surgical phase, an indication of a type of the surgical phase, an indication of properties of the surgical phase, and so forth. In some embodiments, the video footage location may be included in the tag. For example, the tag may include a timestamp, time range, frame number, or other means for associating the phase tag to the video footage location. In other embodiments, the tag may be associated with the video footage location in a database. For example, the database may include information linking the phase tag to the video footage and to the particular video footage location. The database may include a data structure, as described in further detail below.

[0122] Embodiments of the present disclosure may further include analyzing the video footage to identify an event location of a particular intraoperative surgical event within the surgical phase. An intraoperative surgical event may be any event or action that occurs during a surgical procedure or phase. In some embodiments, an intraoperative surgical event may include an action that is performed as part of a surgical procedure, such as an action performed by a surgeon, a surgical technician, a nurse, a physician's assistant, an anesthesiologist, a doctor, or any other healthcare professional. The intraoperative surgical event may be a planned event, such as an incision, administration of a drug, usage of a surgical instrument, an excision, a resection, a ligation, a graft, suturing, stitching, or any other planned event associated with a surgical procedure or phase. In some embodiments, the intraoperative surgical event may include an adverse event or a complication. Some examples of intraoperative adverse events may include bleeding, mesenteric emphysema, injury, conversion to unplanned open surgery (for example, abdominal wall incision), incision significantly larger than planned, and so forth. Some examples of intraoperative complications may include hypertension, hypotension, bradycardia, hypoxemia, adhesions, hernias, atypical anatomy, dural tears, periorator injury, arterial occlusions, and so forth. The intraoperative event may include other errors, including technical errors, communication errors, management errors, judgment errors, decision making errors, errors related to medical equipment utilization, miscommunication, and so forth.

[0123] The event location may be a location or range within the video footage associated with the intraoperative surgical event. Similar to the phase location described above, the event location may be expressed in terms of particular frames of the video footage (e.g., a frame number or a range of frame numbers) or based on time information (e.g., a timestamp, a time range, or beginning and end times), or any other means for identifying a location within the video footage. In some embodiments, analyzing the video footage to identify the event location may include using computer analysis to analyze frames of the video footage. The computer analysis may include any of the techniques or algorithms described above. As with phase identification, event identification may be based on a detection of actions and instrumentalities used in a way that uniquely identifies an event. For example, image recognition may identify when a particular organ is incised, to enable marking of that incision event. In another example, image recognition may be used to note the severance of a vessel or nerve, to enable marking of that adverse event. Image recognition may also be used to mark events by detection of bleeding or other fluid loss. In some embodiments, analyzing the video footage to identify the event location may include using a neural network model (such as a deep neural network, a convolutional neural network, etc.) trained using example video frames including previously-identified surgical events to thereby identify the event location. In one example, a machine learning model may be trained using training examples to identify locations of intraoperative surgical events in portions of videos, and the trained machine learning model may be used to analyze the video footage (or a portion of the video footage corresponding to the surgical phase) and identify the event location of the particular intraoperative surgical event within the surgical phase. An example of such training example may include a video clip together with a label indicating a location of a particular event within the video clip, or an absence of such event.

[0124] Some aspects of the present disclosure may involve associating an event tag with the event location of the particular intraoperative surgical event. As discussed above, a tag may include any means for associating information with data or a portion of data. An event tag may be used to associate data or portions of data with an event, such as an intraoperative surgical event. Similar to the phase tag, associating the event tag with the event location may include writing data to a video file, for example, to the properties of the video file. In other embodiments, associating the event tag with the event location may include writing data to a file or database associating the event tag with the video footage and / or the event location. Alternatively, associating an event tag with an event location may include recording a marker in a data structure, where the data structure correlates a tag with a particular location or range of locations in video footage. In some embodiments, the same file or database may be used to associate the phase tag to the video footage as the event tag. In other embodiments, a separate file or database may be used.

[0125] Consistent with the present disclosure, the disclosed methods may include storing an event characteristic associated with the particular intraoperative surgical event. The event characteristic may be any trait or feature of the event. For example, the event characteristic may include properties of the patient or surgeon, properties or characteristics of the surgical event or surgical phase, or various other traits. Examples of features may include, excessive fatty tissue, an enlarged organ, tissue decay, a broken bone, a displaced disc, or any other physical characteristic associated with the event. Some characteristics may be discernable by computer vision, and others may be discernable by human input. In the latter example, the age or age range of a patient may be stored as an event characteristic. Similarly, aspects of a patient's prior medical history may be stored as an event characteristic (e.g., patient with diabetes). In some embodiments, the stored event characteristic may be used to distinguish intraoperative surgical events from other similar events. For example, a medical practitioner may be permitted to search video footage to identify one or more coronary artery bypass surgeries performed on males over the age of 70 with arrhythmia. Various other examples of stored event characteristics that may be used are provided below.

[0126] The stored event characteristic may be determined in various ways. Some aspects of the disclosed methods may involve determining the stored event characteristic based on user input. For example, a user may input the event characteristic to be stored via a user interface similar to what was described above in connection with the selection of a phase or an event. In another example, a user may input the event characteristic to be stored via voice commands. Various examples of such uses are provided below. Other aspects of the disclosed methods may involve determining the stored event characteristic based on a computer analysis of video footage depicting the particular intraoperative surgical event. For example, the disclosed methods may include using various image and / or video analysis techniques as described above to recognize event characteristics based on the video footage. As an illustrative example, the video footage may include a representation of one or more anatomical structures of a patient and an event characteristic identifying the anatomical structures may be determined based on detecting the anatomical structure in the video footage, or based on detecting the interaction between a medical instrument and the anatomical structure. In another example, a machine learning model may be trained using training examples to determine event characteristics from videos, and the trained machine learning model may be used to analyze the video footage and determine the stored event characteristic. An example of such training example may include a video clip depicting an intraoperative surgical event together with a label indicating a characteristic of the event.

[0127] Some aspects of the present disclosure may include associating at least a portion of the video footage of the particular surgical procedure with the phase tag, the event tag, and the event characteristic in a data structure that contains additional video footage of other surgical procedures, wherein the data structure also includes respective phase tags, respective event tags, and respective event characteristics associated with one or more of the other surgical procedures. A data structure consistent with this disclosure may include any collection of data values and relationships among them. The data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, uni-dimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, in a searchable repository, in a sorted repository, in an indexed repository, or in any manner enabling data access. By way of non-limiting examples, data structures may include an array, an associative array, a linked list, a binary tree, a balanced tree, a heap, a stack, a queue, a set, a hash table, a record, a tagged union, ER model, and a graph. For example, a data structure may include an XML database, an RDBMS database, an SQL database or NoSQL alternatives for data storage / search such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). Data in the data structure may be stored in contiguous or non-contiguous memory. Moreover, a data structure, as used herein, does not require information to be co-located. It may be distributed across multiple servers, for example, that may be owned or operated by the same or different entities. Thus, for example, a data structure may include any data format that may be used to associate video footage with phase tags, event tags, and / or event characteristics.

[0128] Fig. 6 illustrates an example data structure 600 consistent with the disclosed embodiments. As shown in Fig. 6, data structure 600 may comprise a table including video footage 610 and video footage 620 pertaining to different surgical procedures. For example, video footage 610 may include footage of a laparoscopic cholecystectomy, while video footage 620 may include footage of a cataract surgery. Video footage 620 may be associated with footage location 621, which may correspond to a particular surgical phase of the cataract surgery. Phase tag 622 may identify the phase (in this instance a corneal incision) associated with footage location 621, as discussed above. Video footage 620 may also be associated with event tag 624, which may identify an intraoperative surgical event (in this instance an incision) within the surgical phase occurring at event location 623. Video footage 620 may further be associated with event characteristic 625, which may describe one or more characteristics of the intraoperative surgical event, such as surgeon skill level, as described in detail above. Each video footage identified in the data structure may be associated with more than one footage location, phase tag, event location, event tag and / or event characteristic. For example, video footage 610 may be associated with phase tags corresponding to more than one surgical phase (e.g., "Calot's triangle dissection" and "cutting of cystic duct"). Further, each surgical phase of a particular video footage may be associated with more than one event, and accordingly may be associated with more than one event location, event tag, and / or event characteristic. It is understood, however, that in some embodiments, a particular video footage may be associated with a single surgical phase and / or event. It is also understood that in some embodiments, an event may be associated with any number of event characteristics, including no event characteristics, a single event characteristic, two event characteristics, more than two event characteristics, and so forth. Some non-limiting examples of such event characteristics may include skill level associated with the event (such as minimal skill level required, skill level demonstrated, skill level of a medical care giver involved in the event, etc.), time associated with the event (such as start time, end time, etc.), type of the event, information related to medical instruments involved in the event, information related to anatomical structures involved in the event, information related to medical outcome associated with the event, one or more amounts (such as an amount of leak, amount of medication, amount of fluids, etc.), one or more dimensions (such as dimensions of anatomical structures, dimensions of incision, etc.), and so forth. Further, it is to be understood that data structure 600 is provided by way of example and various other data structures may be used.

[0129] Embodiments of the present disclosure may further include enabling a user to access the data structure through selection of a selected phase tag, a selected event tag, and a selected event characteristic of video footage for display. The user may be any individual or entity that may be provided access to data stored in the data structure. In some embodiments, the user may be a surgeon or other healthcare professional. For example, a surgeon may access the data structure and / or video footage associated with the data structure for review or training purposes. In some embodiments, the user may be an administrator, such as a hospital administrator, a manager, a lead surgeon, or other individual that may require access to video footage. In some embodiments the user may be a patient, who may be provided access to video footage of his or her surgery. Similarly, the user may be a relative, a guardian, a primary care physician, an insurance agent, or another representative of the patient. The user may include various other entities, which may include, but are not limited to, an insurance company, a regulatory authority, a police or investigative authority, a medical association, or any other entity that may be provided access to video footage. Selection by the user may include any means for identifying a particular phase tag, event tag, and / or event characteristic. In some embodiments, selection by the user may occur through a graphical user interface, such as on a display of a computing device. In another example, the selection by the user may occur through a touch screen. In an additional example, the selection by the user may occur through voice input, and the voice input may be processed using a speech recognition algorithm. In yet another example, the selection by the user may occur through gestures (such as hand gestures), and the gestures may be analyzed using gesture recognition algorithms. In some embodiments, the user may not select all three of the selected phase tag, the selected event tag, or the selected event characteristic, but may select a subset of these. For example, the user may just select an event characteristic and the user may be allowed access to information associated with the data structure based on the selected event characteristic.

[0130] Fig. 7 is an illustration of exemplary user interface 700 for selecting indexed video footage for display consistent with the disclosed embodiments. User interface 700 may include one or more search boxes 710, 720, and 730 for selecting video footage. Search box 710 may allow the user to select one or more surgical phases to be displayed. In some embodiments, user interface 700 may provide suggested surgical phases based on the phase tags include in data structure 600. For example, as a user starts typing in search box 710, user interface 700 may suggest phase tag descriptions to search for based on the characters the user has entered. In other embodiments, the user may select the phase tag using radio buttons, checkboxes, a dropdown list, touch interface, or any other suitable user interface feature. Similar to with the phase tags, a user may select video footage based on event tags and event characteristics using search boxes 720 and 730, respectively. User interface 700 may also include dropdown buttons 722 and 732 to access dropdown lists and further filter the results. As shown in Fig. 7, selecting dropdown button 732 may allow the user to select an event characteristic based on subcategories of event characteristics. For example, a user may select "Surgeon skill level" in the dropdown list associated with dropdown button 732, which may allow the user to search based on a skill level of the surgeon in search box 730. While "Surgeon skill level," and various other event characteristic subcategories are provided by way of example, it is understood that a user may select any characteristic or property of the surgical procedure. For example, the user may refine the surgeon skill level based on the surgeon, qualifications, years of experience, and / or any indications of surgical skill level, as discussed in greater detail below. A user may be enabled to access the data structure by clicking, tapping, or otherwise selecting search button 740.

[0131] Display of video footage may include any process by which one or more frames of video footage or a portion thereof are presented to the user. In some embodiments, displaying may include electronically transmitting at least a portion of the video footage for viewing by the user. For example, displaying the video footage may comprise transmitting at least a portion of the video footage over a network. In other embodiments, displaying the video footage may include making the video footage available to the user by storing the video footage in a location accessible to the user or a device being used by the user. In some embodiments, displaying the video footage may comprise causing the video footage to be played on a visual display device, such as a computer or video screen. For example, displaying may include sequentially presenting frames associated with the video footage and may further include presenting audio associated with the video footage.

[0132] Some aspects of the present disclosure may include performing a lookup in the data structure of surgical video footage matching the at least one selected phase tag, selected event tag, and selected event characteristic to identify a matching subset of stored video footage. Performing the lookup may include any process for retrieving data from a data structure. For example, based on the at least one selected phase tag, event tag, and selected event characteristic, a corresponding video footage or portion of video footage may be identified from the data structure. A subset of stored video footage may include a single identified video footage or multiple identified video footages associated with selections of the user. For example, the subset of stored video footage may include surgical video footage having the at least one of a phase tag exactly identical to the selected phase tag, event tag exactly identical to the selected event tag, and event characteristic exactly identical to the selected event characteristic. In another example, the subset of stored video footage may include surgical video footage having the at least one of a phase tag similar (e.g., according to a selected similarity measure) to the selected phase tag, an event tag similar (e.g., according to a selected similarity measure) to the selected event tag, and / or an event characteristic similar (e.g., according to a selected similarity measure) to the selected event characteristic. In some embodiments, performing the lookup may be triggered by selection of search button 740, as shown in Fig. 7.

[0133] In some exemplary embodiments, identifying a matching subset of stored video footage includes using computer analysis to determine a degree of similarity between the matching subset of stored video and the selected event characteristic. Accordingly, "matching" may refer to an exact match or may refer to an approximate or closest match. In one example, the event characteristic may comprise a numerical value (such as an amount, a dimension, a length, an area, a volume, etc., for example as described above), and the degree of similarity may be based on a comparison of a numerical value included in the selected event characteristic and a corresponding numerical value of a stored video. In one example, any similarity function (including but not limited to affinity functions, correlation functions, polynomial similarity functions, exponential similarity functions, similarity functions based on distance, linear functions, non-linear functions, and so forth) may be used to calculate the degree of similarity. In one example, graph matching algorithms or hypergraph matching algorithms (such as exact matching algorithms, inexact matching algorithms) may be used to determine the degree of similarity. As another illustrative example, video footage associated with a "preparation" phase tag may also be retrieved for phase tags including terms "prep," "preparing," "preparatory," "pre-procedure," or other similar but not exact matches that may refer to a "preparation" phase tag. The degree of similarity may refer to any measure of how closely the subset of stored video matches the selected event characteristic. The degree of similarity may be expressed as a similarity ranking (e.g., on scale of 1-10, 1-100, etc.), as a percentage match, or through any other means of expressing how closely there is a match. Using computer analysis may include using a computer algorithm to determine a degree of similarity between the selected event characteristic and the event characteristic of one or more surgical procedures included in the data structure. In one example, k-Nearest-Neighbors algorithms may be used to identify the most similar entries in the data structure. In one example, the entries of the data structures, as well as the user inputted event characteristics, may be embedded in a mathematical space (for example, using any dimensionality reduction or data embedding algorithms), distance between the embedding of an entry and the user inputted characteristics may be used to calculate the degree of similarity between the two. Further, in some examples, the entries nearest to the user inputted characteristics in the embedded mathematical space may be selected as the most similar entries to the user inputted data in the data structure.

[0134] Some aspects of the invention may involve causing the matching subset of stored video footage to be displayed to the user, to thereby enable the user to view surgical footage of at least one intraoperative surgical event sharing the selected event characteristic, while omitting playback of video footage lacking the selected event characteristic. Surgical footage may refer to any video or video footage, as described in greater detail above, capturing a surgical procedure. In some embodiments, causing the matching subset of stored video footage to be displayed may comprise executing instructions for playing the video. For example, a processing device performing the methods described herein may access the matching subset of video footage and may be configured to present the stored video footage to the user on a screen or other display. For example, the stored video footage may be displayed in a video player user interface, such as in video playback region 410, as discussed in further detail below with respect to Fig. 4. In some embodiments, causing the matching subset of stored video footage to be displayed to the user may include transmitting the stored video footage for display, as described above. For example, the matching subset of video footage may be transmitted through a network to a computing device associated with the user, such as a desktop computer, a laptop computer, a mobile phone, a tablet, smart glasses, heads up display, a training device, or any other device capable of displaying video footage.

[0135] Omitting playback may include any process resulting in the video lacking the selected event characteristic from being presented to the user. For example, omitting playback may include designating footage as not to be displayed and not displaying that footage. In embodiments where the matching subset of video footage is transmitted, omitting playback may include preventing transmission of video footage lacking the selected event characteristic. This may occur by selectively transmitting only those portions of footage related to the matching subset; by selectively transmitting markers associated with portions of footage related to the matching subset; and / or by skipping over portions of footage unrelated to the matching subset. In other embodiments, the video footage lacking the selected event characteristic may be transmitted but may be associated with one or more instructions not to present the video footage lacking the selected event characteristic.

[0136] According to various exemplary embodiments of the present disclosure, enabling the user to view surgical footage of at least one intraoperative surgical event that has the selected event characteristic, while omitting playback of portions of selected surgical events lacking the selected event characteristic, may include sequentially presenting to the user portions of surgical footage of a plurality of intraoperative surgical events sharing the selected event characteristic, while omitting playback of portions of selected surgical events lacking the selected event characteristic. In other words, one or more portions of video footage may be identified, for example through a lookup function in the data structure, as being associated with the selected event characteristic. Enabling the user to view surgical footage of the at least one intraoperative surgical event that has the selected event characteristic may include sequentially presenting one or more of the identified portions to the user. Any portions of video footage that are not identified may not be presented. In some embodiments, video footage may be selected based on the selected event tag and the selected phase tag. Accordingly, in embodiments consistent with the present disclosure, enabling the user to view surgical footage of at least one intraoperative surgical event that has the selected event characteristic, while omitting playback of portions of selected surgical events lacking the selected event characteristic, may include sequentially presenting to the user portions of surgical footage of a plurality of intraoperative surgical events sharing the selected event characteristic and associated with the selected event tag and the selected phase tag, while omitting playback of portions of selected surgical events lacking the selected event characteristic or not associated the at least one of selected event tag and the selected phase tag.

[0137] As mentioned above, the stored event characteristic may include a wide variety of characteristics relating to a surgical procedure. In some example embodiments, the stored event characteristic may include an adverse outcome of the surgical event. For example, the stored event characteristic may identify whether the event is an adverse event, or whether it was associated with a complication, including the examples described in greater detail above. Accordingly, causing the matching subset to be displayed may include enabling the user to view surgical footage of a selected adverse outcome while omitting playback of surgical events lacking the selected adverse outcome. By way of example, in response to a user's desire to see how a surgeon dealt with a vascular injury during a laparoscopic procedure, rather than displaying to the user the entire procedure, the user might select the vascular injury event, after which the system might display only a portion of the video footage where the event occurred. The stored event characteristic may similarly identify outcomes, including desired and / or expected outcomes. Examples of such outcomes may include full recovery by the patient, whether a leak occurred, an amount of leak that occurred, whether the amount of leak was within a selected range, whether the patient was readmitted after discharge, a length of hospitalization after surgery, or any other outcomes that may be associated with the surgical procedure. In this way, a user may be able to ascertain at the time of viewing, the long-term impact of a particular technique. Accordingly, in some embodiments, the stored event characteristic may include these or other outcomes, and causing the matching subset to be displayed may include enabling the user to view surgical footage of the selected outcome while omitting playback of surgical events lacking the selected outcome.

[0138] In some embodiments, the stored event characteristic may include a surgical technique. Accordingly, the stored event characteristic may identify whether a particular technique is performed. For example, there may be multiple techniques that may be applied at a particular stage of surgery and the event characteristic may identify which technique is being applied. In this way, a user interested in learning a particular technique might be able to filter video results so that only procedures using the specified technique are displayed. Causing the matching subset to be displayed may include enabling the user to view surgical footage of a selected surgical technique while omitting playback of surgical footage not associated with the selected surgical technique. For example, the user may be enabled to view in sequence, non-sequential portions of video captured from either the same surgery or from different surgeries. In some embodiments, the stored event characteristic may include an identity of a specific surgeon. For example, the event characteristic may include an identity of a particular surgeon performing the surgical procedure. The surgeon may be identified based on his or her name, an identification number (e.g., employee number, medical registration number, etc.) or any other form of identity. In some embodiments, the surgeon may be identified based on recognizing representations of the surgeon in the captured video. For example, various facial and / or voice recognition techniques may be used, as discussed above. In this way, if a user wishes to study a technique of a particular surgeon, the user may be enabled to do so. For example, causing the matching subset to be displayed may include enabling the user to view footage exhibiting an activity by a selected surgeon while omitting playback of footage lacking activity by the selected surgeon. Thus for example, if multiple surgeons participate in the same surgical procedure, a user may choose to view only the activities of a subset of the team.

[0139] In some embodiments, the event characteristic may also be associated with other healthcare providers or healthcare professionals who may be involved in the surgery. In some examples, a characteristic associated with a healthcare provider may include any characteristic of a healthcare provider involved in the surgical procedure. Some non-limiting examples of such healthcare providers may include the title of any member of the surgical team, such as surgeons, anesthesiologists, nurses, Certified Registered Nurse Anesthetist (CRNA), surgical tech, residents, medical students, physician assistants, and so forth. Additional non-limiting examples of such characteristics may include certification, level of experience (such as years of experience, past experience in similar surgical procedures, past success rate in similar surgical procedures, etc.), demographic characteristics (such as age), and so forth.

[0140] In other embodiments, the stored event characteristic may include a time associated with the particular surgical procedure, surgical phase, or portion thereof. For example, the stored event characteristic may include a duration of the event. Causing the matching subset to be displayed may include enabling the user to view footage exhibiting events of selected durations while omitting playback of footage of events of different durations. In this way, for example, a user who might wish to view a particular procedure completed more quickly than the norm, might set a time threshold to view specified procedures completed within that threshold. In another example, a user who might wish to view more complex events may set a time threshold to view procedures including events lasting longer than a selected threshold, or the procedures including events that lasted the longest of a selected group of events. In other embodiments, the stored event characteristic may include a starting time of the event, an ending time of the event, or any other time indicators. Causing the matching subset to be displayed may include enabling the user to view footage exhibiting events from selected times within the particular surgical procedure, within the phase associated with the event, or within the selected portion of the particular surgical procedure, while omitting playback of footage of events associated with different times.

[0141] In another example, the stored event characteristic may include a patient characteristic. The term "patient characteristic" refers to any physical, sociological, economical, demographical or behavioral characteristics of the patient, and to characteristics of the medical history of the patient. Some non-limiting examples of such patient characteristics may include age, gender, weight, height, Body Mass Index (BMI), menopausal status, typical blood pressure, characteristics of the patient genome, educational status, level of education, socio-economic status, level of income, occupation, type of insurance, health status, self-rated health, functional status, functional impairment, duration of disease, severity of disease, number of illnesses, illness characteristics (such as type of illness, size of tumor, histology grade, number of infiltrated lymph nodes, etc.), utilization of health care, number of medical care visits, medical care visit intervals, regular source of medical care, family situation, marital status, number of children, family support, ethnicity, race, acculturation, religious, type of religion, native language, characteristics of past medical test performed on the patient (such as type of test, time of test, results of test, etc.), characteristics of past medical treatments performed on the patient (such as type of treatment, time of treatment, results of treatment, etc.), and so forth. Some non-limiting examples of such medical tests may include blood tests, urine tests, stool tests, medical imaging (such as ultrasonography, angiography, Magnetic Resonance Imaging (MRI), Computed Tomography (CT), X-ray, electromyography, Positron Emission Tomography (PET), etc.), physical examination, electrocardiography, amniocentesis, pap test, skin allergy tests, endoscopy, biopsy, pathology, blood pressure measurements, oxygen saturation test, pulmonary function test, and so forth. Some non-limiting examples of such medical treatments may include medication, dietary treatment, surgery, radiotherapy, chemotherapy, physical therapy, psychological therapy, blood transfusion, infusion, and so forth. Accordingly, causing the matching subset to be displayed may include enabling the user to view footage of patients exhibiting a selected patient characteristic while omitting playback of footage of patients lacking the selected patient characteristic.

[0142] In some embodiments, the selected physical patient characteristic may include a type of anatomical structure. As used herein, an anatomical structure may be any particular part of a living organism. For example, an anatomical structure may include any particular organ, tissue, cell, or other structures of the patient. In this way, if for example, a user wishes to observe video relating to surgery on a pleura sack in a lung, that portion of footage may be presented while other non-related portions may be omitted. The stored event characteristic may include various other patient characteristics, such as the patient's demographics, medical condition, medical history, previous treatments, or any other relevant patient descriptor. This can enable a viewer to view surgical procedures on patients matching very particular characteristics (e.g., 70-75 year old Caucasian, with coronary heart disease who previously had bypass surgery. In this way, video of one or more patients matching those specific criteria might be selectively presented to the user.

[0143] In yet another example, the stored event characteristic may include a physiological response. As used herein, the term "physiological response" refers to any physiological change that may have occurred in reaction to an event within a surgical procedure. Some non-limiting examples of such physiological changes may include change in blood pressure, change in oxygen saturation, change in pulmonary functions, change in respiration rate, change in blood composition (count chemistry, etc.), bleeding, leakage, change in blood flow to a tissue, changing in a condition of a tissue (such as change in color, shape, structural condition, functional condition, etc.), change in body temperature, a change in brain activity, a change in perspiration, or any other physical change in response to the surgical procedure. In this way, a user might be able to prepare for eventualities that might occur during a surgical procedure by selectively viewing those eventualities (and omitting playback of non-matching eventualities.).

[0144] In some examples, the event characteristic may include a surgeon skill level. The skill level may include any indication of the surgeon's relative abilities. In some embodiments, the skill level may include a score reflecting the surgeon's experience or proficiency in performing the surgical procedure or specific techniques within the surgical procedure. In this way, a user can compare, by selecting different skill levels how surgeons of varying experience handle the same procedure. In some embodiments the skill level may be determined based on the identity of a surgeon, either determined via data entry (manually inputting the surgeon's ID) or by machine vision. For example, the disclosed methods may include analysis of the video footage to determine an identity of the surgeon through biometric analysis (e.g., face, voice, etc.) and identify a predetermined skill level associated with that surgeon. The predetermined skill level may be obtained by accessing a database storing skill levels associated with particular surgeons. The skill level may be based on past performances of the surgeon, a type and / or level of training or education of the surgeon, a number of surgeries the surgeon has performed, types of surgeries surgeon has performed, qualifications of the surgeon, a level of experience of the surgeon, ratings of the surgeon from patients or other healthcare professionals, past surgical outcomes, past surgical outcomes and complications, or any other information relevant to assessing the skill level of a healthcare professional. In some embodiments, the skill level may be determined automatically based on computer analysis of the video footage. For example, the disclosed embodiments, may include analyzing video footage capturing performance of a procedure, performance of a particular technique, a decision made by the surgeon, or similar events. The skill level of the surgeon may then be determined based on how well the surgeon performs during the event, which may be based on timeliness, effectiveness, adherence to a preferred technique, the lack of injury or adverse effects, or any other indicator of skill that may be gleaned from analyzing the footage.

[0145] In some embodiments, the skill level may be a global skill level assigned to each surgeon or may be in reference to specific events. For example, a surgeon may have a first skill level with regard to a first technique or procedure and may have a second skill level with regard to a different technique or procedure. The skill level of the surgeon may also vary throughout an event, technique and / or procedure. For example, a surgeon may act at a first skill level within a first portion of the footage but may act at a second skill level at a second portion of the footage. Accordingly, the skill level may be a skill level associated with a particular location of the footage. The skill level also may be a plurality of skill levels during an event or may be an aggregation of the plurality of skill levels during the event, such as an average value, a rolling average, or other forms of aggregation. In some embodiments, the skill level may be a general required skill level for performing the surgical procedure, the surgical phase, and / or the intraoperative surgical event and may not be tied to a particular surgeon or other healthcare professional. The skill level may be expressed in various ways, including as a numerical scale (e.g., 1-10, 1-100, etc.), as a percentage, as a scale of text-based indicators (e.g., "highly skilled," "moderately skilled," "unskilled," etc.) or any other suitable format for expressing the skill of a surgeon. While the skill level is described herein as the skill level of a surgeon, in some embodiments the skill level may be associated with another healthcare professional, such as a surgical technician, a nurse, a physician's assistant, an anesthesiologist, a doctor, or any other healthcare professional.

[0146] Embodiments of the present disclosure may further include accessing aggregate data related to a plurality of surgical procedures similar to the particular surgical procedure. Aggregate data may refer to data collected and / or combined from multiple sources. The aggregate data may be compiled from multiple surgical procedures having some relation to the particular surgical procedure. For example, a surgical procedure may be considered similar to the particular surgical procedure if it includes the same or similar surgical phases, includes the same or similar intraoperative events, or is associated with the same or similar tags or properties (e.g., event tags, phase tags, event characteristics, or other tags.).

[0147] The present disclosure may further include presenting to the user statistical information associated with the selected event characteristic. Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information. A person of ordinary skill in the art would appreciate other forms of statistical information that may be presented according to the disclosed embodiments.

[0148] Figs. 8A and 8B are flowcharts illustrating an example process 800 for video indexing consistent with the disclosed embodiments. Process 800 may be performed by a processing device, such as at least one processor. For example, the at least one processor may include one or more integrated circuits (IC), including application-specific integrated circuit (ASIC), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), server, virtual server, or other circuits suitable for executing instructions or performing logic operations. The instructions executed by at least one processor may, for example, be pre-loaded into a memory integrated with or embedded into the controller or may be stored in a separate memory. The memory may include a Random Access Memory (RAM), a Read-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions. In some embodiments, the at least one processor may include more than one processor. Each processor may have a similar construction or the processors may be of differing constructions that are electrically connected or disconnected from each other. For example, the processors may be separate circuits or integrated in a single circuit. When more than one processor is used, the processors may be configured to operate independently or collaboratively. The processors may be coupled electrically, magnetically, optically, acoustically, mechanically or by other means that permit them to interact.

[0149] In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process 800. At step 802, process 800 may include accessing video footage to be indexed, the video footage to be indexed including footage of a particular surgical procedure. The video footage may be accessed from a local memory, such as a local hard drive, or may be accessed from a remote source, for example, through a network connection. In another example, the video footage may be captured using one or more image sensors, or generated by another process. At step 804, process 800 may include analyzing the video footage to identify a video footage location associated with a surgical phase of the particular surgical procedure. As discussed above, the location may be associated with a particular frame, a range of frames, a time index, a time range, or any other location identifier.

[0150] Process 800 may include generating a phase tag associated with the surgical phase, as shown in step 806. This may occur, for example, through video content analysis (VCA), using techniques such as one or more of video motion detection, video tracking, shape recognition, object detection, fluid flow detection, equipment identification, behavior analysis, or other forms of computer aided situational awareness. When learned characteristics associated with a phase are identified in the video, a tag may be generated demarcating that phase. The tag may include, for example, a predefined name for the phase. At step 808, process 800 may include associating the phase tag with the video footage location. The phase tag may indicate, for example, that the identified video footage location is associated with the surgical phase of the particular surgical procedure. At step 810, process 800 may include analyzing the video footage using one or more of the VCA techniques described above, to identify an event location of a particular intraoperative surgical event within the surgical phase. Process 800 may include associating an event tag with the event location of the particular intraoperative surgical event, as shown at step 812. The event tag may indicate, for example, that the video footage is associated with the surgical event at the event location. As with the phase tag, the event tag may include a predefined name for the event. At step 814, in Fig. 8B, process 800 may include storing an event characteristic associated with the particular intraoperative surgical event. As discussed in greater detail above, the event characteristic may include an adverse outcome of the surgical event, a surgical technique, a surgeon skill level, a patient characteristic, an identity of a specific surgeon, a physiological response, a duration of the event, or any other characteristic or property associated with the event. The event characteristic may be manually determined (for example, inputted by a viewer), or may be determined automatically through artificial intelligence applied to machine vision, for example as described above. In one example, the event characteristic may include skill level (such as minimal skill level required, skill level demonstrated during the event, etc.), a machine learning model may be trained using training example to determine such skill levels from videos, and the trained machine learning model may be used to analyze the video footage to determine the skill level. An example of such training example may include a video clip depicting an event together with a label indicating the corresponding skill level. In another example, the event characteristic may include time related characteristics of the event (such as start time, end time, duration, etc.), and such time related characteristics may be calculated by analyzing the interval in the video footage corresponding to the event. In yet another example, the event characteristic may include an event type, a machine learning model may be trained using training examples to determine event types from videos, and the trained machine learning model may be used to analyze the video footage and determine the event type. An example of such training example may include a video clip depicting an event together with a label indicating the event type. In an additional example, the event characteristic may include information related to a medical instrument involved in the event (such as type of medical instrument, usage of the medical instrument, etc.), a machine learning model may be trained using training examples to identify such information related to medical instruments from videos, and the trained machine learning model may be used to analyze the video footage and determine the information related to a medical instrument involved in the event. An example of such training example may include video clip depicting an event including a usage of a medical instrument, together with a label indicative of the information related to the medical instrument. In yet another example, the event characteristic may include information related to an anatomical structure involved in the event (such as type of the anatomical structure, condition of the anatomical structure, change occurred to the anatomical structure in relation to the event, etc.), a machine learning model may be trained using training example to identify such information related to anatomical structures from videos, and the trained machine learning model may be used to analyze the video footage and determine the information related to the anatomical structure involved in the event. An example of such training example may include a video clip depicting an event involving an anatomical structure, together with a label indicative of information related to the anatomical structure. In an additional example, the event characteristic may include information related to a medical outcome associated with the event, a machine learning model may be trained using training example to identify such information related to medical outcomes from videos, and the trained machine learning model may be used to analyze the video footage and determine the information related to the medical outcome associated with the event. An example of such training example may include a video clip depicting a medical outcome, together with a label indicative of the medical outcome.

[0151] At step 816, process 800 may include associating at least a portion of the video footage of the particular surgical procedure with at least one of the phase tag, the event tag, and the event characteristic in a data structure. In this step, the various tags are associated with the video footage to permit the tags to be used to access the footage. As previously described, various data structures may be used to store related data in an associated manner.

[0152] At step 818, process 800 may include enabling a user to access the data structure through selection of at least one of a selected phase tag, a selected event tag, and a selected event characteristic of video footage for display. In some embodiments, the user may select the selected phase tag, selected event tag, and selected event characteristic through a user interface of a computing device, such as user interface 700 shown in Fig. 7. For example, data entry fields, drop down menus, icons, or other selectable items may be provided to enable a user to select a surgical procedure, the phase of the procedure, an event within a procedure and a characteristic of the procedure and patient. At step 820, process 800 may include performing a lookup in the data structure of surgical video footage matching the at least one selected phase tag, selected event tag, and selected event characteristic to identify a matching subset of stored video footage. At step 822, process 800 may include causing the matching subset of stored video footage to be displayed to the user, to thereby enable the user to view surgical footage of at least one intraoperative surgical event sharing the selected event characteristic, while omitting playback of video footage lacking the selected event characteristic. Through this filtering, the user may be able to quickly view only those video segments corresponding to the user's interest, while omitting playback of large volumes of video data unrelated to the user's interest.

[0153] When preparing for a surgical procedure, it may be beneficial for a surgeon to review video footage of surgical procedures having similar surgical events. It may be too time consuming, however, for a surgeon to view the entire video or to skip around to find relevant portions of the surgical footage. Therefore, there is a need for unconventional approaches that efficiently and effectively enable a surgeon to view a surgical video summary that aggregates footage of relevant surgical events while omitting other irrelevant footage.

[0154] Aspects of this disclosure may relate to generating surgical summary footage, including methods, systems, devices, and computer readable media. For example, footage of one surgical procedure may be compared with that of previously analyzed procedures to identify and tag relevant intraoperative surgical events. A surgeon may be enabled to watch a summary of a surgery that aggregates the intraoperative surgical events, while omitting much of the other irrelevant footage. For ease of discussion, a method is described below, with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Other aspects of such a method may occur using non-electronic means. In a broadest sense, the method is not limited to particular physical and / or electronic instrumentalities, but rather may be accomplished using many differing instrumentalities.

[0155] Consistent with disclosed embodiments, a method may involve accessing particular surgical footage containing a first group of frames associated with at least one intraoperative surgical event. Surgical footage may refer to any video, group of video frames, or video footage including representations of a surgical procedure. For example, the surgical footage may include one or more video frames captured during a surgical operation. Accessing the surgical footage may include retrieving video from a storage location, such as a memory device. The surgical footage may be accessed from a local memory, such as a local hard drive, or may be accessed from a remote source, for example, through a network connection. As described in greater detail above, video may include any form of recorded visual media including recorded images and / or sound. The video may be stored as a video file such as an Audio Video Interleave (AVI) file, a Flash Video Format (FLV) file, QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), a Windows Media Video (WMV) file, a Material Exchange Format (MXF) file, or any other suitable video file formats. Additionally or alternatively, in some examples accessing particular surgical footage may include capturing the particular surgical footage using one or more image sensors.

[0156] As described above, the intraoperative surgical event may be any event or action that is associated with a surgical procedure or phase. A frame may refer to one of a plurality of still images which compose a video. The first group of frames may include frames that were captured during the interoperative surgical event. For example, the particular surgical footage may depict a surgical procedure performed on a patient and captured by at least one image sensor in an operating room. The image sensors may include, for example, cameras 115, 121, and 123, and / or 125 located in operating room 101. In some embodiments, the at least one image sensor may be at least one of above an operating table in the operating room or within the patient. For example, the image sensor may be located above the patient, or may be located within a surgical cavity, organ, or vasculature of the patient, as described above. The first group of frames may include representations of the intraoperative surgical event, including anatomical structures, surgical tools, healthcare professionals performing the intraoperative surgical event, or other visual representations of the intraoperative surgical event. In some embodiments, however, some or all of the frames may not contain representations of the intraoperative surgical event, but may be otherwise associated with the event (e.g., captured while the event was being performed, etc.).

[0157] Consistent with the present disclosure, the particular surgical footage may contain a second group of frames not associated with surgical activity. For example, surgical procedures may involve extensive periods of downtime, where significant surgical activity is not taking place and where there would be no material reason for review of the footage. Surgical activity may refer to any activities that are performed in relation to a surgical procedure. In some embodiments, surgical activity may broadly refer to any activities associated with the surgical procedure, including preoperative activity, perioperative activity, intraoperative activity, and / or postoperative activity. Accordingly, the second group of frames may include frames not associated with any such activities. In other embodiments, surgical activity may refer to a narrower set of activity, such as physical manipulation of organs or tissues of the patient being performed by the surgeon. Accordingly, the second group of frames may include various activities associated with preparation, providing anesthesia, monitoring vital signs, gathering or preparing surgical tools, discussion between healthcare professionals, or other activities that may not be considered surgical activity.

[0158] In accordance with the present disclosure, the methods may include accessing historical data based on historical surgical footage of prior surgical procedures. Historical data may refer to data of any format that was recorded and / or stored previously. In some embodiments, the historical data may be one or more video files including the historical surgical footage. For example, the historical data may include a series of frames captured during the prior surgical procedures. This historical data is not limited to video files, however. For example, the historical data may include information stored as text representing at least one aspect of the historical surgical footage. For example, the historical data may include a database of information summarizing or otherwise referring to historical surgical footage. In another example, the historical data may include information stored as numerical values representing at least one aspect of the historical surgical footage. In an additional example, the historical data may include statistical information and / or statistical model based on an analysis of the historical surgical footage. In yet another example, the historical data may include a machine learning model trained using training examples, and the training examples may be based on the historical surgical footage. Accessing the historical data may include receiving the historical data through an electronic transmission, retrieving the historical data from storage (e.g., a memory device), or any other process for accessing data. In some embodiments, the historical data may be accessed from the same resource as the particular surgical footage discussed above. In other embodiments, the historical data may be accessed from a separate resource. Additionally or alternatively, accessing the historical data may include generating the historical data, for example by analyzing the historical surgical footage of prior surgical procedures or by analyzing data based on the historical surgical footage of prior surgical procedures.

[0159] In accordance with embodiments of the present disclosure, the historical data may include information that distinguishes portions of surgical footage into frames associated with intraoperative surgical events and frames not associated with surgical activity. The information may distinguish the portions of surgical footage in various ways. For example, in connection with historical surgical footage, frames associated with surgical and non-surgical activity may already have been distinguished. This may have previously occurred, for example, through manual flagging of surgical activity or through training of an artificial intelligence engine to distinguish between surgical and non-surgical activity. The historical information may identify, for example, a set of frames (e.g., using a starting frame number, a number of frames, an end frame number, etc.) of the surgical footage. The information may also include time information, such as a begin timestamp, an end timestamp, a duration, a timestamp range, or other information related to timing of the surgical footage. In one example, the historical data may include various indicators and / or rules that distinguish the surgical activity from non-surgical activity. Some non-limiting examples of such indicators and / or rules are discussed below. In another example, the historical data may include a machine learning model trained to identify portions of videos corresponding to surgical activity and / or portions of videos corresponding to non-surgical activity, for example based on the historical surgical footage.

[0160] Various indicators may be used to distinguish the surgical activity from non-surgical activity -- either manually, semi-manually, of automatically (for example, via machine learning). For example, in some embodiments, the information that distinguishes portions of the historical surgical footage into frames associated with an intraoperative surgical event may include an indicator of at least one of a presence or a movement of a surgical tool. A surgical tool may be any instrument or device that may be used during a surgical procedure, which may include, but is not limited to, cutting instruments (such as scalpels, scissors, saws, etc.), grasping and / or holding instruments (such as Billroth's clamps, hemostatic "mosquito" forceps, atraumatic hemostatic forceps, Deschamp's needle, Hopfner's hemostatic forceps, etc.), retractors (such as Farabef's Cshaped laminar hook, blunt-toothed hook, sharp-toothed hook, grooved probe, tamp forceps, etc.), tissue unifying instruments and / or materials (such as needle holders, surgical needles, staplers, clips, adhesive tapes, mesh, etc.), protective equipment (such as facial and / or respiratory protective equipment, headwear, footwear, gloves, etc.), laparoscopes, endoscopes, patient monitoring devices, and so forth. A video or image analysis algorithm, such as those described above with respect to video indexing, may be used to detect the presence and / or motion of the surgical tool within the footage. In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth. In some examples, signal processing algorithms may be used to analyze calculated measures of motion for various times within the video footage of the surgical procedure to distinguish the surgical activity from non-surgical activity. Some non-limiting examples of such signal processing algorithms may include machine learning based signal processing algorithms trained using training examples to distinguish the surgical activity from non-surgical activity, artificial neural networks (such as recursive neural networks, long short-term memory neural networks, deep neural networks, etc.) configured to distinguish the surgical activity from non-surgical activity, Markov models, Viterbi models, and so forth.

[0161] In some exemplary embodiments, the information that distinguishes portions of the historical surgical footage into frames associated with an intraoperative surgical event may include detected tools and anatomical features in associated frames. For example, the disclosed methods may include using an image and / or video analysis algorithm to detect tools and anatomical features. The tools may include surgical tools, as described above, or other nonsurgical tools. The anatomical features may include anatomical structures (as defined in greater detail above) or other parts of a living organism. The presence of both a surgical tool and an anatomical structure detected in one or more associated frames, may serve as an indicator of surgical activity, since surgical activity typically involves surgical tools interacting with anatomical structures. For example, in response to a detection of a first tool in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first tool in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In another example, in response to a detection of a first anatomical feature in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first anatomical feature in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect interaction between the detected tools and anatomical features, and distinguishing the surgical activity from non-surgical activity may be based on the detected interaction. For example, in response to a detection of a first interaction in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first interaction in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect actions performed by the detected tools, and distinguishing the surgical activity from non-surgical activity may be based on the detected actions. For example, in response to a detection of a first action in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first action in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect changes in the condition of anatomical features, and distinguishing the surgical activity from non-surgical activity may be based on the detected changes. For example, in response to a detection of a first change in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first change in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event.

[0162] Some aspects of the invention may involve distinguishing in the particular surgical footage the first group of frames from the second group of frames based on the information of the historical data. For example, the information may provide context that is useful in determining which frames of the particular surgical footage are associated with intraoperative events and / or surgical activity. In some embodiments, distinguishing in the particular surgical footage the first group of frames from the second group of frames may involve the use of a machine learning algorithm. For example, a machine learning model may be trained to identify intraoperative events and / or surgical activity using training examples based on the information of the historical data.

[0163] In accordance with the present disclosure, the first and second group of frames may be distinguished by analyzing the surgical footage to identify information similar to the information of the historical data. Fig. 9 is a flowchart illustrating an example process 900 for distinguishing the first group of frames from the second group of frames. It is to be understood that process 900 is provided by way of example. A person of ordinary skill would appreciate various other processes for distinguishing the first group of frames from the second group, consistent with this disclosure. At step 910, process 900 may include analyzing the particular surgical footage to detect a medical instrument. A medical instrument may refer to any tool or device used for treatment of a patient, including surgical tools, as described above. In addition to the surgical tools listed above, medical instruments may include, but are not limited to stethoscopes, gauze sponges, catheters, cannulas, defibrillators, needles, trays, lights, thermometers, pipettes or droppers, oxygen masks and tubes, or any other medical utensils. For example, a machine learning model may be trained using training examples to detect medical instruments in images and / or videos, and the trained machine learning model may be used to analyze the particular surgical footage and detect the medical instrument. An example of such training example may include a video and / or an image of a surgical procedure, together with a label indicating the presence of one or more particular medical instruments in the video and / or in the image, or together with a label indicating an absence of particular medical instruments in the video and / or in the image.

[0164] At step 920, process 900 may include analyzing the particular surgical footage to detect an anatomical structure. The anatomical structure may be any organ, part of an organ, or other part of a living organism, as discussed above. One or more video and / or image recognition algorithms, as described above, may be used to detect the medical instrument and / or anatomical structure. For example, a machine learning model may be trained using training examples to detect anatomical structures in images and / or videos, and the trained machine learning model may be used to analyze the particular surgical footage and detect the anatomical structure. An example of such training example may include a video and / or an image of a surgical procedure, together with a label indicating the presence of one or more particular anatomical structures in the video and / or in the image, or together with a label indicating an absence of particular anatomical structures in the video and / or in the image.

[0165] At step 930, process 900 may include analyzing the video to detect a relative movement between the detected medical instrument and the detected anatomical structure. Relative movement may be detected using a motion detection algorithm, for example, based on changes in pixels between frames, optical flow, or other forms of motion detection algorithms. For example, motion detection algorithms may be used to estimate the motion of the medical instrument in the video and to estimate the motion of the anatomical structure in the video, and the estimated motion of the medical instrument may be compared with the estimated motion of the anatomical structure to determine the relative movement. At step 940, process 900 may include distinguishing the first group of frames from the second group of frames based on the relative movement, wherein the first group of frames includes surgical activity frames and the second group of frames includes non surgical activity frames. For example, in response to a first relative movement pattern in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative movement pattern in the group of frames, the group of frames may be identified as not including non surgical activity frames. Accordingly, presenting an aggregate of the first group of frames may thereby enable a surgeon preparing for surgery to omit the non-surgical activity frames during a video review of the abridged presentation. In some embodiments, omitting the non-surgical activity frames may include omitting a majority of frames that capture non-surgical activity. For example, not all frames that capture non-surgical activity may be omitted, such as frames that immediately precede or follow intraoperative surgical events, frames capturing non-surgical activity that provides context to intraoperative surgical events, or any other frames that may be relevant to a user.

[0166] In some exemplary embodiments of the present disclosure, distinguishing the first group of frames from the second group of frames may further be based on a detected relative position between the medical instrument and the anatomical structure. The relative position may refer to a distance between the medical instrument and the anatomical structure, an orientation of the medical instrument relative to the anatomical structure, or the location of the medical instrument relative to the anatomical structure. For example, the relative position may be estimated based on a relative position of the detected medical instrument and anatomical structure within one or more frames of the surgical footage. For example, the relative position may include a distance (for example, in pixels, in real world measurements, etc.), a direction, a vector, and so forth. In one example, object detection algorithms may be used to determine a position of the medical instrument, and to determine a position of the anatomical structure, and the two determined positions may be compared to determine the relative position. In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth. In other embodiments, distinguishing the first group of frames from the second group of frames may further be based on a detected interaction between the medical instrument and the anatomical structure. An interaction may include any action by the medical instrument that may influence the anatomical structure, or vice versa. For example, the interaction may include a contact between the medical instrument and the anatomical structure, an action by the medical instrument on the anatomical structure (such as cutting, clamping, applying pressure, scraping, etc.), a reaction by the anatomical structure (such as a reflex action), or any other form of interaction. For example, a machine learning model may be trained using training examples to detect interactions between medical instruments and anatomical structures from videos, and the trained machine learning model may be used to analyze the video footage and detect the interaction between the medical instrument and the anatomical structure. An example of such training example may include a video clip of a surgical procedure, together with a label indicating the presence of particular interactions between medical instruments and anatomical structures in the video clip, or together with a label indicating the absence of particular interactions between medical instruments and anatomical structures in the video clip.

[0167] Some aspects of the present disclosure may involve, upon request of a user, presenting to the user an aggregate of the first group of frames of the particular surgical footage, while omitting presentation to the user of the second group of frames. The aggregate of the first group of frames may be presented in various forms. In some embodiments, the aggregate of the first group of frames may include a video file. The video file may be a compilation of video clips including the first group of frames. In some embodiments the user may be presented each of the video clips separately, or may be presented a single compiled video. In some embodiments a separate video file may be generated for the aggregate of the first group of frames. In other embodiments, the aggregate of the first group of frames my include instructions for identifying frames to be included for presentation, and frames to be omitted. Execution of the instructions may appear to the user as if a continuous video has been generated. Various other formats may also be used, including presenting the first group of frames as still images.

[0168] Presenting may include any process for delivering the aggregate to the user. In some embodiments, this may include causing the aggregate to be played on a display, such as a computer screen or monitor, a projector, a mobile phone display, a tablet, a smart device, or any device capable of displaying images and / or audio. Presenting may also include transmitting the aggregate of the first group of frames to the user or otherwise making it accessible to the user. For example, the aggregate of the first group of frames may be transmitted through a network to a computing device of the user. As another example, the location of the aggregate of the first group of frames may be shared with the user. The second group of frames may be omitted by not including the second group of frames in the aggregate. For example, if the aggregate is presented as a video, video clips comprising the second group of frames may not be included in the video file. The first group of frames may be presented in any order, including chronological order. In some instances, it may be logical to present at least some of the first group of frames in non-chronological order. In some embodiments, the aggregate of the first group of frames may be associated with more than one intraoperative surgical event. For example, a user may request to view a plurality of intraoperative surgical events in the particular surgical footage. Presenting to the user an aggregate of the first group of frames may include displaying the first group frames in chronological order with chronological frames of the second group omitted.

[0169] The user may be any individual or entity that may require access to surgical summary footage. In some embodiments, the user may be a surgeon or other healthcare professional. For example, a surgeon may request surgical summary footage for review or training purposes. In some embodiments the user may be an administrator, a manager, a lead surgeon, insurance company personnel, a regulatory authority, a police or investigative authority, or any other entity that may require access to surgical footage. Various other examples of users are provided above in reference to video indexing techniques. The user may submit the request through a computer device, such as a laptop, a desktop computer, a mobile phone, a tablet, smart glasses or any other form of computing device capable of submitting requests. In some embodiments, the request may be received electronically through a network and the aggregate may be presented based on receipt of the request.

[0170] In some exemplary embodiments, the request of the user may include an indication of at least one type of intraoperative surgical event of interest and the first group of frames may depict at least one intraoperative surgical event of the at least one type of intraoperative surgical event of interest. The type of the intraoperative surgical event may be any category in which the intraoperative surgical event may be classified. For example, the type may include the type of procedure being performed, the phase of the procedure, whether or not the intraoperative surgical event is adverse, whether the intraoperative surgical event is part of the planned procedure, the identity of a surgeon performing the intraoperative surgical event, a purpose of the intraoperative surgical event, a medical condition associated with the intraoperative surgical event, or any other category or classification.

[0171] Embodiments of the present disclosure may further include exporting the first group of frames for storage in a medical record of the patient. As described above, the particular surgical footage may depict a surgical procedure performed on a patient. Using the disclosed methods, the first group of frames associated with the at least one interoperative surgical event may be associated with the patient's medical record. As used herein, a medical record may include any form of documentation of information relating to a patient's health, including diagnoses, treatment, and / or care. The medical record may be stored in a digital format, such as an electronic medical record (EMR). Exporting the first group of frames may include transmitting or otherwise making the first group of frames available for storage in the medical record or in a manner otherwise associating the first group of frames with the medical record. This may include, for example, transmitting the first group of frames (or copies of the first group of frames) to an external device, such as a database. In some embodiments, the disclosed methods may include associating the first group of frames with a unique patient identifier and updating a medical record including the unique patient identifier. The unique patient identifier may be any indicator, such as an alphanumerical string, that uniquely identifies the patient. The alphanumeric string may anonymize the patient, which may be required for privacy purposes. In instances where privacy may not be an issue, the unique patient identifier may include a name and / or social security number of the patient.

[0172] In some exemplary embodiments, the disclosed methods may further comprise generating an index of the at least one intraoperative surgical event. As described above, an index may refer to a form of data storage that enables retrieval of the associated video frames. Indexing may expedite retrieval in a manner more efficient and / or effective than if not indexed. The index may include a list or other itemization of intraoperative surgical events depicted in or otherwise associated with the first group of frames. Exporting the first group of frames may include generating a compilation of the first group of frames, the compilation including the index and being configured to enable viewing of the at least one intraoperative surgical event based on a selection of one or more index items. For example, by selecting "incision" through the index, the user may be presented with a compilation of surgical footage depicting incisions. Various other intraoperative surgical events may be included on the index. In some embodiments, the compilation may contain a series of frames of differing intraoperative events stored as a continuous video. For example, the user may select multiple intraoperative events through the index, and frames associated with the selected intraoperative events may be compiled into a single video.

[0173] Embodiments of the present disclosure may further include generating a cause effect summary. The cause-effect summary may allow a user to view clips or images associated with a cause phase of a surgical procedure and clips or images of associated outcome phase, without having to view intermediate clips or images. As used herein "cause" refers to trigger or action that gives rise to a particular result, phenomenon or condition. The "outcome" refers to the phenomenon or condition that can be attributed to the cause. In some embodiments, the outcome may be an adverse outcome. For example, the outcome may include a bleed, mesenteric emphysema, injury, conversion to unplanned open surgery (for example, abdominal wall incision), an incision that is significantly larger than planned, and so forth. The cause may an action, such as an error by the surgeon, that results in or can be attributed to the adverse outcome. For example, the error may include a technical error, a communication error, a management error, a judgment error, a decision-making error, an error related to medical equipment utilization, or other forms of errors that may occur. The outcome may also include a positive or expected outcome, such as a successful operation, procedure, or phase.

[0174] In embodiments where a cause-effect summary is generated, the historical data may further include historical surgical outcome data and respective historical cause data. The historical surgical outcome data may indicate portions of the historical surgical footage associated with an outcome and the historical cause data may indicate portions of the historical surgical footage associated with a respective cause of the outcome. In such embodiments, the first group of frames may include a cause set of frames and an outcome set of frames, whereas the second group of frames may include an intermediate set of frames.

[0175] Fig. 10 is a flowchart illustrating an exemplary process 1000 for generating a cause-effect summary, consistent with the disclosed embodiments. Process 1000 is provided by way of example, and a person of ordinary skill would appreciate various other processes for generating a cause-effect summary consistent with this disclosure. At step 1010, process 1000 may include analyzing the particular surgical footage to identify a surgical outcome and a respective cause of the surgical outcome, the identifying being based on the historical outcome data and respective historical cause data. The analysis may be performed using image and / or video processing algorithms, as discussed above. In some embodiments, step 1010 may include using a machine learning model trained to identify surgical outcomes and respective causes of the surgical outcomes using the historical data to analyze the particular surgical footage. For example, the machine learning model may be trained based on historical data with known or predetermined surgical outcomes and respective causes. The trained model may then be used to identify surgical outcomes and respective causes in other footage, such as the particular surgical footage. An example of a training examples used to train such machine learning model may include a video clip of a surgical procedure, together with a label indicating a surgical outcome corresponding to the video clip, and possibly a respective cause of the surgical outcome. Such training example may be based on the historical data, for example including a video clip from the historical data, including an outcome determined based on the historical data, and so forth.

[0176] At step 1020, process 1000 may include detecting, based on the analyzing, the outcome set of frames in the particular surgical footage, the outcome set of frames being within an outcome phase of the surgical procedure. The outcome phase may be a timespan or portion of a surgical procedure that is associated with an outcome as described above. At step 1030, process 1000 may include detecting, based on the analyzing, a cause set of frames in the particular surgical footage, the cause set of frames being within a cause phase of the surgical procedure remote in time from the outcome phase. In some embodiments, the outcome phase may include a surgical phase in which the outcome is observable, and the outcome set of frames may be a subset of frames in the outcome phase. The cause phase may be a timespan or portion of the surgical procedure that is associated with a cause of the outcome in the outcome phase. In some embodiments, the cause phase may include a surgical phase in which the cause occurred, and the cause set of frames may be a subset of the frames in the cause phase. The intermediate set of frames may be within an intermediate phase interposed between the cause set of frames and the outcome set of frames. At step 1040, process 1000 may include generating a cause-effect summary of the surgical footage, wherein the cause-effect summary includes the cause set of frames and the outcome set of frames and omits the intermediate set of frames. In some embodiments, the cause-effect summary may be similar to the aggregate of the first group of frames, as described above. Accordingly, the cause-effect summary may include a compilation of video clips associated with the cause set of frames and outcome set of frames. The aggregate of the first group of frames presented to the user, as described above, may include the cause effect summary.

[0177] Fig. 11 is a flowchart illustrating an example process 1100 for generating surgical summary footage, consistent with the disclosed embodiments. Process 1100 may be performed by a processing device. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process 1100. At step 1110, process 1100 may include accessing particular surgical footage containing a first group of frames associated with at least one intraoperative surgical event and a second group of frames not associated with surgical activity. As discussed in further detail above, the first group of frames may be associated with multiple intraoperative surgical events and may not necessarily be consecutive frames. Further, in some embodiments, the first group of frames may include a cause set of frames and an outcome set of frames, and the second group of frames may include an intermediate set of frames, as discussed above with respect to process 1000.

[0178] At step 1120, process 1100 may include accessing historical data based on historical surgical footage of prior surgical procedures, wherein the historical data includes information that distinguishes portions of surgical footage into frames associated with intraoperative surgical events and frames not associated with surgical activity. In some embodiments, the information that distinguishes portions of the historical surgical footage into frames associated with an intraoperative surgical event may include an indicator of at least one of a presence or a movement of a surgical tool and / or an anatomical feature. At step 1130, process 1100 may include distinguishing in the particular surgical footage the first group of frames from the second group of frames based on the information of the historical data.

[0179] At step 1140, process 1100 may include, upon request of a user, presenting to the user an aggregate of the first group of frames of the particular surgical footage, while omitting presentation to the user of the second group of frames. The request of the user may be received from a computing device which may include a user interface enabling the user to make the request. In some embodiments, the user may further request frames associated with a particular type or category of intraoperative events. Based on the steps described in process 1100, the user may be presented a summary including frames associated with intraoperative events and omitting frames not associated with surgical activity. The summary may be used, for example, by a surgeon as a training video that aggregates the intraoperative surgical events, while omitting much of the other irrelevant footage.

[0180] When preparing for a surgical procedure, it may be beneficial for a surgeon to review video footage of several surgical procedures having similar surgical events. Conventional approaches may not allow a surgeon to easily access video footage of surgical procedures having similar surgical events. Further, even if the footage is accessed, it may be too time consuming to watch the entire video or to find relevant portions of the videos. Therefore, there is a need for unconventional approaches that efficiently and effectively enable a surgeon to view a video compiling footage of surgical events from surgeries performed on different patients.

[0181] Aspects of this disclosure may relate to surgical preparation, including methods, systems, devices, and computer readable media. In particular, a compilation video of differing events in surgeries performed on different patients may be presented to a surgeon or other user. The compilation may include excerpts of surgical video of differing intraoperative events from similar surgical procedures, which may be automatically aggregated in a composite form. A surgeon may be enabled to input case-specific information, to retrieve the compilation of video segments selected from similar surgeries on different patients. The compilation may include one intraoperative event from one surgery and other different intraoperative events from one or more second surgeries. For example, different complications that occur when operating on different patients may all be included in one compilation video. In situations where videos of multiple surgical procedures contain the same event with a shared characteristic (e.g., a similar technique employed), the system may omit footage from one or more surgical procedures to avoid redundancy.

[0182] For ease of discussion, a method is described below, with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Other aspects of such a method may occur using non-electronic means. In a broadest sense, the method is not limited to particular physical and / or electronic instrumentalities, but rather may be accomplished using many differing instrumentalities.

[0183] Consistent with disclosed embodiments, a method may involve accessing a repository of a plurality of sets of surgical video footage. As used herein, a repository may refer to any storage location or set of storage locations where video footage may be stored electronically. For example, the repository may include a memory device, such as a hard drive and / or flash drive. In some embodiments, the repository may be a network location such as a server, a cloud storage location, a shared network drive, or any other form of storage accessible over a network. The repository may include a database of surgical video footage captured at various times and / or locations. In some embodiments, the repository may store additional data besides the surgical video footage.

[0184] As described above, surgical video footage may refer to any video, group of video frames, or video footage including representations of a surgical procedure. For example, the surgical footage may include one or more video frames captured during a surgical operation. A set of surgical video footage may refer to a grouping of one or more surgical videos or surgical video clips. The video footage may be stored in the same location or may be selected from a plurality of storage locations. Although not necessarily so, videos within a set may be related in some way. For example, video footage within a set may include videos, recorded by the same capture device, recorded at the same facility, recorded at the same time or within the same timeframe, depicting surgical procedures performed on the same patient or group of patients, depicting the same or similar surgical procedures, depicting surgical procedures sharing a common characteristic (such as similar complexity level, including similar events, including usages of similar techniques, including usages of similar medical instruments, etc.), or sharing any other properties or characteristics.

[0185] The plurality of sets of surgical video footage may reflect a plurality of surgical procedures performed on differing patients. For example, a number of different individuals who underwent the same or similar surgical procedure, or who underwent surgical procedures where a similar technique was employed may be included within a common set or a plurality of sets. Alternatively or in addition, one or more sets may include surgical footage captured from a single patient but at different times. The plurality of surgical procedures may be of the same type, for example, all including appendectomies, or may be of different types. In some embodiments, the plurality of surgical procedures may share common characteristics, such as the same or similar phases or intraoperative events.

[0186] The plurality of sets of surgical video footage may further include intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics. Examples of such events, outcomes, and characteristics are described throughout the present disclosure. A surgical outcome may include outcomes of the surgical procedure as a whole (e.g., whether the patient recovered or recovered fully, whether patient was readmitted after discharge, whether the surgery was successful), or outcomes of individual phases or events within the surgical procedure (e.g., whether a complication occurred or whether a technique was successful).

[0187] Some aspects of the present disclosure may involve enabling a surgeon preparing for a contemplated surgical procedure to input case-specific information corresponding to the contemplated surgical procedure. A contemplated surgical procedure may include any surgical procedure that has not already been performed. In some embodiments, the surgical procedure may be a planned surgical procedure that the surgeon intends to perform on a patient. In other embodiments the contemplated surgical procedure may be a hypothetical procedure and may not necessarily be associated with a specific patient. In some embodiments, the contemplated surgical procedure may be experimental and may not be in widespread practice. The case-specific information may include any characteristics or properties of the contemplated surgical procedure or of a contemplated or hypothetical patient. For example, the case-specific information may include, but is not limited to, characteristics of the patient the procedure will be performed on, characteristics of the surgeon performing the procedure, characteristics of other healthcare professionals involved in the procedure, the type of procedure being performed, unique details or aspects of the procedure, the type of equipment or tools involved, types of technology involved, complicating factors of the procedure, a location of the procedure, the type of medical condition being treated or certain aspects thereof, a surgical outcome, an intraoperative event outcome, or any other information that may define or describe the contemplated surgical procedure. For example, the case-specific information may include a patient's age, weight, medical condition, vital signs, other physical characteristics, past medical history, family medical history, or any other type of patient-related information that might have some direct or indirect bearing on a potential outcome. The case-specific information may also include an indicator of the performing surgeon's skill level, a surgical technique employed, a complication encountered, or any other information about the surgeon, the procedure, the tools used, or the facility.

[0188] The case-specific information may be input in various ways. In some embodiments, the surgeon may input the case-specific information through a graphical user interface. The user interface may include one or more text fields, prompts, drop-down lists, checkboxes or other fields or mechanisms for inputting the information. In some embodiments, the graphical user interface may be associated with the computing device or processor performing the disclosed methods. In other embodiments, the graphical user interface may be associated with an external computing device, such as a mobile phone, a tablet, a laptop, a desktop computer, a computer terminal, a wearable device (including smart watches, smart glasses, smart jewelry, head-mounted displays, etc.), or any other electronic device capable of receiving a user input. In some embodiments, the case-specific information may be input at an earlier time or over a period of time (e.g., several days, several months, several years, or longer). Some or all of the case-specific information may be extracted from a hospital or other medical facility database, an electronic medical record, or any other location that may store patient data and / or other medical data. In some embodiments, the case-specific information corresponding to the contemplated surgical procedure may be received from an external device. For example, the case-specific information may be retrieved or otherwise received from an external computing device, a server, a cloud-computing service, a network device, or any other device external to the system performing the disclosed methods. In one example, at least part of the case-specific information corresponding to the contemplated surgical procedure may be received from an Electronic Health Record (EMR) or from a system handling the EMR (for example, an EMR of a particular patient the procedure will be performed on, an EMR associated with the contemplated surgical procedure, etc.), from a scheduling system, from electronic records corresponding to a medical professional associated with the contemplated surgical procedure or from a system handling the electronic record, and so forth.

[0189] In some exemplary embodiments, the case-specific information may include a characteristic of a patient associated with the contemplated procedure. For example, as mentioned earlier, the case-specific information may include characteristics of a contemplated patient. Patient characteristics may include, but are not limited to, a patient's gender, age, weight, height, physical fitness, heart rate, blood pressure, temperature, medical condition or disease, medical history, previous treatments, or any other relevant characteristic. Other exemplary patient characteristics are described throughout the present disclosure. In some embodiments, a characteristic of the patient may be entered directly by the surgeon. For example, a patient characteristic may be entered through a graphical user interface, as described above. In other embodiments, the characteristic of the patient may be retrieved from a database or other electronic storage location. In some embodiments, the characteristic of the patient may be received from a medical record of the patient. For example, a patient characteristic may be retrieved from the medical record or other information source based on an identifier or other information input by the surgeon. For example, the surgeon may enter a patient identifier and the medical record of the patient and / or the patient characteristic may be retrieved using the patient identifier. As describe herein, the patient identifier may be anonymous (e.g., an alphanumeric code or machine readable code) or it may identify the patient in a discernable way (e.g., patient name or social security number). In some examples, the case-specific information may include characteristics of two or more patients associated with the contemplated procedure (for example, for contemplated surgical procedures that involves two or more patients, such as transplants)

[0190] In accordance with the present disclosure, the case-specific information may include information relating to a surgical tool. The surgical tool may be any device or instrument used as part of a surgery. Some exemplary surgical tools are described throughout the present disclosure. In some embodiments, the information relating to the surgical tool may include at least one of a tool type or a tool model. A tool type may refer to any classification of the tool. For example, the tool type may refer to the kind of instrument being used (e.g., "scalpel," "scissors," "forceps," "retractor," or other kinds of instruments). Tool type may include various other classifications, such as whether the tool is electronic, whether the tool is used for a minimally invasive surgery, the materials the tool is constructed of, a size of the tool, or any other distinguishing properties. The tool model may refer to the specific make and / or manufacturer of the instrument (e.g., "15921 Halsted Mosquito Forceps").

[0191] Embodiments of the present disclosure may further include comparing the case-specific information with data associated with the plurality of sets of surgical video footage to identify a group of intraoperative events likely to be encountered during the contemplated surgical procedure. Data associated with the plurality of sets of surgical videos may include any stored information regarding the surgical video footage. The data may include information identifying intraoperative surgical events, surgical phases, or surgical event characteristics depicted in or associated with the surgical video footage. The data may include other information such as patient or surgeon characteristics, properties of the video (e.g., capture date, file size, information about the capture device, capture location, etc.) or any other information pertaining to the surgical video footage. The data may be stored as tags or other data within the video files. In other embodiments, the data may be stored in a separate file. In some embodiments the surgical video footage may be indexed to associate the data with the video footage. Accordingly, the data may be stored in a data structure, such as data structure 600, described above. In one example, comparing the case-specific information with data associated one or more surgical video footage (for example, with the plurality of sets of surgical video footage) may include calculating one or more similarity measures between the case-specific information and the data associated one or more surgical video footage, for example using one or more similarity functions. Further, in one example, the calculated similarity measures may be compared with selected threshold to determine if an event that occurred in the one or more surgical video footage is likely to occur in the contemplated surgical procedure, for example using a k-Nearest Neighbors algorithm to predict that events commonly occurring the k most similar surgical video footage are likely to be encountered during the contemplated surgical procedure. In some examples, a machine learning model may be trained using training examples to identify intraoperative events likely to be encountered during specific surgical procedures from information related to the specific surgical procedures, and the trained machine learning model may be used to analyze the case-specific information of the contemplated surgical procedure and identify the group of intraoperative events likely to be encountered during the contemplated surgical procedure. An example of such training example may include information related to a particular surgical procedure, together with a label indicating intraoperative events likely to be encountered during the particular surgical procedure.

[0192] The group of intraoperative events likely to be encountered during the contemplated surgical procedure may be determined based on the data. For example, the case-specific information may be compared to the data associated with the plurality of sets of surgical video footage. This may include comparing characteristics of the contemplated surgical procedure (as represented in the case-specific information) to identify surgical video footage associated with surgical procedures having the same or similar characteristics. For example, if the case-specific information includes a medical condition of a patient associated with the contemplated procedure, sets of surgical video footage associated with surgical procedures on patients with the same or similar medical conditions may be identified. By way of another example, a surgeon preparing to perform a catheterization on a 73 year old male with diabetes, high cholesterol, high blood pressure, and a family history of heart disease, may enter that case-specific information in order to draw video footage for review of patients sharing similar characteristics (or patients predicted to present similarly to the specific patient). The group of intraoperative events likely to be encountered during the contemplated surgical procedure may include intraoperative surgical events that were encountered during the surgical procedures associated with the identified surgical video footage. In some embodiments, multiple factors may be considered in identifying the surgical video footage and / or the group of intraoperative events likely to be encountered.

[0193] Whether an intraoperative event is considered likely to be encountered during the contemplated surgical procedure may depend on how frequently the intraoperative event occurs in surgical procedures similar to the contemplated surgical procedure. For example, the intraoperative event may be identified based on the number of times it occurs in similar procedures, the percentage of times it occurs in similar procedures, or other statistical information based on the plurality of sets of surgical video footage. In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold. For example, an intraoperative event may be identified if it occurs in more than 50% of similar surgical procedures, or any other percentage. In some embodiments, the group of intraoperative events may include tiers of intraoperative events based on their likelihood of occurrence. For example, group may include a tier of intraoperative events with a high likelihood of occurrence and one or more tiers of intraoperative events with a lower likelihood of occurrence.

[0194] In accordance with some embodiments of the present disclosure, machine learning or other artificial intelligence techniques may be used to identify the group of intraoperative events. Accordingly, comparing the case-specific information with data associated with the plurality of sets of surgical video footage may include using an artificial neural network to identify the group of intraoperative events likely to be encountered during the contemplated surgical procedure. In one example, the artificial neural network may be configured manually, may be generated from a combination of two or more other artificial neural networks, and so forth. In one example, the artificial neural network may be fed training data correlating various case-specific information with intraoperative events likely to be encountered. In some embodiments, the training data may include one or more sets of surgical video footage included in the repository and data associated with the surgical footage. The training data may also include non-video related data, such as patient characteristics or past medical history. Using an artificial neural network, a trained model may be generated based on the training data. Accordingly, using the artificial neural network may include providing the case-specific information to the artificial neural network as an input. As an output of the model, the group of intraoperative events likely to be encountered during the contemplated surgical procedure may be identified. Various other machine learning algorithms may be used, including a logistic regression, a linear regression, a regression, a random forest, a K-Nearest Neighbor (KNN) model (for example as described above), a K-Means model, a decision tree, a cox proportional hazards regression model, a Naive Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, or any other form of machine learning model or algorithm.

[0195] Some aspects of the present disclosure may further include using the case-specific information and the identified group of intraoperative events likely to be encountered to identify specific frames in specific sets of the plurality of sets of surgical video footage corresponding to the identified group of intraoperative events. The specific frames in specific sets of the plurality of sets of surgical video footage may be locations in the video footage where the intraoperative events occur. For example, if the group of intraoperative events includes a complication, the specific frames may include video footage depicting the complication or otherwise associated with the complication. In some embodiments, the specific frames may include some surgical video footage before or after occurrence of the intraoperative event, for example, to provide context for the intraoperative event. Further, the specific frames may not necessarily be consecutive. For example, if the intraoperative event is an adverse event or outcome, the specific frames may include frames corresponding to the adverse outcome and a cause of the adverse outcome, which may not be consecutive. The specific frames may be identified based on frame numbers (e.g., a frame number, a beginning frame number and an ending frame number, a beginning frame number and a number of subsequent frames, etc.), based on time information (e.g., a start time and stop time, a duration, etc.), or any other manner for identifying specific frames of video footage.

[0196] In some embodiments, the specific frames may be identified based on indexing of the plurality of surgical video footage. For example, as described above, video footage may be indexed to correlate footage locations to phase tags, event tags, and or event characteristics. Accordingly, identifying the specific frames in specific sets of the plurality of sets of surgical video footage may include performing a lookup or search for the intraoperative events using a data structure, such as data structure 600 as described in relation to Fig. 6.

[0197] In accordance with the present disclosure, the identified specific frames may include frames from the plurality of surgical procedures performed on differing patients. Accordingly, the identified specific frames may form a compilation of footage associated with intraoperative events from surgical procedures performed on different patients, which may be used for surgical preparation. For example, the best video clip examples (in terms of video quality, clarity, representativeness, compatibility with the contemplated surgical procedure, etc.) may be chosen from differing procedures performed on differing patients, and associated with each other so that a preparing surgeon can view the best of a group of video clips, for example without having to separately review video of each case, one by one.

[0198] Embodiments of the present disclosure may further include omitting portions of the identified specific frames, for example, to avoid redundancy, to shorten the resulting compilation, to remove less relevant or less informative portions, and so forth. Accordingly, some embodiments may include determining that a first set and a second set of video footage from differing patients contain frames associated with intraoperative events sharing a common characteristic. The first set and second set of video footage may comprise frames of the identified specific frames corresponding to the identified group of intraoperative events. The common characteristic may be any characteristic of the intraoperative events that is relevant to determining whether frames from the first set and the second set should both be included. The common characteristic may be used to determine whether the first set and the second set are redundant. For example, the intraoperative event may be a complication that occurs during the surgical procedure and the common characteristic may be a type of complication. If the complications in first and seconds sets of frames are of the same type, it may not be efficient or beneficial for a surgeon preparing for surgery to view both the first set and second set of frames. Thus, only one set may be chosen for presentation to the surgeon, with the other set being omitted. In some embodiments of the present disclosure, the common characteristic may include a characteristic of the differing patients. For example, the common characteristic may include a patient's age, weight, height, or other demographics, may include patient condition, and so forth. Various other patient characteristics described throughout the present disclosure may also be shared. In other embodiments, the common characteristic may include an intraoperative surgical event characteristic of the contemplated surgical procedure. The intraoperative surgical event characteristic may include any trait or property of the intraoperative event. For example, an adverse outcome of the surgical event, a surgical technique, a surgeon skill level, an identity of a specific surgeon, a physiological response, duration of the event, or any other characteristic or property associated with the event.

[0199] According to various exemplary embodiments of the present disclosure, determining that a first set and a second set of video footage from differing patients contain frames associated with intraoperative events sharing a common characteristic may include using an implementation of a machine learning model to identify the common characteristic. In one example, a machine learning model may be trained using training examples to identify frames of video footage having particular characteristics, and the trained machine learning model may be used to analyze the first set and the second set of video footage from differing patients to identify the frames associated with intraoperative events sharing a common characteristic. An example of such training example may include a video clip together with a label indicating particular characteristics of particular frames of the video clip. Various machine learning models are described above and may include a logistic regression model, a linear regression model, a regression model, a random forest model, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naive Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, a deep learning model, or any other form of machine learning model or algorithm. Some embodiments of the present disclosure may further include using example video footage to train the machine learning model to determine whether two sets of video footage share the common characteristic, and wherein implementing the machine learning model includes implementing the trained machine learning model. In one example, the example video footage may be training footage, which may include pairs of sets of video footage known to share the common characteristic. The trained machine learning model may be configured to determine whether two sets of video footage share the common characteristic.

[0200] The disclosed embodiments may further include omitting an inclusion of the second set from a compilation to be presented to the surgeon and including the first set in the compilation to be presented to the surgeon. As used herein, a compilation may include a series of frames that may be presented for continuous and / or consecutive playback. In some embodiment, the compilation may be stored as a separate video file. In other embodiments, the compilation may be stored as instructions to present the series of frames from their respective surgical video footage, for example, stored in the repository. The compilation may include additional frames besides those included in the first set, including other frames from the identified specific frames.

[0201] Some aspects of the present disclosure may further include enabling the surgeon to view a presentation including the compilation containing frames from the differing surgical procedures performed on differing patients. The presentation may be any form of visual display including the compilation of frames. In some embodiments the presentation may be a compilation video. The presentation may include other elements, such as menus, controls, indices, timelines, or other content in addition to the compilation. In some embodiments, enabling the surgeon to view the presentation may include outputting data for displaying the presentation using a display device, such as a screen (e.g., an OLED, QLED LCD, plasma, CRT, DLPT, electronic paper, or similar display technology), a light projector (e.g., a movie projector, a slide projector), a 3D display, smart glasses, or any other visual presentation mechanism, with or without audio presentation. In other embodiments, enabling the surgeon to view the presentation may include storing the presentation in a location that is accessible by one or more other computing devices. Such storage locations may include a local storage (such as a hard drive of flash memory), a network location (such as a server or database), a cloud computing platform, or any other accessible storage location. Accordingly, the presentation may be accessed from an external device to be displayed on the external device. In some embodiments, outputting the video may include transmitting the video to an external device. For example, enabling the surgeon to view the presentation may include transmitting the presentation through a network to a user device or other external device for playback on the external device.

[0202] The presentation may stitch together disparate clips from differing procedures, presenting them to the surgeon in the chronological order in which they might occur during surgery. The clips may be presented to play continuously, or may be presented in a manner requiring the surgeon to affirmatively act in order for a succeeding clip to play. In some instances where it may be beneficial for the surgeon to view multiple alternative techniques or to view differing responses to adverse events, multiple alternative clips from differing surgical procedures may be presented sequentially.

[0203] Some embodiments of the present disclosure may further include enabling a display of a common surgical timeline including one or more chronological markers corresponding to one or more of the identified specific frames along the presentation. For example, the common surgical timeline may be overlaid on the presentation, as discussed above. An example surgical timeline 420 including chronological markers is shown in Fig. 4. The chronological markers may correspond to markers 432, 434, and / or 436. Accordingly, the chronological markers may correspond to a surgical phase, an intraoperative surgical event, a decision making junction, or other notable occurrences the identified specific frames along the presentation. The markers may be represented as shapes, icons, or other graphical representations along the timeline, as described in further detail above. The timeline may be presented together with frames associated with a surgery performed on a single patient, or may be presented together with a compilation of video clips from surgeries performed on a plurality of patients.

[0204] In accordance with some embodiments of the present disclosure, enabling the surgeon to view the presentation may include sequentially displaying discrete sets of video footage of the differing surgical procedures performed on differing patients. Each discrete set of video footage may correspond to a different surgical procedure performed on a different patient. In some embodiments, sequentially displaying the discrete sets of video footage may appear to the surgeon or another user as a continuous video. In other embodiments playback may stop or pause between the discrete sets of video footage. The surgeon or other user may manually start the next set of video footage in the sequence.

[0205] In accordance with some embodiments of the present disclosure, the presentation may include a display of a simulated surgical procedure based on the identified group of intraoperative events likely to be encountered and / or the identified specific frames in specific sets of the plurality of sets of surgical video footage corresponding to the identified group of intraoperative events. For example, a machine learning algorithm (such as a Generative Adversarial Network) may be used to train a machine learning model (such as an artificial neural network, a deep learning model, a convolutional neural network, etc.) using training examples to generate simulations of surgical procedures based on groups of intraoperative events and / or frames of surgical video footage, and the trained machine learning model may be used to analyze the identified group of intraoperative events likely to be encountered and / or the identified specific frames in specific sets of the plurality of sets of surgical video footage corresponding to the identified group of intraoperative events and generate the simulated surgical procedure.

[0206] In some embodiments, sequentially displaying discrete sets of video footage may include displaying an index of the discrete sets of video footage enabling the surgeon or other user to select one or more of the discrete sets of video footage. The index may be a text-based index, for example, listing intraoperative events, surgical phases, or other indicators of the different discrete sets of video footage. In other embodiments, the index may be a graphical display, such as a timeline as described above, or a combination of graphical and textual information. For example, the index may include a timeline parsing the discrete sets into corresponding surgical phases and textual phase indicators. In such embodiments, the discrete sets may correspond to different surgical phases of the surgical procedure. The discrete sets may be displayed using different colors, with different shading, with bounding boxes or separators, or other visual indicators to distinguish the discrete sets. The textual phase indicators may describe or otherwise identify the corresponding surgical phase. The textual phase indicators may be displayed within the timeline, above the timeline, below the timeline or in any location such that they identify the discrete sets. In some embodiments, the timeline may be displayed in a list format and the textual phase indicators may be included within the list.

[0207] In accordance with the present disclosure, the timeline may include an intraoperative surgical event marker corresponding to an intraoperative surgical event. The intraoperative surgical event marker may correspond to an intraoperative surgical event associated with a location in the surgical video footage. The surgeon may be enabled to click on the intraoperative surgical event marker to display at least one frame depicting the corresponding intraoperative surgical event. For example, clicking on the intraoperative surgical event may cause a display of the compilation video to skip to a location associated with the selected marker. In some embodiments, the surgeon may be able to view additional information about the event or occurrence associated with the marker, which may include information summarizing aspects of the procedure or information derived from past surgical procedures, as described in greater detail above. Any of the features or functionality described above with respect to timeline overlay on surgical video may also apply to the compilation videos described herein.

[0208] Embodiments of the present disclosure may further include training a machine learning model to generate an index of the repository based on the intraoperative surgical events, the surgical outcomes, the patient characteristics, the surgeon characteristics, and the intraoperative surgical event characteristics and generating the index of the repository. Comparing the case-specific information with data associated with the plurality of sets may include searching the index. The various machine learning models described above, including a logistic regression model, a linear regression model, a regression model, a random forest model, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naive Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, a deep learning model, or any other form of machine learning model or algorithm may be used. A training data set of surgical video footage with known intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics may be used to train the model. The trained model may be configured to determine intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics based on additional surgical video footage not included in the training set. When applied to surgical video footage in the repository, the video footage may be tagged based on the identified properties. For example, the video footage may be associated with a footage location, phase tag, event location, and / or event tag as described above with respect to video indexing. Accordingly, the repository may be stored as a data structure, such as data structure 600, described above.

[0209] Fig. 12 is a flowchart illustrating an example process 1200 for surgical preparation, consistent with the disclosed embodiments. Process 1200 may be performed by a processing device, such as one or more collocated or dispersed processors as described herein. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process 1200. Process 1200 is not necessarily limited to the steps shown in Fig. 1200 and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process 1200. At step 1210, process 1200 may include accessing a repository of a plurality of sets of surgical video footage reflecting a plurality of surgical procedures performed on differing patients. The plurality of sets of surgical video footage may include intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics. In some embodiments, the repository may be indexed, for example using process 800, to facilitate retrieval and identification of the plurality of sets of surgical video footage.

[0210] At step 1220, process 1200 may include enabling a surgeon preparing for a contemplated surgical procedure to input case-specific information corresponding to the contemplated surgical procedure. As described above, the contemplated surgical procedure may be a planned procedure, a hypothetical procedure, an experimental procedure, or another procedure that has not yet occurred. The case-specific information may be manually input by the surgeon, for example through a user interface. In some embodiments, some or all of the case-specific information may be received from a medical record of the patient. The case-specific information may include a characteristic of a patient associated with the contemplated procedure, information includes information relating to a surgical tool (e.g., a tool type, a tool model, a tool manufacturer, etc.), or any other information that may be used to identify relevant surgical video footage.

[0211] At step 1230, process 1200 may include comparing the case-specific information with data associated with the plurality of sets of surgical video footage to identify a group of intraoperative events likely to be encountered during the contemplated surgical procedure. The group of intraoperative events likely to be encountered may be determined, for example, based on machine learning analyses performed on historical video footage, historical data other than video data, or any other form of data from which a prediction may be derived. At step 1240, process 1200 may include using the case-specific information and the identified group of intraoperative events likely to be encountered to identify specific frames in specific sets of the plurality of sets of surgical video footage corresponding to the identified group of intraoperative events. The identified specific frames may include frames from the plurality of surgical procedures performed on differing patients, as described earlier.

[0212] At step 1250, process 1200 may include determining that a first set and a second set of video footage from differing patients contain frames associated with intraoperative events sharing a common characteristic, as described earlier. At step 1260, process 1200 may include omitting an inclusion of the second set from a compilation to be presented to the surgeon and including the first set in the compilation to be presented to the surgeon, as described earlier.

[0213] At step 1270, process 1200 may include enabling the surgeon to view a presentation including the compilation containing frames from the differing surgical procedures performed on differing patients. As described above, enabling the surgeon to view the presentation may include outputting data to enable displaying the presentation on a screen or other display device, storing the presentation in a location accessible to another computing device, transmitting the presentation, or any other process or method that may cause the enable the presentation and / or compilation to be viewed.

[0214] When preparing for a surgical procedure, it may be beneficial for a surgeon to review video footage of past surgical procedures. However, in some instances, only particularly complex portions of the surgical procedures may be relevant to the surgeon. Using conventional approaches, it may be too difficult and time consuming for a surgeon to identify portions of a surgical video based on the complexity of the procedure. Therefore, there is a need for unconventional approaches for efficiently and effectively analyzing complexity of surgical footage and enabling a surgeon to quickly review relevant portions of a surgical video.

[0215] Aspects of this disclosure may relate to surgical preparation, including methods, systems, devices, and computer readable media. In particular, when preparing for a surgical procedure, surgeons may wish to view portions of surgical videos that have particular complexity levels. For example, within a generally routine surgical video, a highly skilled surgeon may wish to view only a single event that was unusually complex. Finding the appropriate video and the appropriate location in the video, however, can be time consuming for the surgeon. Accordingly, in some embodiments, methods and systems for analyzing complexity of surgical footage are provided. For example, the process of viewing surgical video clips based on complexity may be accelerated by automatically tagging portions of surgical video with a complexity score, thereby permitting a surgeon to quickly find the frames of interest based on complexity.

[0216] For ease of discussion, a method is described below, with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Other aspects of such a method may occur using non-electronic means. In a broadest sense, the method is not limited to particular physical and / or electronic instrumentalities, but rather may be accomplished using many differing instrumentalities.

[0217] Consistent with disclosed embodiments, a method may involve analyzing frames of the surgical footage to identify in a first set of frames an anatomical structure. As described above, surgical footage may refer to any video, group of video frames, or video footage including representations of a surgical procedure. For example, the surgical footage may include one or more video frames captured during a surgical operation. The first set of frames may be a grouping of one or more frames included within the surgical footage. In some embodiments, the first set of frames may be consecutive frames, however, this is not necessarily true. For example, the first set of frames may include a plurality of groups of consecutive frames.

[0218] As discussed above, an anatomical structure may be any particular part of a living organism, including, for example organs, tissues, ducts, arteries, cells, or other anatomical parts. The first set of frames may be analyzed to identify the anatomical structure using various techniques, for example as described above. In some embodiments, the frames of the surgical footage may be analyzed using object detection algorithms, as described above. For example, the object detection algorithms may be detected objects based on one or more of appearance, image features, templates, and so forth. In some embodiments, identifying the anatomical structure in a first set of frames includes using a machine learning model trained to detect anatomical structures, for example as described above. For example, images and / or videos along with identifications of anatomical structures known to be depicted in the images and / or videos may be input into a machine learning model as training data. As a result, the trained model may be used to analyze the surgical footage to identify in the first set of frames, an anatomical structure. For example, an artificial neural network configured to identify anatomical structures in images and / or videos may be used to analyze the surgical footage to identify in the first set of frames an anatomical structure. Various other machine learning algorithms may be used, including a logistic regression, a linear regression, a regression, a random forest, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naive Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, a deep learning model, or any other form of machine learning model or algorithm.

[0219] Some aspects of the present disclosure may further include accessing first historical data, the first historical data being based on an analysis of first frame data captured from a first group of prior surgical procedures. Generally, frame data may include any image or video data depicting surgical procedures as described herein. The first historical data and / or the first frame data may be stored on one or more storage locations. Accordingly, accessing the first historical data may include retrieving the historical data from a storage location. In other embodiments, accessing the first historical data may include receiving the first historical data and / or the first frame data, for example, from an image capture device or a computing device. Consistent with embodiments of the present disclosure, accessing the first historical data may include retrieving or receiving the first frame data and analyzing the first frame data to identify the first historical data.

[0220] Historical data may be any information pertaining to prior surgical procedures. Some non-limiting examples of such historical data are described above. In some embodiments, the first historical data may include complexity information associated with the first group of prior surgical procedures. The complexity information may include any data indicating a complexity level of the surgery, as discussed further below. The first historical data may include any other information pertaining to the first group of surgical procedures that may be gleaned from the first frame data. For example, the first frame data may include or indicate information associated with the prior surgical procedures, including anatomical structures involved, medical tools used, types of surgical procedures performed, intraoperative events (including adverse events) occurring in the procedures, medical conditions exhibited by the patient, patient characteristics, surgeon characteristics, skill levels of surgeons or other healthcare professionals involved, timing information (e.g., duration of interactions between medical tools and anatomical structures, duration of a surgical phase or intraoperative event, time between appearance of a medical tool and a first interaction between the medical tool and an anatomical structure, or other relevant duration or timing information), a condition of an anatomical structure, a number of surgeons or other healthcare professionals involved, or any other information associated with the prior surgical procedures.

[0221] In embodiments where the first historical data includes complexity information, such information may be indicative of or associated with the complexity of a surgical procedure or a portion thereof. For example, the first historical data may include an indication of a statistical relation between a particular anatomical structure and a particular surgical complexity level. The statistical relation may be any information that may indicate some correlation between the particular surgical complexity level and the particular anatomical structure. For example, when a particular vessel is exposed in a surgical procedure, a particular portion of an organ is lacerated, or a particular amount of blood is detected, such events may statistically correlate to a surgical complexity level. Similarly, detection of a high volume of fat or a poor condition of an organ may also correlate to a complexity level. These are just examples, any condition or event that correlates to surgical complexity may serve as an indication of surgical complexity

[0222] In some embodiments, the first historical data may be identified from the first frame data using one or more image or video analysis algorithms, including object detection algorithms and / or motion detection algorithms. In other embodiments, the first historical data may be identified from the first frame data using a machine learning model trained to identify historical data based on frame data. For example, a machine learning model may be trained using training examples to identify historical data (as described above) from frame data, and the trained machine learning model may be used to analyze the first frame data to determine the first historical data. An example of such training example may include an image and / or a video depicting a surgical procedure or a portion of a surgical procedure, together with a label indicating the complexity level of the surgical procedure or of the portion of a surgical procedure. For example, such label may be generated manually, may be generated by a different process, may be read from memory, and so forth.

[0223] Embodiments of the present disclosure may involve analyzing the first set of frames using the first historical data and using the identified anatomical structure, to determine a first surgical complexity level associated with the first set of frames. As used herein, a complexity level may be a value or other classifier indicating a relative complexity of a surgical procedure or portion of a surgical procedure. For example, the complexity may be based on a difficulty of the surgical procedure relative to other surgical procedures. The difficulty may be based on the surgeon skill level required to perform one or more techniques involved in the surgical procedure, a likelihood of occurrence of an adverse event (such as tear, a bleed, an injury, or other adverse events), a success rate of the surgical procedure, or any other indicator of difficulty of the procedure. Surgical procedures with higher relative difficulty levels may be associated with higher complexity levels.

[0224] As another illustrative example, the complexity level may be based on a duration or time requirement for completing the surgical procedure or portions thereof. For example, procedures or techniques requiring longer performance times may be considered more complex and may be associated with a higher complexity level. As another example, the complexity level may be based on the number of steps required to perform the surgical procedure or portions thereof. For example, procedures or techniques requiring more steps may be considered more complex and may be associated with a higher complexity level. In some embodiments, the complexity level may be based on the type of surgical techniques or procedures being performed. Certain techniques or procedures may have a predetermined complexity and the complexity level may be based on the complexity of the techniques or procedures involved. For example, a cholecystectomy may be considered more complex than an omentectomy and, accordingly, surgical procedures involving the cholecystectomy may be assigned a higher complexity level. Other factors that may be relevant to a complexity level may include information relating to disease severity, complicating factors, anatomical structures involved, types of medical tools used, types of surgical procedures performed, intraoperative events (including adverse events) occurring in the procedures, a physiological response of the patient, a medical condition exhibited by the patient, patient characteristics, surgeon characteristics, a skill level of a surgeon or other healthcare provider involved, timing information (e.g., duration of interactions between medical tools and anatomical structures, a duration of a surgical phase or intraoperative event, time between appearance of a medical tool and a first interaction between the medical tool and an anatomical structure, or other relevant duration or timing information), a condition of an anatomical structure, a number of surgeons or other healthcare professionals involved, or any other information associated with the prior surgical procedures. A surgical complexity level may not be limited to any of the examples above and may be based on a combination of factors, including the examples provided above.

[0225] The surgical complexity level may be represented in various manners. In some embodiments, the complexity level may be represented as a value. For example, the surgical complexity level may be a value within a range of values corresponding to a scale of complexity (e.g., 0-5, 0-10, 0-100, or any other suitable scale). A percentage or other score may also be used. Generally, a higher value may indicate a higher complexity level, however, in some embodiments, the surgical complexity may be an inverse of the value. For example, a complexity level of 1 may indicate a higher complexity than a complexity level of 7. In other embodiments, the complexity level may be represented as a text-based indicator of complexity. For example, the first set of frames may be assigned a complexity level of "high complexity," "moderate complexity," "low complexity," or various other classifiers. In some embodiments, the surgical complexity level may correspond to a standardized scale or index used to represent surgical complexities. The surgical complexity level may be specific to a particular type of surgical procedure (or a subset of surgical procedure types), or may be a universal complexity level applicable to any surgical procedure.

[0226] As mentioned above, the first surgical complexity level may be determined by analyzing the first set of frames using historical data. Analyzing the first set of frames may include any process for determining the complexity level based on information included in the first set of frames. Examples of analysis for determining surgical complexity levels are provided in greater detail below.

[0227] Further, the first surgical complexity level may be determined using the identified anatomical structure. In some embodiments, a type of anatomical structure involved in the procedure may be at least partially indicative of the surgical complexity level. For example, procedures performed on certain anatomical structures (e.g., anatomical structures associated with the brain or heart of a patient) may be considered more complex. In some embodiments, the condition of the anatomical structure may also be relevant to determining the complexity level, as discussed in more detail below.

[0228] Some aspects of the present disclosure may involve analyzing frames of the surgical footage to identify in a second set of frames a medical tool, the anatomical structure, and an interaction between the medical tool and the anatomical structure. For example, the second set of frames may indicate a portion of the surgical footage in which a surgical operation is being performed on the anatomical structure. A medical tool may include any apparatus or equipment used as part of a medical procedure. In some embodiments, the medical tool may be a surgical tool, as discussed above. For example, the medical tool may include, but is not limited to, cutting instruments, grasping and / or holding instruments, retractors, tissue unifying instruments and / or materials, protective equipment, laparoscopes, endoscopes, patient monitoring devices, patient imaging devices, or similar tools. As discussed above, the interaction may include any action by the medical instrument that may influence the anatomical structure, or vice versa. For example, the interaction may include a contact between the medical instrument and the anatomical structure, an action by the medical instrument on the anatomical structure (such as cutting, clamping, grasping, applying pressure, scraping, etc.), a physiological response by the anatomical structure, or any other form of interaction.

[0229] As with the first set of frames, the second set of frames may be a grouping of one or more frames included within the surgical footage. The second set of frames may be consecutive frames, or may include a plurality of groups of consecutive frames. In some embodiments, the first set of frames and the second set of frames may be completely distinct. In other embodiments, the first set of frames and the second set of frames may include at least one common frame appearing in both the first set of frames and the second set of frames. As with the first set of frames, the second set of frames may be analyzed to identify the medical tool, the anatomical structure, and the interaction between the medical tool and the anatomical structure using various techniques. In some embodiments, the frames of the surgical footage may be analyzed using object detection algorithms (e.g. appearance-based detection algorithms, image feature based detection algorithms, template based detection algorithms, etc.) and / or motion detection algorithms. In some embodiments, identifying the medical tool, the anatomical structure, and the interaction between the medical tool and the anatomical structure in the second set of frames may include using a machine learning model trained to detect medical tools, anatomical structures, and interactions between medical tools and anatomical structures. For example, a machine learning model may be trained using training examples to detect medical tools and / or anatomical structures and / or interactions between medical tools and anatomical structures from images and / or videos, and the trained machine learning model may be used to analyze the second set of frames to detect the medical tools and / or the anatomical structures and / or the interactions between medical tools and anatomical structures. An example of such training example may include an image and / or a video clip of a surgical procedure, together with a label indicating at least one of a medical tool depicted in the image and / or in the video clip, an anatomical structure depicted in the image and / or in the video clip, and an interaction between a medical tool and an anatomical structure depicted in the image and / or in the video clip.

[0230] In some exemplary embodiments, identifying the anatomical structure in the first set of frames may be based on an identification of a medical tool and a first interaction between the medical tool and the anatomical structure. In some embodiments, the medical tool identified in the first set of frames may be the same tool as the medical tool identified in the second set of frames. Accordingly, the interaction between the medical tool and the anatomical structure in the second set of frames may be a later interaction between the medical tool and the anatomical structure. This may be helpful, for example, in determining a time between the first interaction and the later interaction, which may be at least partially indicative of a surgical complexity level.

[0231] Embodiments of the present disclosure may further include accessing second historical data, the second historical data being based on an analysis of second frame data captured from a second group of prior surgical procedures. In some embodiments, the first group of prior surgical procedures and the second group of prior surgical procedures may be of a same type. For example, first historical data and second historical data may relate to a first group of appendectomies and a second group of appendectomies, respectively. A first group and second group may differ according to a characteristic. By way of one non-limiting example, the first group may involve patients exhibiting peritonitis, and the second group may include patients who did not exhibit peritonitis.

[0232] In some embodiments, first frame data and second frame data may be identical (i.e., the first historical data and the second historical data may be based on the same frame data). For example, first historical data and second historical data may be based on different analysis of the same frame data. As an illustrative example, first frame data may include estimates of surgical contact force not included in second frame data, consistent with the present embodiments. In some embodiments, first historical data and second historical data may be based on different subsets of the same frame data (e.g., different surgical phases and / or different surgical procedures).

[0233] In some embodiments, the first frame data and the second frame data may be different (i.e., accessed or stored in different data structures). For example, different frames of the same surgical procedures may be analyzed to generate the first historical data than the second historical data.

[0234] In other embodiments the first group of prior surgical procedures and the second group of prior surgical procedures may be different in at least one aspect. For example, the first and second group may include appendectomies but may differ in that the first group includes appendectomies in which an abnormal fluid leakage event was detected while no abnormal fluid leakage events were detected in the second group. In some embodiments, the first group of prior surgical procedures and the second group of prior surgical procedures may have at least one surgical procedure in common (e.g., both groups may include an incision). In other embodiments, however, the first group of prior surgical procedures and the second group of prior surgical procedures may have no surgical procedures in common.

[0235] In some embodiments, a method may include tagging a first set of frames with a first complexity level, tagging a second set of frames with the second complexity level, and storing first set of frames with the first tag and the second set of frames with the second tag in a data structure. This may enable a surgeon to select the second complexity level, and thereby cause the second set of frames to be displayed, while omitting a display of the first set of frames. In some embodiments, a method may include receiving a selection of a complexity level (e.g., receiving a selection based on user input to an interface). Further, a method may include accessing a data structure to retrieve selected frames. A method may include displaying frames tagged with the selected complexity level while omitting frames tagged without the selected complexity level.

[0236] Similar to the first historical data and frame data, the second historical data and frame data may be stored in one or more storage locations. In some embodiments, the second historical data may be stored in the same storage location as the first historical data. In other embodiments, the first and second historical data may be stored in separate locations. Consistent with other embodiments, ac...

Claims

1. A system for generating decision support data for surgical videos, the system comprising: at least on processor configured to perform operations including: receiving video footage of a surgical procedure performed by a surgeon on a patient in an operating room; implementing a machine learning model on the received video footage to determine an existence of a surgical decision-making junction, wherein the determination of the existence of the surgical decision-making junction is based on both a detected physiological response of an anatomical structure and a motion associated with a surgical tool, the machine learning model being a result of training a machine learning algorithm to detect decision-making junctions using training data based on historical video footage depicting surgical situations; accessing, in a data structure, a correlation between an outcome and a specific action taken at the decision-making junction; and based on the determined existence of the decision-making junction and the accessed correlation, outputting a recommendation to at least one of a user device or a user interface related to the specific action.

2. The system of claim 1, configured to permit performance of the operations in real time during the surgical procedure.

3. The system of any of claims 1-2, wherein prior to accessing the correlation, the system is configured to generate the correlation based on the historical video footage depicting differing courses of action and differing outcomes in a common surgical situation and store the correlation in the data structure.

4. The system of any of claims 1-3, wherein the video footage includes images from at least one of an endoscope and an intracorporeal camera.

5. The system of any of claims 1-4, wherein the recommendation includes a recommendation to conduct a medical test, and wherein the operations further optionally include: receiving a result of the medical test; and based on the determined existence of the decision-making junction and the accessed correlation and the received result of the medical test, outputting a second recommendation to at least one of the user device or the user interface to undertake a particular action.

6. The system of any of claims 1-5, wherein the specific action includes bringing an additional surgeon to the operating room.

7. The system of any of claims 1-6, wherein the decision-making junction includes at least one of inappropriate access or exposure, retraction of an anatomical structure, misinterpretation of an anatomical structure or a fluid leak, and wherein the recommendation optionally includes a confidence level that a desired surgical outcome will occur if the specific action is taken or a confidence level that a desired outcome will not occur if the specific action is not taken.

8. The system of any of claims 1-7, wherein the recommendation is based on time elapsed since a particular point in the surgical procedure, and wherein the recommendation optionally includes an indication of an undesired surgical outcome likely to occur if the specific action is not undertaken.

9. The system of any of claims 1-8, wherein the recommendation is based on one or more of a skill level of the surgeon, a condition of a tissue of the patient, and a surgical event that occurred in the surgical procedure prior to the decision-making junction.

10. The system of any of claims 1-9, wherein the specific action includes a plurality of steps.

11. The system of any of claims 1-10, wherein the operations further include receiving a vital sign of the patient and wherein the recommendation is based on the accessed correlation and the vital sign.

12. The system of any of claims 1-11, wherein the surgeon is a surgical robot and the recommendation is provided in the form of an instruction to the surgical robot.

13. The system of any of claims 1-12, wherein the recommendation of the specific action includes creation of a stoma.

14. A non-transitory computer readable medium including instructions that, when executed by at least one processor, cause the at least one processor to execute operations that provide decision support data for surgical procedures, the operations comprising: receiving video footage of a surgical procedure performed by a surgeon on a patient in an operating room; implementing a machine learning model on the received video footage to determine an existence of a surgical decision-making junction, wherein the determination of the existence of the surgical decision-making junction is based on both a detected physiological response of an anatomical structure and a motion associated with a surgical tool, the machine learning model being a result of training a machine learning algorithm to detect decision-making junctions using training data based on historical video footage depicting surgical situations; accessing, in a data structure, a correlation between an outcome and a specific action taken at the decision-making junction; and based on the determined existence of the decision-making junction and the accessed correlation, outputting a recommendation to at least one of a user device or a user interface related to the specific action.