System and method for analyzing surgical operation video

Abstract

To provide a system and a method for analyzing and reviewing surgical operation video that enables a surgeon to view an operation event to give decision support and / or efficiently and effectively analyzes the surgical operation video so as to facilitate after-operation activities.SOLUTION: A method for reviewing surgical operation video includes the steps of: accessing at least one video of a surgical procedure; causing the at least one video to be output for display; overlaying on the at least one video output for display a surgical timeline, wherein the surgical timeline includes markers identifying at least one of a surgical phase, an intraoperative surgical event, and a decision making junction; and enabling a surgeon, while viewing playback of the at least one video, to select one or more markers on the surgical timeline, and thereby causing a display of the video to skip to a location associated with the selected marker.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 62 / 808,500, filed February 21, 2019. No. 62 / 808,512, filed February 21, 2019; U.S. Provisional Patent Application No. 62 / 838,066, filed April 24, 2019, 2020 U.S. Provisional Patent Application No. 62 / 960,466, filed January 13, 2020, and This application is based on the benefit of priority to U.S. Provisional Patent Application No. 62 / 967,283, filed on January 29, 2014. The contents of the aforementioned applications are incorporated herein by reference in their entireties. It can be seen. [Background technology]

[0002] The disclosed embodiments generally relate to a system and method for analysis of videos of surgical procedures. Regarding. In preparing for a surgical procedure, the surgeon may consider certain surgical events, including events that may have specific characteristics. It can be beneficial to watch video footage depicting the elephant. It can be useful to capture and analyze the data to provide various types of decision support to the surgeon. ,It may be useful to analyze surgical videos to facilitate ,postoperative actions. Summary of the Invention [Problem to be solved by the invention]

[0003] Thus, allowing the surgeon to view surgical events, providing decision support, and / or Or, non-traditional methods for efficiently and effectively analyzing surgical videos to facilitate postoperative actions. A method like this is needed. [Means for solving the problem]

[0004] Embodiments consistent with the present disclosure provide systems and methods for analyzing surgical videos. The disclosed system and method may be implemented using conventional hardware and software as well as the disclosed A machine specifically constructed and / or programmed to perform the functions associated with a given method step. may be implemented using a combination of dedicated hardware and software, such as a Consistent with other disclosed embodiments, the non-transitory computer-readable storage medium may include at least The steps and methods described herein may be performed by a single processing device. The device may store program instructions for performing any of the methods and / or techniques.

[0005] Consistent with the disclosed embodiments, systems, methods, and methods relating to surgical video review are provided. A computer-readable medium is disclosed. and outputting at least one video for display. The embodiment may include a video feed on at least one video output for display. The method may further include overlaying a surgical timeline. Identify at least one of the following: surgical stage, intraoperative surgical event, and decision junction. The surgical timeline may include markers that identify the surgeon's surgical procedure. Allows you to select one or more markers on the surgical timeline while watching the replay, which This may cause the video display to skip to the position associated with the selected marker.

[0006] In one embodiment, one or more markers are associated with a decision junction of a surgical procedure. The decision junction marker may include a corresponding decision junction marker. Selection is based on the surgeon's selection of two or more separate video clips from two or more corresponding other surgical procedures. Furthermore, two or more video clips may be used to show different actions. In another embodiment, the selection of the decision junction marker may be An indication of one or more alternative possible decisions associated with the selected decision junction marker. may cause.

[0007] Consistent with the disclosed embodiments, systems, methods, and methods relating to video indexing, and a computer-readable medium. accessing video footage to be indexed, including video footage of the location. The images may identify video image locations associated with operative stages of a particular surgical procedure. A stage tag may be generated, and the stage tag may be used to determine the position of the video footage. Video indexing may be performed by analyzing video footage to identify surgical stages. Identifies the event location of a specific intraoperative surgical event within a floor and associates the event tag with a specific intraoperative surgical event. Further, the method may include associating the location of the intraoperative surgical event with the location of the intraoperative surgical event. The event characteristics that are detected may be stored.

[0008] Video indexing involves indexing at least a portion of the video footage of a particular surgical procedure to a The step tags, event tags, and event tags in the data structure containing further video footage of the surgical procedure. The data structure may further include associating the data with other surgical procedures. Each stage tag, each event tag, and The user may select a selected stage of the video footage for display. Access the data structure by selecting a tag, a selected event tag, and a selected event property. Thereafter, a matching subset of stored video footage is identified. To do this, at least one selected stage tag, a selected event tag, and a selected A lookup may be performed in the data structure for surgical video footage that matches the event characteristics. A matching subset of the stored video footage may be displayed to the user, This allows the user to view the selected event characteristics while omitting playback of video footage that lacks the selected event characteristics. The surgical footage of at least one intraoperative surgical event that shares the selected event characteristics can be viewed.

[0009] Consistent with the disclosed embodiments, systems, methods, and methods relating to the generation of surgical overview videos are provided. A computer-readable medium is disclosed. A first group of frames associated with the surgical action and a second group of frames not associated with the surgical action. The embodiment may include accessing a particular surgical video including the group. Steps to access historical data associated with historical surgical footage of previous surgical procedures a database in which the historical data associates a portion of the historical surgical footage with an intraoperative surgical event; and a further step including information distinguishing between frames associated with surgical actions and frames not associated with surgical actions. Based on the information of the historical data, the first time a frame is detected in a particular surgical video, the first time a frame is detected in the ... A group may be distinguished from a second group of frames. Based on user requirements, whereas an aggregation of a first group of frames of a particular surgical video may be presented to the user. , the second group of frames may be omitted from presentation to the user.

[0010] In some embodiments, the disclosed embodiments analyze specific surgical footage to determine the surgical The method may further include identifying causes of each of the outcomes and surgical outcomes. Based on historical outcome data and respective historical causal data. , the outcome set of frames in a particular surgical video may be detected. may be within the outcome stage of the surgical procedure. A set of causes for a frame may be detected. The set of causes for a frame may be determined from the outcome stage to the time There may be several stages of the surgical procedure separated by a distance, while an intermediate set of frames may be Even within an intermediate stage interposed between the cause set of the frame and the outcome set of the frame. A causality summary of the surgical video may then be generated, where the causality summary is , including a cause set of frames and an outcome set of frames, as well as an intermediate set of frames. The aggregation of the first group of frames presented to the user includes a causal summary. Good too.

[0011] Consistent with the disclosed embodiments, systems, methods, and computers related to surgical preparation are provided. A data-readable medium is disclosed. It reflects the placement of the patient and also includes intraoperative events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative procedures. accessing a repository of a plurality of sets of surgical video footage including event characteristics. The method may include a step of: a surgeon preparing for a proposed surgical procedure; The method may further include allowing entry of case-specific information corresponding to the procedure. To identify groups of intraoperative events likely to be encountered during the illustrated surgical procedure, Case-specific information may be compared to data associated with multiple sets of surgical video footage. Additionally, multiple sets of surgical video footage corresponding to identified groups of intraoperative events are analyzed. Case-specific information and the likelihood of encountering a problem are used to identify a specific frame within a particular set of cases. Identified groups of intraoperative events with high likelihood of occurrence may be used. A frame may contain frames from multiple surgical procedures performed on different patients.

[0012] In an embodiment, the first and second sets of video footage from different patients share a common feature. determining that the frame includes frames associated with intraoperative events that share a common gender; The second set is omitted from the compilation to be presented and presented to the surgeon. and including the first set in a compilation to be compiled. In some embodiments, frames from different surgical procedures performed on different patients may be included in the composition. The method may include making the presentation including the pile available for viewing by the surgeon.

[0013] Consistent with the disclosed embodiments, systems, methods, and methods relating to analyzing the complexity of surgical videos are provided. and a computer-readable medium. The method may include identifying anatomical structures in the first set of frames. The illustrated embodiment may further include accessing the first historical data. The historical data for one surgical procedure is the first frame data captured from the first group of previous surgical procedures. The first set of frames may be based on an analysis of the data. First, use the historical data and the identified anatomical structures to determine the degree level. A first set of frames may be analyzed using:

[0014] Some embodiments analyze frames of the surgical video to determine in a second set of frames: to identify medical devices, anatomical structures, and interactions between medical devices and anatomical structures. The disclosed embodiment may further include the step of accessing the second historical data. wherein second historical data is captured from a second group of previous surgical procedures. The second frame of data may include a step of: determining a second surgical complexity level associated with the set of second historical data; A second set of frames may be analyzed using the identified interactions. stomach.

[0015] An embodiment includes tagging a first set of frames with a first surgical complexity level. tagging a second set of frames with a second surgical complexity level; a first set of frames with a tag of 1 and a second set of frames with a tag of 2 and generating a data structure including the external Allows the surgeon to select a second surgical complexity level, thereby The second set of frames is displayed while omitting the display of the first set of frames.

[0016] Consistent with the disclosed embodiments, a system for enabling adjustment of operating room schedules is provided. A system, method, and computer readable medium are disclosed. The step tracks the ongoing surgical procedure from an image sensor located within the surgical operating room. Receiving visual data and accessing a data structure containing historical surgical data Analyze visual and historical surgical data of steps and ongoing surgical procedures to and determining an estimated completion time for the surgical procedure. The step of adjusting the schedule includes the step of accessing a schedule for the surgical operating room. The schedule may further include a schedule associated with the completion of an ongoing surgical procedure. Further, adjusting the operating room schedule may include adjusting the operating room schedule. Based on the estimated completion time of the ongoing surgical procedure, a schedule is created where the predicted completion time is associated with completion. calculating whether the time is likely to result in variance from the time spent; Notifications are output when variances are calculated, allowing subsequent users of the surgical suite to adjust their schedules. and allowing the module to be adjusted accordingly.

[0017] Consistent with the disclosed embodiments, a method for analyzing surgical images for insurance reimbursement determinations is provided. A system, method, and computer readable medium for determining insurance reimbursement are disclosed. The operation of analyzing surgical images to obtain a surgical result is performed using video captured during a surgical procedure on a patient. accessing an on-frame and providing at least one medical device, at least one solution an anatomical structure and a connection between at least one medical device and at least one anatomical structure; At least one interaction was identified in the video frame captured during the surgical procedure. analyzing the video frames for identifying medical devices, anatomical structures, and Access to a database of reimbursement codes that correlate to interactions between anatomical structures The operation may include a step of connecting at least one medical device and at least one anatomical Identify at least one interaction between the physical structure and the reimbursement code database. to determine at least one reimbursement code associated with the surgical procedure. The method may further include the step of:

[0018] Consistent with the disclosed embodiments, a system, method, and system for generating a post-operative report of a surgical procedure are provided. A method and computer readable medium for generating a post-operative report of a surgical procedure are disclosed. The operations include receiving an input of a patient identifier and receiving an input of a healthcare provider identifier. The steps of the procedure are recorded and the surgical footage is recorded of the surgical procedure performed on the patient by the healthcare provider. and receiving an input of the surgical video. analyzing the image data to derive image-based information for generating a post-operative report of the surgical procedure; causing the derived image-based information to generate a post-operative report of the surgical procedure; It may further include.

[0019] Consistent with the disclosed embodiments, a method is provided for determining and notifying omitted events in a surgical procedure. A system, method, and computer-readable medium for enabling the The behavior of the video captured during a particular surgical procedure is used to enable the determination and notification of the elephant's movements. Identify the steps to access the frame and the recommended sequence of events for the surgical procedure. accessing the stored data and converting the accessed frame into a recommended set. The relationship between a specific surgical procedure and the recommended sequence of events for the surgical procedure compared with the sequence of events identifying an indication of a deviation between the surgical and non-surgical stages and determining the name of the intraoperative surgical event associated with the deviation; and providing a notification of the deviation including the name of the intraoperative surgical event associated with the deviation. The method may include a step.

[0020] Some embodiments of the present disclosure provide a method for real-time decision support for surgical procedures. Some such embodiments include systems, methods, and computer-readable media for: Such an embodiment may involve at least one processor. receiving video footage of a surgical procedure performed by a surgeon on the patient; a scanner that accesses at least one data structure containing image-related data that characterizes the surgical procedure; Surgical decision making involves determining the presence of a junction in real time. The received video footage may be analysed using image related data to In another data structure, the outcome and the specific action to be taken at the decision junction are The correlation between the determined existence of decision junctions and the Based on the correlation accessed, a particular action may be taken or a particular action may be avoided. Recommendations may be output to the surgeon such as:

[0021] The disclosed embodiments provide for estimating the contact forces acting on anatomical structures during the disclosed surgical procedures. The present invention includes a system, method, and computer-readable medium for determining receiving image data of the surgical procedure from at least one image sensor in analyzing the received image data to determine the identity of the anatomical structure and The method may involve determining the state of the anatomical structure reflected in the anatomical structure. Based on the determined condition, a contact force threshold associated with the anatomical structure may be selected. The actual contact force acting on the anatomical structure is determined and a selected contact force threshold is calculated. Thereafter, if the actual contact force indication exceeds the selected contact force threshold, A notification may be output based on the determination.

[0022] Some embodiments of the present disclosure include a system for updating a predicted outcome during a surgical procedure; The present invention relates to a method and a computer-readable medium for capturing images of a surgical procedure. a first event during a surgical procedure from at least one image sensor positioned to capture the first event; The first event may involve receiving associated image data. a predicted outcome associated with the surgical procedure based on the received image data associated with the surgical procedure; and at least one imaging device positioned to capture images of the surgical procedure. and receiving image data from the two image sensors that is associated with a second event during the surgical procedure. Embodiments may include: predicting a second event based on received image data associated with the second event; A change in the measured outcome may be determined that reduces the predicted outcome below a threshold. Corrective actions taken are based on image-related data about previous surgical procedures contained in the data structure. may be identified and recommended.

[0023] Some embodiments of the present disclosure include a system for enabling fluid leak detection during surgery; The present invention relates to a method and a computer-readable medium for recording intraluminal video of a surgical procedure. The processor analyzes the frames of the endoluminal video to generate an endoluminal image. The embodiment may be configured to determine abnormal fluid leakage conditions in the video. Corrective action may be developed when a fluid leak situation is determined.

[0024] Consistent with the disclosed embodiments, a system, method, and method for predicting post-discharge risk are provided. A computer-readable medium is disclosed. accessing frames of video captured during a particular surgical procedure, accessing stored historical data identifying events and associated outcomes; It analyzes the accessed frames and determines the access based on information obtained from historical data. identifying at least one particular intraoperative event in the accessed frames; Based on the information obtained from the historical data and the identified at least one intraoperative event, determining a predicted outcome associated with a particular surgical procedure; and associating the predicted outcome with the patient. and outputting the predicted outcome in a manner that includes labeling the outcome.

[0025] The foregoing summary provides only a few examples of the disclosed embodiments to give an idea of ​​the disclosure. and is not intended to summarize all aspects of the disclosed embodiments. Moreover, the following detailed description is merely exemplary and explanatory and is not to be construed as limiting the invention as claimed. This does not limit the following.

[0026] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. An embodiment will be illustrated. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a perspective view of an example operating room consistent with disclosed embodiments. [Figure 2] FIG. 1 is a perspective view of a camera consistent with disclosed embodiments. [Figure 3] FIG. 1 is a perspective view of an example of a surgical instrument consistent with disclosed embodiments. [Figure 4] 1 shows an example of a timeline overlaid on a video of a surgical procedure consistent with disclosed embodiments. [Figure 5] 1 is a flowchart illustrating an example of a process for reviewing surgical videos consistent with disclosed embodiments. [Figure 6] FIG. 1 is a schematic diagram of an example data structure consistent with disclosed embodiments. [Figure 7] FIG. 1 is a schematic diagram of an example of a user interface for selecting indexed video footage for display consistent with disclosed embodiments. [Figure 8A] 1 is a flowchart illustrating an example of a process for video indexing consistent with disclosed embodiments. [Figure 8B] 1 is a flowchart illustrating an example of a process for video indexing consistent with disclosed embodiments. [Figure 9] 1 is a flowchart illustrating an example of a process for distinguishing a first group of frames from a second group of frames, consistent with disclosed embodiments. [Figure 10] 1 is a flowchart illustrating an example of a process for generating a causal summary consistent with disclosed embodiments. [Figure 11] 1 is a flowchart illustrating an example of a process for generating a surgical overview video consistent with disclosed embodiments. [Figure 12] 1 is a flowchart illustrating an example of a process for surgical preparation consistent with disclosed embodiments. [Figure 13] 1 is a flowchart illustrating an example of a process for analyzing the complexity of surgical footage consistent with disclosed embodiments. [Figure 14]FIG. 1 is a schematic diagram of an exemplary system for managing various data collected during a surgical procedure and controlling various sensors consistent with disclosed embodiments. [Figure 15] 1 is an exemplary timeline consistent with the disclosed embodiments. [Figure 16] 1 is an exemplary form for entering information regarding a schedule consistent with the disclosed embodiments. [Figure 17A] 1 illustrates an exemplary data structure consistent with the disclosed embodiments. [Figure 17B] 1 shows an exemplary plot of historical completion time data consistent with disclosed embodiments. [Figure 18] 1 illustrates an example of a machine learning model consistent with disclosed embodiments. [Figure 19] 1 illustrates an exemplary process for adjusting operating room schedules consistent with disclosed embodiments. [Figure 20] 1 is an exemplary data structure for storing correlations between reimbursement codes and information obtained from surgical footage, consistent with disclosed embodiments. [Figure 21] FIG. 1 is a block diagram of an exemplary machine learning method consistent with disclosed embodiments. [Figure 22] 1 is a flowchart of an exemplary process for analyzing surgical images to determine insurance reimbursement, consistent with disclosed embodiments. [Figure 23] 1 is an exemplary post-operative report including fields consistent with disclosed embodiments. [Figure 24A] 1 is an example of a process including a structure for generating a post-operative report consistent with disclosed embodiments. [Figure 24B] 10 is another example of a process including a structure for generating a post-operative report consistent with disclosed embodiments. [Figure 25] FIG. 1 is a flow diagram of an exemplary process for generating a post-operative report consistent with disclosed embodiments. [Figure 26] 1 is a schematic diagram of a typical sequence of events consistent with disclosed embodiments. [Figure 27] 1 illustrates an exemplary comparison of a sequence of events consistent with disclosed embodiments. [Figure 28] 1 illustrates an exemplary process that enables determination and notification of omitted events consistent with disclosed embodiments. [Figure 29] 1 is a flowchart illustrating an exemplary process for decision support regarding a surgical procedure consistent with disclosed embodiments. [Figure 30] 1 is a flowchart illustrating an exemplary process for estimating contact forces acting on an anatomical structure during a surgical procedure, consistent with disclosed embodiments. [Figure 31] 1 is a flowchart illustrating an exemplary process for updating a predicted outcome during a surgical procedure consistent with disclosed embodiments. [Figure 32] 1 is a flowchart illustrating an exemplary process for enabling intra-operative fluid leak detection consistent with disclosed embodiments. [Figure 32A] 1 is an exemplary graph illustrating the relationship between intraoperative events and outcomes consistent with disclosed embodiments. [Figure 32B] 10 is an exemplary probability distribution graph for different events with and without the presence of intraoperative events, consistent with disclosed embodiments. [Figure 33] 1 illustrates exemplary probability distribution graphs for different events consistent with disclosed embodiments. [Figure 34] 1 shows exemplary probability distribution graphs for different events as a function of event characteristics, consistent with disclosed embodiments. [Figure 35A] 1 illustrates an exemplary machine learning model consistent with disclosed embodiments. [Figure 35B] 1 illustrates exemplary inputs for a machine learning model consistent with disclosed embodiments. [Figure 36] 1 illustrates an exemplary process for predicting post-discharge risk consistent with disclosed embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0028] Unless otherwise specified, as will be apparent from the following description, throughout the specification: "Process," "Calculate," "Compute," "Determine," "Generate," "Configure," "," "selection," "provision," "application," "acquisition," "monitoring," "provision," "identification," "segmentation" "mentation," "classification," "analysis," "association," "extraction," "storage," "reception," "transmission" Discussions using terms such as "signal" are based on data expressed as physical quantities, e.g., electron quantities. manipulate and / or transform data representing physical objects into other data "Computer," "processor," " "controller," "processing unit," "arithmetic unit," and "processing module" The term includes, by way of non-limiting example, personal computers, wearable computers, Smart glasses, tablets, smartphones, servers, computing systems, Cloud computing platforms, communication devices, and in some cases built-in memory A processor having a logic (e.g., a digital signal processor (DSP), an image signal processor, processor (ISR), microcontroller, field programmable gate array (F PGA), Application Specific Integrated Circuit (ASIC), Central Processing Unit (CPA), Graphics Graphics Processing Unit (GPU), Visual Processing Unit (VPU), Single-core Processor processor, a multi-core processor, a core within a processor, or any other electronic computing device any type of electronic device with data processing capabilities, including any combination of the above Should be interpreted expansively to encompass child devices, components, or units is.

[0029] The operations of the teachings herein do not necessarily require specially constructed or programmed components to perform the functions described. The program may be executed by a computer running the program.

[0030] As used herein, the words "for example," "such as," "e.g.," and "for example," and These variations illustrate non-limiting embodiments of the presently disclosed subject matter. "in one case," "in some cases," "in other cases," or variations thereof Reference to a feature does not necessarily imply that the particular feature, structure, or characteristic described is a minimum of all of the subject matter disclosed herein. It means that at least one embodiment may include any of the above. As used herein, terms do not necessarily refer to the same embodiment(s). The term "and / or" as used herein means any and all of one or more of the associated listed items. This includes any combination.

[0031] Features of the subject matter disclosed herein are described in the context of specific embodiments, for brevity's sake. However, features described in connection with one embodiment may also apply to other embodiments. Similarly, it should be understood that any combinations described in connection with a particular The features described herein, either alone or in any other context than in the specified combination, shall not be considered as separate embodiments. Good too.

[0032] In embodiments of the presently disclosed subject matter, one or more of the steps shown in the figures may be performed in a different order. and / or one or more groups of steps may be performed simultaneously, or vice versa. The figure shows a system architecture according to an embodiment of the subject matter disclosed herein. 1 shows a general schematic diagram of the device. Each module in the diagram is defined and explained herein. Any software, hardware, and / or firmware that performs such functions The modules in the diagram may be centralized in one location or Alternatively, it may be distributed across two or more locations.

[0033] Examples of the presently disclosed subject matter include the components set forth in the following description or shown in the drawings. The application is not limited to details of construction and arrangement. Furthermore, the phraseology and terms used herein are for the purpose of explanation only. It should be understood that this should not be considered limiting.

[0034] In this document, parts not shown within the drawings are labeled with numbers that appear in the previous drawings. Elements in subsequent drawings may have the same purpose and description as in the previous drawing.

[0035] The drawings in this document may be to any scale. Different figures may have different scales. Also, even within the same drawing, different scales may be used, e.g. Different scales for different fields of view or different scales for two adjacent objects It can be used.

[0036] Consistent with the disclosed embodiments, "at least one processor" refers to a processor that receives input or multiple Any physical device or group of devices that has electrical circuitry that performs logical operations on inputs. For example, at least one processor may be configured to implement an application specific integrated circuit (AIC). ASIC, microchip, microcontroller, microprocessor, central processing CPU (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP) ), Field Programmable Gate Array (FPGA), Server, Virtual Server, or One or more collections of circuits, including all or part of other circuits suitable for executing instructions or performing logical operations The instructions executed by the at least one processor may include an integrated circuit (IC). For example, the data may be pre-loaded into a memory that is integrated with or embedded in the controller. The random number may be stored in a separate memory. RAM, read-only memory (ROM), hard disk, optical disk disk, magnetic media, flash memory, other persistent, fixed or volatile memory, or Any other mechanism capable of storing instructions may be used. In the embodiment, the at least one processor may include two or more processors. The processors may have a similar structure, or the processors may be electrically connected or disconnected from each other. For example, the processor may be a separate circuit or or may be integrated into a single circuit. If multiple processors are used, the processors The processor may be configured to operate independently or in concert. electrically, optically, acoustically, mechanically, or by any other means that allows them to interact. Therefore, they may be combined.

[0037] The disclosed embodiments may include and / or access a data structure. A data structure consistent with this disclosure may represent any set of data values ​​and the relationships between them. Data may be organized linearly, horizontally, hierarchically, relationally, non-relationally, simply, dimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, In an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner The data may be stored in any manner that allows for data access. As data structures, there are arrays, associative arrays, linked lists, binary trees, balanced trees, chunks, and stacks. Tacks, queues, sets, hash tables, records, tagged associations, ER models, and For example, the data structure may include a graph, e.g., MongoDB, Redis , Couchbase, Datastax Enterprise Graph, Ela stic Search, Splunk, Solr, Cassandra, Amazon Datastores such as DynamoDB, Scylla, HBase, and Neo4J XML databases, RDBMS databases, SQL databases for storage / search The data structures may include the components of the disclosed system. component or remote computing component (e.g., cloud-based The data in the data structure may be stored in contiguous or non-contiguous memory. Furthermore, as used herein, a data structure refers to a structure in which information is co-located. Data structures may, for example, be owned by the same or different entities. or distributed across multiple servers that may be operated by the same or similar application. The term "data structure" as used herein includes multiple data structures.

[0038] In some embodiments, a machine learning algorithm (also referred to as a machine learning model in this disclosure) (which is trained using training examples, e.g., in the case described below) Some non-limiting examples of such machine learning algorithms include classification algorithms. algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms Rhythms (e.g., object detectors, face detectors, people detectors, motion detectors, edge detectors, etc.) , visual recognition algorithms (e.g., face recognition, person recognition, object recognition, etc.), speech recognition algorithms Rhythm, mathematical embedding algorithms, natural language processing algorithms, support vector machines Shin, Random Forest, Nearest Neighbor Algorithm, Deep Learning Algorithm, Artificial Neural Network Neural network algorithms, convolutional neural network algorithms, recursive neural networks Neural network algorithms, linear machine learning models, nonlinear machine learning models, Examples include sample algorithms, trained machine learning, etc. The algorithms include prediction models, classification models, regression models, clustering models, segmentation models, Contoured models, artificial neural networks (e.g., deep neural networks, convolutional neural networks) neural networks, recurrent neural networks, etc.), random forests In some examples, the model may include inference models such as metric models, support vector machines, etc. The training examples may include input examples and desired outputs corresponding to the input examples. In some cases, training examples can be used to train machine learning algorithms. This results in a trained machine learning algorithm, and The machine learning algorithms developed are able to estimate outputs for inputs that are not included in the training examples. In some instances, it may be used to train machine learning algorithms. The engineers, scientists, processes, and machines that perform the verification and / or testing may further use the verification and / or testing examples. For example, validation examples and / or test examples may be input along with desired outputs corresponding to the input examples. The input examples may include the trained machine learning algorithm and / or intermediate training The trained machine learning algorithm generates output on input examples of validation examples and / or test examples. may be used to estimate the desired output, and the estimated output may be compared to the corresponding desired output. and a machine learning algorithm and / or algorithms trained based on the results of the comparison. Intermediately trained machine learning algorithms may be evaluated. In this case, the machine learning algorithm may have parameters and hyperparameters. In this case, hyperparameters are determined manually by a human or by a process external to the machine learning algorithm. It is automatically set by the algorithm (e.g., hyperparameter search algorithm) and machine learning. The parameters of the learning algorithm are determined by the machine learning algorithm according to the training examples. In some implementations, the hyperparameters are set for the training and validation examples. The parameters are set accordingly, and the training examples and selected hyperparameters are is set according to

[0039] In some embodiments, for example in the cases described below, A machine learning algorithm (also referred to in this disclosure as a trained machine learning model) In some examples, the trained The machine learning algorithms used are inference models that generate inferred outputs given inputs. For example, a trained machine learning algorithm may be used as a classification algorithm. The inferred output may include a algorithm, the input may include samples, and the inferred output may include samples. It may also include classifications (e.g., inferred labels, inferred tags, etc.). The trained machine learning algorithm may include a regression model, and the input may be a sample and the inferred output may include an inferred value of the sample. In another example, the trained machine learning algorithm includes a clustering model. The input may include samples, and the inferred output may include at least one class. In a further example, the trained machine learning The learning algorithm may include a classification algorithm, the input may include an image, and The inferred output may include a classification of the item depicted in the image. The trained machine learning algorithm may include a regression model, and the input may include an image. and the inferred output may be an inferred value of the item depicted in the image (e.g., Estimates of items such as the size, volume, and age of the person depicted in the image, and the cost of the product depicted in the image In a further example, the trained machine learning algorithm The rhythm may include an image segmentation model, the input may include an image, and The discussed output may include a segmentation of the image. The selected machine learning algorithm may include an object detector, the input may include an image, or Additionally, the inferred output may be one or more detected objects in the image and / or one or more of the objects in the image. In some examples, a trained machine learning algorithm may The rhythm can be one or more formulas and / or one or more functions and / or one or more rules and / or The input may include one or more procedures, and may be a formula and / or a function and / or a rule and / or a procedure. may be used as input to a device, and the inferred output may be expressed as a formula and / or function and / or or may be based on the output of rules and / or actions (e.g., formulas and / or functions and / or A formula and / or function and / or rule that selects one of the outputs of the rule and / or action. and / or use statistical measures of the outcome of the treatment).

[0040] In some embodiments, the artificial neural network analyzes the input and generates a corresponding Some such artificial neural networks may be configured to generate outputs that Some non-limiting examples are shallow artificial neural networks, deep artificial neural networks, neural networks, feedback artificial neural networks, feedforward artificial neural networks Networks, Autoencoder Artificial Neural Networks, Probabilistic Artificial Neural Networks Work, time delay artificial neural network, convolutional artificial neural network, link Current artificial neural networks, long short-term memory artificial neural networks, etc. In some instances, the artificial neural network may be manually configured. For example, the structure of the artificial neural network may be manually selected, The type of artificial neuron in the neural network may be selected manually, network parameters (such as the parameters of the artificial neurons in an artificial neural network) etc.) may be manually selected, and so on. In some examples, artificial neural The network may be configured using machine learning algorithms. For example, the user , hyperparameters for artificial neural networks and / or machine learning algorithms The data may be selected, and the machine learning algorithm may be, for example, backpropagation. Using gradient descent, using stochastic gradient descent, using mini-batch gradient descent Hyperparameters are used to determine the parameters of artificial neural networks. In some cases, two or more other parameters and training examples may be used. By combining artificial neural networks into a single artificial neural network, Thus, an artificial neural network is created from two or more other artificial neural networks. It may be made.

[0041] In some embodiments, the image data is processed (e.g., as described in the methods, steps, etc., described herein). Analyzing (by the group and module) involves analyzing the image data to produce a preprocessed image. acquiring image data and subsequently providing image data and / or pre-processed image data; The method may include analyzing some of such image data to obtain a desired outcome. Non-limiting examples include one or more images, videos, frames, footage, 2D image data, 3D image data, As will be appreciated by those skilled in the art, the following are examples: , the image data may be pre-processed using other types of pre-processing methods. In some examples, Image data is converted using a conversion function to obtain the converted image data. The data may be pre-processed, and the pre-processed image data may include transformed image data. For example, the transformed image data may include one or more convolutions of the image data. For example, the transformation functions are low-pass, high-pass, band-pass, and all-pass. In some examples, the transformation function may include a nonlinear function. In some examples, the image data may be subjected to a Gaussian convolution, for example. At least a part of the image data is filtered by using only the image data or by using a median filter. In some examples, the image data may be pre-processed by smoothing. The data may be pre-processed to obtain different views of the data. For example, the pre-processed image data may be The data may include a representation of at least a portion of the image data in the frequency domain, a representation of at least a portion of the image data in the frequency domain, discrete Fourier transform of at least a portion of the image data, discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, Partial display, loss of at least a portion of the image data. Partial display, loss of at least a portion of the image data. Includes loss indications, ordered time series of any of the above, any combination of the above, etc. In some examples, the image data may be pre-processed to extract edges, The pre-processed image data may also include extracted edge-based and / or related information. In some examples, the image data may be processed to extract image features from the image data. Some non-limiting examples of such image features are edge-based and and / or edge-related information, corners, blobs, ridges, scale-invariant feature transforms (S It may also include IFT (Inter-Frequency Transition) features, temporal features, etc.

[0042] In some embodiments, the image data is processed (e.g., as described in the methods, steps, etc., described herein). Parsing (by groups and modules) involves generating one or more rules, functions, actions, or artifacts. Neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms algorithm, behavior detection algorithm, motion detection algorithm, background subtraction algorithm, estimation Analyzing image data and / or preprocessed image data using a theoretical model, etc. Some non-limiting examples of such inference models include manually pre-programmed Programmed inference models, classification models, regression models, machine learning on training examples The results of training algorithms such as algorithms and deep learning algorithms are listed. where the training examples may include examples of data instances, Also, in some cases, the data instances may have corresponding desired labels and / or results. be labeled.

[0043] In some embodiments, the image data is processed (e.g., as described in the methods, steps, etc., described herein). The analysis (by the group and module) determines the pixels, boxes, etc. contained in the image data. This may include analyzing cells, point clouds, range data, etc.

[0044] FIG. 1 illustrates an exemplary operating room 101 consistent with disclosed embodiments. A patient 143 is , shown on the operating table 141. The operating room 101 is equipped with audio sensors, video / image sensors, and and other sensors, as well as video and audio data and other sensors during surgical procedures. Various light sources (e.g., light source 119 shown in FIG. 1) are used to facilitate the capture of data from the sensors. For example, the operating room 101 may include a video / image data One or more microphones (e.g., audio sensor 1 as shown in FIG. 1) for capturing audio data. 11), several cameras (e.g., overhead cameras 115, 121, 123, and table cameras Some cameras (e.g., camera 115, 123 and 125) on an operating table 141 (e.g., a camera is attached to the patient 14 on which the surgical procedure is performed). (capture video / image data at three body positions 127) While the camera 121 captures video / image data from other parts of the operating room 101, For example, the camera 121 may capture video / image data of a surgeon 131 performing a procedure. In some cases, the camera may capture anesthesia data placed in the operating room 101. Capture video / image data related to surgical team personnel such as surgeons, nurses, and surgical technicians In addition, the operating room camera can capture video / video related to medical equipment placed in the room. Image data can be captured.

[0045] In various embodiments, one or more of the cameras 115, 121, 123, and 125 are movable. For example, as shown in FIG. 1, the camera 115 may be configured to indicate the pitch direction. The camera 115 rotates as shown by arrow 135A and arrow 135B indicating the yaw direction of the camera 115. In various embodiments, the pitch and yaw angles of a camera (e.g., camera 115) can be calculated. The angle can be electronically controlled to point the camera 115 at a region of interest (ROI), It is necessary to capture video / image data of the cardiac region of interest (ROI). For example, the camera 115 , surgical instruments (also called surgical instruments) in position 127, anatomical structures, surgeon 131 It may be configured to track the movement of the hand, the incision, the anatomy, etc. In the camera 115, a laser 137 (e.g., an infrared laser) is used for accurate tracking. In some cases, the camera 115 may be equipped with a video / image data A computer using image recognition algorithms to position the camera to capture the data. They can be tracked automatically via a computer-based camera control application. The camera control application identifies anatomical structures and controls the hand at specific locations within the anatomical structures. Identify surgical instruments, surgeon's hands, bleeding, movement, etc., and adjust the camera 115 to the appropriate yaw angle and pitch. By rotating it by an angle, its position can be tracked by the camera 115. In this embodiment, the camera control application controls the various cameras 115, 121, 123. and 125 positions (i.e., yaw and pitch angles) to allow for different Video / image data can be captured from the ROI. Additionally or alternatively, The surgeon can control the position of the various cameras 115, 121, 123 and 125. and / or a human surgeon controls the camera position using a camera control application. Can be supervised.

[0046] The cameras 115, 121, 123, and 125 focus and zoom in on one or more ROIs. In an exemplary embodiment, the camera 115 may further include a zoom lens for capturing the image. for zooming in close to an ROI (e.g., a surgical tool near an anatomical structure). The camera 121 may include a zoom lens 138. The camera 121 may capture a larger area around the ROI. A zoom lens 139 can be included to capture video / image data from a wide area. For example, camera 121 may capture video / image data of the entire location 127 . In some embodiments, the video / image data acquired from the camera 121 may be used to The camera control application may then analyze the image to identify the ROI during the camera 115 is configured to zoom towards the ROI identified by the camera 121. It is possible.

[0047] In various embodiments, the camera control application controls the various cameras during a surgical procedure. It may be configured to adjust the position, focus, and magnification. For example, a camera control application The application can direct the camera 115 to track anatomical structures, The cameras 121 and 125 can be instructed to track the instrument. and 125 can track the same ROI (e.g., a surgical tool) from different viewing angles. For example, video / image data obtained from different viewing angles can be used to visualize the anatomy. Determine the position of the surgical instrument relative to the surface, determine the state of the anatomical structure, and Determine the pressure applied, or any other information for which multiple viewing angles may be useful As another example, bleeding can be detected by one camera and one or more Other cameras above can be used to identify the source of bleeding.

[0048] In various embodiments, the positions, orientations, settings, etc. of the cameras 115, 121, 123, and 125 and / or zoom control is rule-based and based on the algorithm developed for a given surgical procedure. For example, a camera control application can control a surgical instrument. Instruct camera 115 to track and aim camera 121 at position 127, capturing the image of the surgeon's hand. Instruct camera 123 to track movement and point camera 125 at anatomy. The algorithm may be configured to adjust the camera 115, Any suitable method for determining the position, orientation, setting and / or zoom of 121, 123 and 125. For example, an algorithm may be used to find a solution that will cause bleeding during a procedure. At least one camera can be directed at an area of ​​the anatomy (e.g., camera control Cameras 115, 121, 123 and Some non-limiting examples of 125 settings include image pixel resolution, frame rate, Image and / or color correction and / or enhancement algorithms, zoom, position, orientation, aspect ratio , shutter speed, aperture, and focus.

[0049] In various cases, a camera (e.g., camera 115) may capture a moving or deforming object (e.g., (where the camera 115 tracks a moving surgical instrument or a moving / pulsating anatomical structure) When tracking an object, the camera control application must ensure that a moving or deforming object does not move out of the camera's field of view. To prevent escape, a maximum allowable zoom of the camera 115 can be determined. In an embodiment, the camera control application initially selects the first zoom of the camera 115. and evaluating whether the moving or deforming object escapes from the field of view of the camera, and Adjust the camera zoom as needed to prevent objects from escaping the camera's field of view In various embodiments, the camera zoom can be adjusted based on the direction and speed of the moving or deforming object. can be readjusted based on

[0050] In various embodiments, one or more image sensors may include moving cameras 115, 121, 123 and The cameras 115, 121, 123 and 125 may be, for example, triangular cameras. Use surveying to determine the size of anatomical structures and to determine the distance between different ROIs For example, Figure 2 shows a case where the distance between two cameras is D1 as shown in Figure 2. An exemplary camera 115 (115 shown in FIG. 2) is supported by a movable element so as to ) and 121. Both cameras point towards ROI 223. Knowing the position and the orientation of the object relative to the camera (e.g., angles A1 and A2, as shown in Figure 2) For example, the same object or the same object in the images captured by 115 and 121 may be detected. By knowing the distance D2 D1 and D3 can be calculated using, for example, the law of sines and the known distance between the two cameras D1. In an exemplary embodiment, the camera 115 (115, view 2) If the camera rotates by a small angle A3 (measured in degrees) to face ROI 225, then ROI 22 The distance between 3 and ROI 225 is approximated by A3D2 (for a small angle A3) A different triangulation process can be used to obtain greater accuracy. Knowing the distance between 223 and 225 allows us to determine the length scale of the anatomical structure. Furthermore, it is possible to measure the distance between various points of the anatomical structure and the distance from various points to one or more points. By measuring the distance to the camera above, a point cloud representing the surface of the anatomy can be determined. Such point clouds can be used to reconstruct 3D models of anatomical structures. Additionally, the distance between one or more surgical instruments and different points on the anatomy can be measured. , the appropriate location of one or more surgical instruments near the anatomical structure can be determined. In some other examples, one or more of the cameras 115, 121, 123, and 125 may be 3D Cameras (e.g., stereo cameras, active stereo cameras, time-of-flight cameras, photodetectors) and ranging cameras) to capture actual and / or The relative positions and / or sizes and / or actual distances between objects are determined by a 3D camera. The decision can be made based on the captured 3D information.

[0051] Also, returning to FIG. 1, a light source (e.g., light source 119) may be configured to track one or more ROIs. In an exemplary embodiment, the light source 119 can be moved in yaw and pitch. It can be rotated by an angle, possibly towards the ROI (e.g., position 127). In some cases, the light source 119 may be located within the ROI. One or more optical elements (e.g., lenses, flat or curved mirrors, etc.) to focus the light In some cases, the light source 119 may include a light color (e.g., the light color may be different depending on the time). This may include a variety of white light, light with a selected spectrum, etc. In an exemplary embodiment, the light 119 may be configured such that the spectrum and intensity of the light is: The light may be configured to vary across the surface of the anatomical structure illuminated by the light. For example, in some cases, the light 119 may be applied to at least some portions of the surface of the anatomical structure. The radiation may include infrared wavelengths that can produce heat.

[0052] In some embodiments, the operating room may be embedded in various components that may or may not be shown in FIG. Examples of such sensors include audio sensors, image sensors, Sensors, motion sensors, positioning sensors, chemical sensors, temperature sensors, barometers, pressure sensors, proximity sensors, electrical impedance sensors, voltage sensors, current sensors, or 143, including any type of medical or physiological sensor configured to monitor the environment or

[0033] List any other detectors that can provide feedback regarding the surgical procedure. It is possible.

[0053] In some embodiments, the audio sensor 111 converts audio into digital information (e.g., one or more audio sensors configured to capture audio by converting it into It may also include a

[0054] In various embodiments, the temperature sensor is connected to an infrared camera for infrared imaging (e.g., infrared The infrared camera 117 may include a This allows for the measurement of the surface temperature of the anatomical structure at different points in the structure. Similar to the cameras D115, 121, 123 and 125, the infrared camera 117 is yaw-angled or Additionally or alternatively, the camera 117 can be rotated using a pitch angle. The imaging system may include an image sensor configured to capture images from any light spectrum. , infrared imaging sensors, hyperspectral imaging sensors, etc.

[0055] FIG. 1 shows views from different cameras 115, 121, 123 and 125, as well as other information. For example, the display screen 113 may include a display screen 113 capable of displaying information about the surgical instrument. It may show a zoomed-in image of the tip and surrounding tissue of the anatomical structure proximate to the surgical instrument. can.

[0056] FIG. 3 illustrates an exemplary implementation of a surgical tool 301 that may include multiple sensors and light sources. Consistent with this embodiment, the surgical instrument may be a medical device, a medical instrument, an electrical or refers to mechanical instruments, surgical instruments, diagnostic instruments, and / or any other means that may be used during surgery As shown, the fixture 301 includes cameras 311A ​​and 311B, a light source 311C, and a 13A and 313B, and tip portions 323A and 323B for contacting tissue 331. The cameras 311A ​​and 311B may include data connections 319A and 319B. In an exemplary embodiment, the device The device 321 transmits data to a data receiving device using wireless communication or using wired communication. In an exemplary embodiment, the device 321 can transmit data via WiFi, Bluetooth, NFC communication, inductive communication, or to transmit data to a data receiving device Any other suitable wireless communication for transmitting data may be used. It may include any form of receiver capable of receiving a transmission. Alternatively, the device 321 may transmit data to a data receiving device using optical signals. (For example, device 321 may use optical signals transmitted over air or optical fiber. In some embodiments, the device 301 includes a sensor 311A ​​and 311B。 311B. The local memory for storing at least a portion of the data received. Furthermore, the device 301 may perform video / audio processing before transmitting data to the data receiving device. A processor for compressing the image data may be included.

[0057] In various embodiments, for example, if device 301 is wireless, the device may have an internal power supply. a power source (e.g., battery, rechargeable battery, etc.) and / or a power source for recharging the battery a power supply, an indicator to indicate the amount of power remaining for the power supply, and a It can contain one or more input controls (e.g., buttons) for controlling the In some embodiments, control of device 301 may be via any suitable connection (e.g., WiFi, B An external device (e.g., a smartphone) that communicates with the device 301 via Bluetooth, etc. This can be achieved using a variety of devices, including smartphones, tablets, and smart glasses. In this example, the input controls of the device 301 are used to control various parameters of the sensor or light source. For example, the input controls can be used to control the light sources 313A and 313B. If the light source can be moved (e.g. the light source is moved by a yaw angle), and pitch angle) to control the color of the light source. If you can control the focus of the light source, move the camera (e.g., the camera is (which can be rotated using yaw and pitch angles) to cameras 311A ​​and 311B control the movement of the camera 311A ​​and 311B, and control the zoom and / or capture parameters of the cameras 311A ​​and 311B. or any other suitable control of the cameras 311A-311B and light sources 313A-313B. The camera 311A ​​has a first set of parameters. The camera 311B may have a second set of parameters different from the first set of parameters. These parameters may be selected using the appropriate input controls. Similarly, light source 313A has a first set of parameters. Alternatively, light source 313B may have a second set of parameters different from the first set of parameters. These parameters may be selected using the appropriate input controls. good.

[0058] Additionally, the instrument 301 can be adapted to various properties of the tissue 331 via the tips 323A and 323B. and configured to measure data relating to the device and transmit the measured data to the device. For example, the tips 323A and 323B may be used to measure the electrical resistance and / or Alternatively, impedance, temperature of tissue 331, mechanical properties of tissue 331, etc. may be measured. To determine the elastic properties of the tissue 331, for example, the distal ends 323A and 323B The tip may be first separated at angle 317 and applied to tissue 331. 7, and the movement of the tip Such pressure can be measured (e.g., by measuring the pressure of the first via a piezoelectric element 327 that may be located between the first branch 312A and the second branch 312B Based on the change in angle 317 (i.e., strain) and the measured pressure (i.e., stress), 317, the elastic properties of the tissue 331 can be measured. The distance between the sections 323A and 323B can be measured and this distance can be recorded in the device 321. Such distance measurements can be transmitted from various cameras 11, as shown in FIG. The various video / image data lengths that can be captured by the 5, 121, 123 and 125 It can be used as kale.

[0059] Instrument 301 is only one example of a possible surgical instrument, and may include a scalpel, a grasper (e.g., forceps), a clamp, a Lamps and articulators, needles, retractors, cutters, dilators, suction tips and tubes, sheaths Other surgical instruments such as ring devices, irrigation and injection needles, scopes and probes are optional. In various cases, the sensor and light source may include a suitable sensor and light source. The type of surgical instrument used in the surgical procedure may depend on the type of instrument. These other surgical instruments may collect data and generate any suitable data, as shown in Figure 3. The device may include a device similar to device 301 for transmitting data to a receiving device. .

[0060] When preparing for a surgical procedure, the surgeon may view video footage of a surgical procedure with similar surgical events. Reviewing the images can be beneficial; however, it can be time-consuming for surgeons to view the entire video. It takes too long or it takes too long to skip around and find the relevant part of the surgery video. Therefore, it is important to record images of relevant surgical events while omitting other unrelated images. Enabling surgeons to efficiently and effectively view surgical video summaries that aggregate images Non-traditional approaches are needed.

[0061] Aspects of the present disclosure include methods, systems, devices, and computer-readable media. It can be about reviewing videos. The interface allows the surgeon to see the surgical timeline. simultaneously view and review surgical videos (your own, others' surgeries, or compilations) The timeline may allow for actions or events that occur during a surgical procedure. These markers can be used to indicate when a surgeon is performing a particular action. This allows for skipping, thereby streamlining surgical procedure review. In some embodiments, it is possible to mark important decision junction points. The surgeon can see the different actions taken at those decision junction points. It is possible.

[0062] For ease of description, aspects of the method may be described as systems, devices, and computer-readable media. The methods are described below with the understanding that they apply equally to the body. For example, such methods Some aspects of the present invention may be performed electronically over wired, wireless, or both networks. Other aspects of such methods may be performed using non-electronic means. In this regard, the method is not limited to any particular physical and / or electronic means, but rather may be implemented using many This can be achieved using different means.

[0063] Consistent with disclosed embodiments, a method may include accessing at least one video of a surgical procedure. As explained in more detail above, the video may be recorded. This may include any form of recorded visual media containing recorded images and / or sound. Videos include, for example, Audio Video Interleaved (AVI) files, Flush Video Format (FLV) files, QuickTime file formats Movie (MOV), MPEG (MPG, MP4, M4P, etc.), Windows Me dia Video (WMV) file, Material Exchange Format ( MXF) files, or any other suitable video file format It can be stored as a file.

[0064] A surgical procedure is any procedure that involves or includes manual or operative manipulation of a patient's body. Surgical procedures may include medical procedures such as cutting, abrading, suturing, or physical alteration of body tissues and organs. Examples of such surgical procedures are provided above. Video of a surgical procedure is any video captured during a surgical procedure and associated with the surgical procedure. In some embodiments, the surgical procedure may include a series of still images. may be depicted in one or more of the still images included in the video. For example, The video is recorded by an image capturing device such as a camera in the operating room or in the patient's cavity. Accessing the video of the surgical procedure may be performed using a storage device (one or more memory units). video server, cloud storage platform, or any other storage platform to extract video from another device via a communication device, receiving signals, capturing video using an image sensor, or transmitting data or files This may include any other means for electronically accessing files.

[0065] Some aspects of the present disclosure include outputting at least one video for display. Outputting at least one video means that the video is output to a computer or at least may also include any process that is generated, distributed, or served using a single processor. As used herein, a "display" refers to a device that displays video to a user for playback. In some embodiments, the video Output to a screen (e.g., OLED, QLED LCD, Plasma, CR T, DLPT, e-paper, or similar display technologies), optical projectors (e.g. , movie projectors, slide projectors), 3D displays, mobile devices a display such as a computer screen, electronic glasses, or any other form of visual and / or audio presentation. In other embodiments, the video may be presented using a play device. Outputting the video for display may be performed by one or more other computing devices. Such storage location may include storing the video in a location accessible by the user. These include local storage (such as flash memory hard drives), network Location (e.g., server or database), cloud computing platform, or any other accessible storage location. Accessible from a separate computing device for display on a In some embodiments, outputting the video may include outputting the video to an external device. For example, outputting a video for display may involve transmitting it to a user. transmitting the video over a network to a user device for playback on the user device. It can be done.

[0066] An embodiment of the present disclosure provides at least one output for displaying a surgical timeline. It may further include overlaying it on a video. A "timeline" may refer to any depiction in which a sequence of events may be tracked or defined. In some embodiments, the timeline may include, for example, markers or other indicators of events along a bar. A graphical representation of an event may be used, using a thin bar or line representing time with a mark. A timeline is a text-based list of events arranged in chronological order. A surgical timeline may be a timeline depicting events related to a surgical procedure. By way of example, the surgical timeline may include the timelines generated during the surgical procedure, as detailed above. In some embodiments, the timeline may be a timeline of events or actions that occur during surgery. The lines may contain text information identifying parts of the surgical procedure. For example, A timeline may be a descriptive list of intraoperative events or operative stages within a surgical procedure. In other embodiments, the user may hover over a graphic marker on the timeline or other By activating the method, the descriptor associated with the marker may be displayed.

[0067] Overlaying the surgical timeline onto at least one video is Includes an optional way to display the surgical timeline so two videos can be viewed simultaneously In some embodiments, overlaying the video may include adding a small amount of This may include displaying a surgical timeline that overlaps at least partially with the surgical timeline. For example, a surgical timeline may appear as a horizontal bar along the top or bottom of the video, or as a line along the side of the video. In other embodiments, the overlaying may be presented as a vertical bar along the , including presenting a surgical timeline along with the video. For example, the video , with the surgical timeline presented above, below, and / or to the side of the video on the display A surgical timeline may be displayed over the video while it is playing. Therefore, "overlay" as used herein refers to the more general Generally, it refers to simultaneous display. Simultaneous display may or may not be constant. For example, The video output must appear before the end of the surgical procedure depicted in the displayed video. Alternatively, the overlay may appear during substantially all of the video processing. do.

[0068] FIG. 4 is an exemplary illustration overlaid on a video of a surgical procedure consistent with disclosed embodiments. 4 shows a typical timeline 420 for displaying one or more frames of video in sequence. In the example shown in FIG. The line 420 may be displayed as a horizontal bar representing time, with the leftmost portion of the bar representing the video The rightmost part of the bar represents the start time of the timeline, and the rightmost part of the bar represents the end time. The time may include a position indicator 424 that indicates the current playback position of the video relative to the time input. The colored areas 422 of the line 420 may represent progress within the timeline 420 ( For example, a video already viewed by the user or before the currently presented frame In some embodiments, the location indicator 424 may be used to indicate when a user selects a location indicator. Just as you can move to different positions in the video by moving the 424, In some embodiments, the surgical timeline may be interactive. Markers identifying at least one of the following: floor, intraoperative surgical event, and decision junction. For example, the timeline 420 may include one or more markers 432, 433. 4, and / or 436. Such markers are described in more detail below. It will be described.

[0069] In the example shown in FIG. 4, the timeline 420 represents the video either physically, temporally, or both. In some embodiments, the timeline may be displayed overlaid on the video playback area 410. The timeline 420 does not have to be displayed all the time. You can automatically switch to a collapsed or hidden view while watching a video. When the user takes an action to interact with the timeline 420, the expanded view shown in FIG. For example, a user can move the mouse pointer while watching a video. and move the mouse pointer over the collapsed timeline to select a specific area. Navigate to the timeline 420 area, click or tap on the video playback area, or interact with the timeline 420. Any other action that can indicate an intention to use the device may be performed. The timeline 420 is located above the video playback area 410, or above the video playback area 410. at various other locations relative to the video playback area 410, including above, below, or within the control bar 612. In some embodiments, the timeline 420 may be a video progression bar. For example, a separate display including a position indicator 424 and a colored region 422 may be used. A video progress bar may be displayed in the control bar 412, and a timeline 420 may be displayed In such an embodiment, the timeline may be a separate timeline of events related to the treatment. The timeline 420 does not have the same scale or time range as the video or video progress bar. For example, a video progress bar can represent the time scale and range of the video. However, the timeline 420 may represent the time frame of a surgical procedure, which may not be the same. (e.g., if the video includes a surgical summary, as explained in detail above). In some embodiments, the video playback area 410 may be, for example, as described above with reference to FIG. Allowing a user to search for video footage via the user interface 700 The surgical timeline shown in FIG. These are provided as examples only, and one skilled in the art will recognize various other configurations that may be used.

[0070] An embodiment of the present disclosure provides a surgical timeline for a surgeon to view at least one video playback. Allows you to select one or more markers on the screen, thereby selecting the video to display. The method may further include skipping to a position associated with the selected marker. When used, "playback" means the start of a video in which one or more frames of the video are displayed to the user. It can include any presentation. Typically, playback involves the use of images to play video and / or audio. Although playback may involve sequentially displaying frames, playback may also involve displaying individual frames.

[0071] Consistent with the disclosed embodiments, a "marker" is associated with a location within a surgical timeline. As mentioned above, the location can be any visual indicator within the video. It can refer to a specific location. For example, the location can refer to a specific frame or frames within a video. This may be a range of frames, a specific timestamp, or any other indicator of a position within the video. Markers can be represented on the timeline in a variety of ways. In some embodiments, markers are icons or other graphical indicators that appear at various positions along the timeline. Markers may be lines, bands, dots, geometric shapes (e.g., diamonds, squares, etc.). , triangle, or any other shape), bubble, or any other graphic or visual representation In some embodiments, the marker may be text-based. For example, a marker may contain text information such as a name, description, code, and timestamp. In another example, as described above, the surgical timeline may be displayed as a list. Thus, a marker may be a text-based tag that references a specific location in the video. Markers 432, 434, and 436 are shown in FIG. Markers are shown in the callout with an icon indicating the type of marker associated with the location. The markers may be represented as bubbles in the timeline 420 that indicate positions within the video. It can refer to a specific point along the

[0072] Marker selection may include any action by the user directed at a particular marker. In some embodiments, selecting a marker involves selecting the marker via a user interface. Clicking or tapping on the marker, touching the marker on the touch-sensitive screen The marker can be glanced at through smart glasses, and the marker can be accessed through a voice interface. pointing at the marker, gesturing to the marker, or any other action that causes the marker to be selected. This can include performing a process whereby selecting a marker changes the video display. Skip to the position associated with the selected marker. Skip can involve selectively displaying particular frames within the video. means to stop displaying the frame at the current position in the video (e.g., the video is currently playing If the frame is in the selected position, the frame is displayed at the position associated with the selected marker. For example, as shown in FIG. Once selected, the frame at the position associated with the marker 432 is displayed in the video playback area 410. In some embodiments, the video may continue to play from that point. The position indicator 424 moves to the position in the timeline 420 associated with the marker 432. The colored region 422 can be updated accordingly. Although described as allowing the physician to select one or more markers, It is understood that this is merely an example and that the present disclosure is not limited to any form of user. Technologists, nurses, physician assistants, anesthesiologists, physicians, or any other healthcare professionals; Various other users, including patients, insurance companies, medical students, etc., can then view the overlaid timeline. Users can view and interact with the in- Other examples of users are provided herein.

[0073] According to an embodiment of the present disclosure, the markers are generated based on information in the video at a predetermined location. In some embodiments, the computer-generated timeline may be automatically generated and included in the timeline. Use analysis to analyze frames of video footage and select markers to include at various positions in the timeline. Computer analysis can identify This can include any form of electronic analysis. In some embodiments, computer analysis is , one or more image recognition algorithms to identify features in one or more frames of video footage; The computer analysis may include using a computer algorithm. For example, multiple frames may be used to detect motion or other changes between frames. In some embodiments, the computer analysis may include: Iola-Jones object detection, Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradient HOG features, Convolutional Neural Networks (CNN), or any other form Other exemplary object detection algorithms include the object detection algorithms Examples of algorithms include video tracking algorithms, motion detection algorithms, and feature detection algorithms. Algorithms, color-based detection algorithms, texture-based detection algorithms, Shape-based detection algorithm, boosting-based detection algorithm, face detection algorithm algorithm, or any other suitable algorithm for analyzing video frames. In one example, a machine learning model can be trained to generate markers for a video. The trained machine learning model may be trained using the business model examples. The generated video may be used to analyze the video and generate markers for that video. The markers may include the marker's location in the video, the marker type, and the marker's characteristics. An example of such a training example is In addition, various aspects such as the marker's position in the video, the marker's type, and the marker's characteristics may be specified. Include a video clip showing at least a portion of the surgical procedure along with the desired marker information. can be done.

[0074] This computer analysis may include surgical steps, intraoperative events, event characteristics, and / or events occurring on video footage. For example, in some embodiments, the computer The data analysis may be used to identify one or more medical devices used in a surgical procedure, for example, as described above. Based on the identification of the medical device, the medical device can be identified. The location of the scalpel or other instrument can identify specific intraoperative events. A marker identifying the incision will appear on the timeline at this location. In some embodiments, computer analysis may be used, for example as described above. For example, the disclosed method can be used to identify anatomical structures in video footage. Identify the person's organs, tissues, fluids, or other structures and display a timeline and their respective locations. In some embodiments, determining the location of the video markers may include determining the markers to include in the location of the video markers. The placement can be determined based on the interaction between the medical device and the anatomy, and The intraoperative event, the type of surgical procedure, the event characteristics, or other information useful in identifying the marker location. For example, visual action recognition algorithms can be used to analyze the video. The markers can be placed to detect interactions between medical devices and anatomical structures. Other examples of features that may be detected in video footage to identify the presence of a surgeon or other medical professional the movements of the patient, the characteristics of the surgeon or other medical professional, the sequence of the action being performed sequences, timing of actions or events, characteristics of anatomy, medical conditions, or specific external To identify surgical procedures, surgical stages, intraoperative events, and / or event characteristics appearing in video footage Any other information that can be used for the purpose of the present invention may be mentioned.

[0075] In some embodiments, the marker locations are determined using a trained machine learning model. For example, a machine learning model can be trained using training examples. Each training example is then stored in a separate file along with a label indicating its location in the video. Videos known to be associated with surgical procedures, surgical stages, intraoperative events, and / or event characteristics It can also include video footage. It uses trained machine learning models to detect marker positions. Similar stages and events can be identified in other video footage to determine the Logistic regression model, linear regression model, regression model, random forest model, k-nearest neighbor (KNN) model, k-means model, decision tree, Cox proportional hazards regression model , Naive Bayes model, Support Vector Machine (SVM) model, Gradient Boosting algorithms, artificial neural networks (e.g., deep neural networks) neural networks, convolutional neural networks, or any other form of machine learning model A variety of machine learning models can be used, including algorithms.

[0076] In some embodiments, video markers are used in conjunction with the video indexing techniques described above. As mentioned above, the video footage can be recorded using the hand signals identified within the video footage. This information may be indexed based on surgical stage, intraoperative events, and / or event characteristics. may be stored in a data structure such as data structure 600 as described with reference to FIG. The data structure can store video positions and / or events associated with stages and events within the video footage. In some embodiments, the markers displayed on the timeline may include: These positions in the video can then be indexed. Any of the above techniques or processes for marking a timeline may be used to mark a timeline for presentation. The same can be applied to determining the position of the target.

[0077] According to various exemplary embodiments of the present disclosure, the marker may be at least one of a color or a critical level. The coding of a marker may be determined by the type, characteristics, or any indicator of a characteristic. The coding determines which locations in the video are of interest. This can be useful for the user to visually determine whether the marker is correct. When the surgery timeline is configured, the marker colors displayed in the surgery timeline will be based on a predefined color scheme. For example, a marker can be characterized based on the Different types of intraoperative surgical events can have different colors. In this state, markers associated with incision, excision, resection, ligation, grafting, or various other events may be used. In other embodiments, adverse intraoperative events may be displayed in one color (e.g., A planned event may be associated with a different color (e.g., green). In some embodiments, a color scale can be used. For example, the severity of an adverse event may be indicated by a color scale ranging from yellow to red, or by other appropriate color scale. It can be expressed in kale.

[0078] In some embodiments, the location and / or size of the marker is associated with a critical level. The criticality level is the level of the event, action, technique, step, or other occurrence identified by the marker. Therefore, as used herein, "criticality" can represent the relative importance of The term "level" refers to the immediate need for action to prevent dangerous outcomes within a surgical procedure. For example, a critical level can be a numerical indicator within a certain range of values ​​(e.g. , "1.12", "3.84", "7", "-4.01", etc.) In another example, the critical level may be a finite number of discrete levels (e.g., "Level 0", "Level 1", "Level 2"). "Level 1," "Level 2," "High Critical," "Low Critical," "Non-Critical," etc. can.

[0079] Although colors are provided as an example to differentiate the appearance of markers to represent information, various Various other techniques may be used. For example, markers may be of various sizes, shapes, positions, orientations, etc. , font size, font type, font color, marker animation, or other visual In some embodiments, the markers may have the properties that they are associated with. Depending on the type of event, action, or step being performed, different icons may be associated with it. For example, as shown in FIG. 4, a marker 432 may be associated with a decision junction. may have a different icon than the marker 434, which may be associated with another type of event, such as The icon can represent the type of intraoperative event associated with that location, for example , a marker 436 can indicate that an incision was made at this location in the video. Cons (or other visual characteristics) are used to distinguish between unplanned and planned events, and between errors. the type (e.g., miscommunication error, judgment error, or other form of error), the specific error that occurred; Adverse events, type of technique performed, surgical stage performed, location of intraoperative surgical events (e.g., location in the abdominal wall), the surgeon performing the procedure, the outcome of the surgical procedure, or can be used to distinguish various other information.

[0080] In some exemplary embodiments, the one or more markers are used to determine the decision making process for a surgical procedure. In some embodiments, the decision junction markers may include decision junction markers corresponding to the decision junctions. So, such decision junction markers are visually distinct from other forms or types of markers. As an illustrative example, a decision junction marker can be As shown in Figure 4 by Ka432, the location is associated with the decision junction. As used herein, decision making may have an icon indicating that the A junction is a procedure where a decision is made, or a type of decision or multiple choices are made. It can refer to any part of the process where a decision of the selected type is made. For example: The decision-making junction marker allows multiple courses of motion, allowing the surgeon to select one course. You can indicate the location of a video depicting a surgical procedure where you choose to follow a different course. For example, a surgeon may decide to withdraw from a planned surgical procedure, take precautions, or or whether tissue is removed, or whether specific instruments are used, or whether specific surgical techniques are used. or any other intraoperative decision that a surgeon may face. A decision junction is a part of a procedure where a decision is made that has a significant impact on the outcome of the procedure. In another example, a decision junction may have no clear decision direction. It can refer to a part of a procedure where a new decision must be made. A fixed junction is when the surgeon is faced with two or more viable alternatives and The better alternative among the available alternatives (e.g., an alternative that is predicted to reduce a particular risk) alternatives that are predicted to improve outcomes, alternatives that are predicted to reduce costs, etc. It is recommended to select at least a certain number of factors (e.g., at least two factors, based on at least 5 factors, at least 10 factors, at least 100 factors, etc. In a further example, a decision junction can refer to a part of a procedure based on the surgeon's faces a particular type of decision, and the particular type is included in the group of selected decision types. It can refer to part of the procedure that is performed.

[0081] Decision junctions can be detected using the computational analysis described above. In some embodiments, video footage may be analyzed to indicate that a decision has been made. The specific actions or sequences of actions performed by the surgeon can be identified. For example, if the surgeon pauses during the procedure or starts using a different medical device , or change to a different course of action, this may indicate that a decision has been made. In this embodiment, the decision junction is a surgical stage identified in the video footage of the location. It can be distinguished based on stage or intraoperative events, for example to detect adverse events such as bleeding. This means that decisions must be made about how to deal with adverse events. As another example, a particular stage of a surgical procedure may be associated with multiple possible courses of action. Therefore, detecting this surgical stage in the video footage can be helpful in decision-making. In some embodiments, the trained machine Learning models can be used to identify decision junctions. A machine learning model is used to detect decision junctions within the dataset. You can train a machine learning model to analyze video. We can analyze the training example to detect decision junctions. An example of this is a video clip with a label indicating the location of the decision junction, or Video clips with labels indicating the absence of decision junctions in the video clips may include:

[0082] The selection of decision-making junction markers is important when the surgeon has two or more corresponding other surgical procedures. It may be possible to view two or more separate video clips from Allows the viewer to compare different techniques. Different video clips are currently being presented to the user. It may also be any video clip showing a procedure other than the one shown. An alternative is to extract the image from other video footage that is not included in the current video being output for display. Alternatively, if the current video footage contains a compilation of different actions, The alternate footage may be drawn from a different position in the currently displayed video footage. Medical procedures are not permitted other than the specific procedure shown in the current video output for viewing. It may be any surgical procedure. In some embodiments, other surgical procedures may be performed for indication. The surgical procedure may be the same type as that shown in the video being output for , performed at different times, in different patients, and / or by different surgeons. In some embodiments, the other surgical procedure may not be the same type of procedure, and may be a decision junction. share the same or similar decision junctions as those identified by the decision markers. In some embodiments, two or more video clips may depict different actions. For example, two or more video clips can be separated by a decision junction marker. This can represent a choice of actions other than those taken in the current video, as represented by do.

[0083] The separate video clips can be presented in a variety of ways. In some embodiments, When you select a decision junction marker, two or more separate video clips are automatically For example, one or more of the other video clips may be displayed in the video playback area 410. In some embodiments, the video playback area may display another video clip. and / or may be split or divided to show one or more of the current videos. In some embodiments, the other video clip may be positioned above, below, or to the side of the video playback area 410. In some embodiments, the other video clip may be displayed in a different area of ​​the second It may be displayed in a window, on another screen, or in any other space other than the playback area 410 . According to another embodiment, selecting the decision marker selects a button to view another video clip. It can open a menu or otherwise display options, e.g., a decision command. Selecting a name marker will take you to another video clip that contains a depiction of the action in another related video clip. Another video clip can be displayed as a thumbnail, text-based description of the content, video review (e.g., playing a lower resolution version or shortened clip) The menu may be overlaid on the video. , may be displayed in conjunction with the video or in a separate area.

[0084] According to an embodiment of the present disclosure, the selection of a decision junction marker determines the selected decision. Causes the display of one or more alternative possible decisions associated with the decision junction marker As with other videos, other possible decisions can be made to change the timeline and / or video. It may be overlaid on the video, or in separate areas such as above, below and / or to the side of the video. may be displayed in a separate window, on a separate screen, or in any other suitable manner. The alternative possible decisions are the alternative decisions that the surgeon could have made at the decision junction. The list may also include images (e.g., showing alternative actions), flow charts, ,statistics (e.g., success rate, failure rate, ,usage rate, or other statistical information), detailed description, ,hyper Links or other related to other possible decisions that may be relevant to the surgeon viewing the playback. Such a list may be interactive, allowing the viewer to select from the list Allows you to select a different course of action, thereby displaying video footage of that course of action It can be done.

[0085] Additionally, in some embodiments, one or more of the following may be used to determine whether the decision is related to one or more other possible decisions: The above probable outcomes can be displayed together with one or more other possible decision displays. For example, a list of alternative decisions can be generated by adding a probable outcome for each of the alternative decisions. The predicted outcomes may include what would have happened if the surgeon had made different possible decisions. Such information may be useful for training purposes. For example, the surgeon may decide that he or she could have taken a more appropriate action than the video. In some embodiments, the patient's condition can be monitored and future treatment can be planned accordingly. Each of the alternative possible decisions may be associated with a number of probable outcomes, each of which A probability may be provided. The one or more predicted outcomes may be determined in a variety of ways. In some embodiments, the predicted outcomes are based on known probabilities associated with alternative possible decisions. For example, a patient may have a history of a previous surgical procedure with a similar decision junction. The aggregate data can be used to predict the outcomes of different possible decisions related to the marker. In some embodiments, the probabilities and / or data may be based on one or more of the current surgical procedures. For example, the patient's characteristics (such as the patient's medical condition, age, weight, medical history, or other characteristics), the surgeon's skill level, The difficulty of the procedure, the type of procedure, or other factors may be taken into account. Other characteristics can also be analyzed, including the event characteristics described above with respect to sequencing.

[0086] According to the present disclosure, a surgical procedure decision junction is associated with a first patient. Each similar decision junction may have similar characteristics to the first patient. The surgical procedure may be selected from a list of previous surgical procedures associated with the patient. Based on each similar decision junction and similar estimated outcome, or Pre-selected or automatically selected due to similarities between the patient and patients from previous surgical procedures. These similarities or characteristics may include the patient's sex, age, weight, height, etc. , physical fitness, heart rate, blood pressure, temperature, whether the patient exhibits a particular medical condition or disease, medical Medical history, or any other traits or conditions that may be relevant, may be mentioned.

[0087] Similarly, in some embodiments, the surgical procedure decision junction is a first medical Each similar past decision junction may be associated with an expert. From previous surgical procedures associated with healthcare professionals with similar characteristics to healthcare professionals of 1 These characteristics may include the age, medical background, and experience level of the healthcare professional. (e.g., the number of times the surgeon has performed this or similar surgical procedure, the number of surgical procedures the surgeon has performed, (e.g., total number of surgical procedures), skill level, training history, success rate of this or other surgical procedure, or other characteristics that may be relevant include, but are not limited to:

[0088] In some exemplary embodiments, the surgical procedure decision junction is The same past decision junction associated with the first prior event in In one example, the prior event is selected from previous surgical procedures that include a prior event similar to the prior event. The event may be classified as a first antecedent event based on, for example, the type of antecedent event, the characteristics of the antecedent event, etc. They may be determined to be similar, for example, if the similarity index between them exceeds a selected threshold. If the prior event is greater than the first prior event, the prior event may be determined to be similar to the first prior event. Some non-limiting examples of antecedent events are described above. The occurrence and / or characteristics of antecedent events may be different. For example, the surgeon may be involved in determining the probable outcome of a possible decision. When faced with a disease, comorbidities can at least partially determine the most appropriate outcome. However, in the absence of complications, a different outcome may be appropriate. The first antecedent event is the This may include, but is not limited to, any of the intraoperative events described. Some non-limiting characteristics of the elephant may include any of the event characteristics described above. For example, the primary antecedent events are bleeding, mesenteric emphysema, injury, and unplanned conversion to open. , incision significantly larger than planned, hypertension, hypotension, bradycardia, hypoxemia, adhesions, Adverse events or complications such as hernia, atypical anatomy, dural tear, peritoneal injury, or arterial occlusion The first prior event may include successful incision, administration of medication, Use of surgical instruments, excision, resection, ligation, grafting, suturing, stitching, or any other event, etc. Positive or planned events may also be mentioned.

[0089] According to the present disclosure, the decision-making junction for a surgical procedure is associated with a medical condition. Each similar decision junction may be a group of patients with similar medical conditions. The medical condition may be selected from the patient's health or a previous surgical procedure associated with the condition. In some embodiments, the medical condition of the patient may include any condition of the patient that is related to well-being. The condition can be a condition that is being treated by a surgical procedure. The condition can be a distinct medical condition. A medical condition can be determined in a variety of ways. In some embodiments, the medical condition is determined based on data associated with the plurality of images. For example, a video may be tagged with information including a medical condition. In other embodiments, the medical condition is determined by analysis of at least one video. and can be based on the appearance of an anatomical structure appearing in at least one video. For example, the color of the tissue, the relative color of one tissue to the color of another tissue, the size of the organ, the size of another tissue, the relative size of one organ to the size of the others, the appearance of the gallbladder or other organs, lacerations or The system analyzes the presence of other marks, or any other visual indicators related to the anatomy, to identify the disease. In one example, a machine learning model can determine a medical condition from a video. The trained function may be trained using training examples to The machine learning model analyzes at least one video footage and is used to determine a medical condition. An example of such a training example is a set of labels indicating one or more medical conditions. Along with the video, a video clip of the surgical procedure can be included.

[0090] In some aspects of the present disclosure, at each similar past decision junction, Information about the distribution of past decisions made, along with a representation of alternative possible decisions, For example, as mentioned above, a particular decision junction may be displayed as a sequence of actions. Past decisions may be associated with multiple possible decisions regarding the same or similar intentions. performed by the surgeon in a previous surgical procedure when facing a decision junction For example, each of the past decisions can be a different possible decision as described above. Therefore, as used herein, Each similar past decision junction represents the past surgical decision when the past decision was made. In some embodiments, each decision junction The past decision junctions of the same kind are the decision junctions identified by the markers. For example, if the decision junction is an adverse event such as bleeding, In this case, past decisions may be based on how other surgeons have dealt with bleeding in previous surgical procedures. In other embodiments, the decision junctions may not be identical. For example, a possible procedure performed by a surgeon facing a dural tear may be The decision to do so may be similar to other forms of lacerations and therefore should be based on past experience associated with dural tears. The distribution of past decisions may relate to other forms of lacerations. They can be identified by analyzing video footage using data analysis techniques. In some embodiments, past decisions can be easily accessed to display the distribution of past decisions. The video indexing techniques described above are used to index the video so that it can be accessed. In one example, a distribution can be a set of distributions that share a common characteristic, e.g. A condition that presents the distribution of past decisions made at similar past decision junctions. In another example, the distribution may include a conditional distribution, e.g., a distribution of all similar past intentions. It includes unconditional distributions that provide the distribution of past decisions made at a decision junction. This can be done.

[0091] The distribution shown shows how each possible decision is related to each similar past decision. How common were the other possible decisions associated with the transaction? In some embodiments, the displayed distribution may show the number of times each decision was made. For example, a particular decision junction may contain Decision A, Decision B, and Decision C. We can have three alternative possible decisions for a given C. Similar decision junctions Each of these alternative decisions was made based on previous decisions made in For example, if decision A is made 167 times and decision B is made 47 times, , decision C may be made 13 times. The distribution is a list of each of the alternative possible decisions. The displayed distribution may also be displayed, for example, Each of the decisions can be displayed as a ratio, percentage, or other statistical information. For example, a distribution can be used to show the relative frequency of decisions A, B, and C, respectively. It can be shown that this was done in 73.6%, 20.7%, and 5.7% of the cases. In this embodiment, the distribution may be a bar graph, a histogram, a pie chart, a distribution curve, or any other graph showing the distribution. As a graphical representation of a distribution, such as any other graphical representation that may be used to It can be displayed.

[0092] In some embodiments, only a subset of the decisions may be displayed. For example, the most common Only relevant decisions may be displayed based on the number of times the decision was made (e.g., more than a threshold number of times). Related to similar medical conditions, patient characteristics, health care professional characteristics, and / or prior events To identify similar past decision junctions, including identifying surgical procedures The various methods described above can be used.

[0093] In some embodiments, one or more predicted outcomes are associated with each similar decision junction. It may be the result of analysis of multiple videos of previous surgical procedures, including The image repository can be used for various computational purposes, such as the object and / or motion detection algorithms mentioned above. The data is analyzed using data analysis techniques, thereby determining the decision makers identified by the markers. Identify videos that contain decision junctions that share the same or similar characteristics as a decision junction. This allows you to view the decision junctions in the video presented in the timeline. The same or similar surgical steps, intraoperative events, and / or Another possible method is to identify other video footage that has the event characteristics. The outcome of the decision can be estimated based on the outcomes of previous surgical procedures. If a particular method of performing the procedure consistently results in complete recovery by the patient, this outcome is may be estimated for possible determinations and displayed on a timeline.

[0094] In some exemplary embodiments, the analysis includes analyzing the implementation of a computer vision algorithm. The computer vision algorithm may include the use of the computer vision algorithms described above. Such computer algorithms may be the same as or similar to any of the An example of such a algorithm may include the object detection and tracking algorithm described above. Another example of a computer vision algorithm is the use of trained machine learning models. Other non-limiting examples of such computer vision algorithms include: For example, the specific time at which a decision junction marker occurs in the video is If identified based on an adverse event, other video footage with the same or similar adverse event will be included. Video footage can determine the outcome of decisions made in past surgical videos. This may be further analyzed to determine the identity of the individual or group of individuals, using the same or similar computer analysis methods as described above. In some embodiments, this may include techniques for identifying the outcome of a decision. This may include analyzing the video. For example, the decision junction may be a tear or other If an adverse event is related to an anatomical structure, the anatomical structure must be identified and the adverse event resolved. whether the patient recovered, how quickly they recovered, whether further adverse events occurred, and whether the patient survived. The decision is evaluated at various frames after the decision to determine whether the decision was successful or other indicators of outcome. It is possible.

[0095] In some embodiments, additional information may also be used to determine the outcome. The analysis is based on one or more electronic medical records associated with multiple videos of previous surgical procedures. For example, determining the outcome can be determined by whether the patient recovered or not. Certain decisions were made to determine how quickly the injury occurred, and whether there were any further complications. Such information may include reference to electronic medical records related to the device. may be useful for predicting potential outcomes later in life outside of the scope of video imaging. For example, the outcome may occur days, weeks, or months after the surgical procedure. In embodiments, additional information may be used to inform the analysis which videos to include in the analysis. For example, information collected from medical records can be used to identify similar patients' medical histories, disease histories, and type, type of diagnosis, medical history (including previous surgical procedures), and healthcare professional identification information Identify videos that share relevant information, the skill level of healthcare professionals, or any other relevant data. Videos that share these or other characteristics can be considered separate entities. This provides a more accurate idea of ​​what outcomes can be expected for a given decision. This can be done.

[0096] Similar decision junctions are those that are similar to the current decision junction. In some embodiments, each Similar decision junctions are surgical procedure decision junctions that follow a similarity metric. The metric is how closely related the decision junctions are. Such a metric may be any value, classification, or other indicator of the degree of association. Based on computer vision analysis to determine how well a procedure or technique is suited The metric may also be determined based on the number of decisions that the decision junctions have in common. The determination may be based on the number of characteristics and the degree to which the characteristics match. For example, Two decision-making junctions with patients with different behaviors and physical characteristics involve two more distinctive A higher similarity may be assigned based on the similarity metric than a typical patient. Other characteristics and / or considerations may also be used. The metric can be based on any similarity measure, such as those mentioned above. For example, ,The similarity metric may be the same as the similarity ,index, or may be a function of the similarity index. Well, and so on.

[0097] In addition to or instead of decision junction markers, various other marker types In some embodiments, the markers may be used to indicate the occurrence of intraoperative surgical events. The video may include intraoperative surgical event markers that can be associated with positions in the associated video. Examples of various intraoperative surgical events that can be identified by markers include the video indexing mentioned above. In some embodiments, the present invention provides a method for fabricating a semiconductor device using a microprocessor. The intraoperative surgical event marker is a general-purpose marker that indicates that an intraoperative surgical event has occurred at that location. In other embodiments, the intraoperative surgical event marker may be based on the type of event, whether the event is an adverse event, characteristics of intraoperative surgical events, including whether they were Exemplary markers are shown in FIG. 4. As an illustrative example, the following is shown for marker 434: The icons shown may be used to represent general intraoperative surgical event markers, while marker 4 36 provides more specific intraoperative surgical event markers, such as identifying that an incision occurred at that location. The markers shown in FIG. 4 are provided as examples and various other forms of Markers can be used.

[0098] These intraoperative surgical event markers can be automatically identified as described above. The computer analysis methods described above can be used to analyze medical devices, anatomical structures, surgeon characteristics, patient characteristics, and Characteristics of objects, event characteristics, or other features can be identified in the video footage. The interaction between the medical device and the anatomical structure is determined by the time an incision, suturing, or other intraoperative event is performed. In some embodiments, the intraoperative surgical event marker can indicate: Based on information provided in a data structure such as data structure 600 described above with reference to FIG. It can be identified.

[0099] Consistent with the disclosed embodiments, the selection of intraoperative surgical event markers may be varied to allow the surgeon to select different surgical events. Additional video clips from the procedure may be available for viewing. The different video clips show different ways in which selected intraoperative surgical events were handled. For example, in current videos, surgeons follow one technique to perform an incision or other action. By selecting intraoperative surgical event markers, the surgeon can It is possible to see alternative techniques that may be used to perform other operations. The intraoperative surgical event may be an adverse event such as bleeding, and another video clip may be used to show the surgeon's Other ways in which an adverse event was addressed may also be depicted. In some embodiments, the markers may be used to If related to events, the selection of intraoperative surgical event markers allows the surgeon to distinguish between different surgical procedures. For example, different surgical procedures can be performed using a They may be of different types (e.g., laparoscopic vs. thoracoscopic), but with the same or similar intraoperative procedures. Also, surgical procedures vary depending on the medical condition, patient characteristics, and the type of surgery. , different medical professionals, or other distinctions. By selecting can be done.

[0100] The different video clips may be displayed in a variety of ways, as well as in other embodiments described herein. For example, selecting an intraoperative surgical event marker may display a menu from which The surgeon can select a different video clip. The menu will change to show the selected intraoperative procedure. A description of the different ways the event was handled, a thumbnail of the video clip, and a link to view the video clip. Corrections and / or other information associated with the video clips, e.g. when they were recorded Date, type of surgical procedure, name or identification of the surgeon performing the surgical procedure, or optional Other relevant information may be included.

[0101] According to some embodiments of the present disclosure, the at least one video is arranged in procedural chronological order. It may include a compilation of footage from multiple surgical procedures. can refer to the sequence of events that occur relative to a surgical procedure. Arranging the piles in procedural chronological order allows different events from different patients to be recorded in a single procedure. This may include placing the results in the order that they would have occurred if performed on a patient In other words, the compilation was done from various surgeries on different patients, but the reproduction of the compilation In some embodiments, the images are displayed in the order in which they appear in the surgical procedure. The compilation can depict complications from multiple surgical procedures. In this embodiment, one or more markers may be associated with multiple surgical procedures and may be associated with a common time. may be displayed on a line, so the viewer interacts with a single timeline. However, the video footage presented along the timeline may be from different procedures and / or different patients. Exemplary complications that may be displayed are those described above with respect to video indexing. It is stated.

[0102] FIG. 5 illustrates an exemplary process for reviewing surgical videos consistent with disclosed embodiments. 5 is a flowchart illustrating a process 500. The process 500 may be implemented by one or more microprocessors. In some embodiments, the method may be performed by at least one processor, such as a processor. , process 500 is not necessarily limited to the steps shown and may include any of the steps described herein. Any of the various embodiments may be included in the process 500. In step 510 The process 500 may include accessing at least one video of a surgical procedure, for example, as described above. The at least one video may include a single surgical procedure, as described above. It may include video footage from a surgical procedure or may be a compilation of footage from multiple surgeries. The process 500 may include, in step 520, displaying at least one video. As described above, the method may include outputting at least one video Outputting for display means displaying at least one video on a screen or other display device. transmitting a signal to display at least one video signal to another computing device; storing it in an accessible location on your device and transmitting at least one video; or any other process or step that allows video to be displayed. can be done.

[0103] In step 530, the process 500 generates an output to display the surgical timeline. and overlaying the surgical timeline on at least one video recorded in the surgical procedure. The in- ... In some embodiments, the surgical timeline is displayed along with the video. Markers may be represented as shapes, icons, or other symbols along the timeline. The embodiment may be represented as a graph, a chart, or other graphical representation. In other embodiments, the timeline may be a chronological sequence of steps, events, and / or It can also be a text-based list of decision junctions. It may be text-based or contained in a list.

[0104] Step 540 involves the surgeon reviewing the surgical timeline while viewing a playback of at least one video. Allows the user to select one or more markers on the video, thereby changing the video display This may include skipping to a location associated with the marker. In this state, the surgeon may select a marker associated with the surgical site, which may include information from previous surgical procedures. Additional information about the event or occurrence can be viewed. For example, markers can be used to identify intraoperative surgical events. By selecting a marker, the surgeon can associate the marker with an intraoperative surgical event. For example, a surgeon can view additional video clips of previous surgical procedures. , similar intraoperative surgical events were handled differently, different techniques were used, or outcomes were different. You can see clips from other surgeries that have changed. In some embodiments, the markers , may be decision junction markers representing decisions made during a surgical procedure. By selecting a decision junction marker, the surgeon can decide which decision involves another decision. Such information can be used to find similar decision junctions, A list or distribution of possible decisions, the estimated outcomes of different possible decisions, or any other relevant The process described in process 500 may include a video of a previous surgical procedure containing the information. Based on the steps, the surgeon or other user can use the timeline interface to Surgical videos can be reviewed more effectively and efficiently.

[0105] In preparation for a surgical procedure, the surgeon reviews videos of similar surgical procedures that have been performed. However, it is often beneficial for the surgeon to be involved in preparing for the surgical procedure. Identifying successive images or portions of images can be tedious and time consuming. Therefore, surgical video footage can be efficiently and effectively indexed based on the content of the footage. It is not a conventional method that allows a surgeon or other medical professional to easily access and review the data. There is a need for new methods.

[0106] Aspects of the present disclosure include methods, systems, devices, and computer-readable media. For example, surgical events within a surgical phase can be related to surgical film indexing. The viewer can skip directly to the event and activate the specified characteristics. In some embodiments, the user may view only events that involve a surgical phase. (e.g., incision) with characteristics (e.g., specific complications) You can specify an event (inadvertent damage) and create a video clip of one or more events that share that characteristic. A pop-up can be presented to the user.

[0107] For ease of description, aspects of the method may be described as systems, devices, and computer-readable media. The methods are described below with the understanding that they apply equally to the body. For example, such methods Some aspects of the present invention may be performed electronically over wired, wireless, or both networks. Other aspects of such methods may be performed using non-electronic means. In this regard, the method is not limited to any particular physical and / or electronic means, but rather may be implemented using many This can be achieved using different means.

[0108] Consistent with the disclosed embodiments, a method may be used to access indexed video footage. The video footage to be indexed may include scanning for specific surgical As used herein, video includes recorded images and / or audio. It can include any form of recorded visual media, including audio. For example, video can be recorded as shown in Figure 1. As discussed above in connection with the images from cameras 115, 121, 123 and / or 125, etc. The image may include a sequence of one or more images captured by a capture device. The video files may be stored as individual files and may include corresponding audio data. In some embodiments, the video may be stored in a combined format such as a video file. The image may be stored as raw data and / or images output from the image capture device. In other embodiments, video can be processed. For example, a video file can be Audio Video Interleave (AVI), Flash Video Format ( FLV), QuickTime File Format (MOV), MPEG (MPG, M P4, M4P, etc.), Windows Media Video (WMV), material exchange file Format (MXF), uncompressed format, lossy compression format, lossless compression format .mat, or any other suitable video file format.

[0109] Video footage can refer to the length of video captured by an image capture device In some embodiments, the video footage is displayed as a sequence of images in the order in which they were originally captured. For example, video footage can refer to the length of a video, including the duration of a video compilation. In other embodiments, the video may include non-compiled video to form The video footage is first captured sequentially to remove frames related to immobility. The frames may be compiled in one or more ways, such as to compile frames that are not included in the original. Accessing video footage means reading the video footage from a storage location such as a memory device. The video footage may be retrieved from local memory, such as a local hard drive. It may be accessed, for example, from a remote source via a network connection. Consistent with the present disclosure, indexing may be used to more efficiently and / or effectively search. It can refer to the process of storing data so that it can be retrieved. Indexing is done so that video footage can be identified based on characteristics or indicators. This may include associating one or more characteristics or metrics with the video footage.

[0110] A surgical procedure is any procedure that involves or includes manual or operative manipulation of a patient's body. Surgical procedures may include medical procedures such as cutting, abrading, suturing, or physical alteration of body tissues and organs. Some examples of such surgical procedures include laparoscopy, Surgery, thoracoscope surgery, bronchoscope surgery, microsurgery, open surgery, robotic surgery, appendectomy, cervical Endarterectomy, carpal tunnel release, cataract surgery, cesarean section, cholecystectomy, colon resection (e.g., partial partial colectomy, total colectomy, etc.), coronary angioplasty, coronary artery bypass, debridement (e.g. wounds, burns, infections, etc.), free skin grafts, hemorrhoidectomies, hip replacements, Hysterectomy, hysteroscopy, inguinal hernia repair, knee arthroscopy, knee replacement, mastectomy ( (e.g., partial mastectomy, total mastectomy, modified radical mastectomy, etc.), prostatectomy, Prostatectomy, shoulder arthroscopy, spinal surgery (e.g., spinal fusion, laminectomy, foraminotomy) surgery, discectomy, disc replacement, interlaminar implants, etc.), tonsillectomy, cochlear implant surgery , brain tumor (e.g., meningioma) resection, interventional surgery such as percutaneous transluminal coronary angioplasty, transcatheter aortic valve replacement, minimally invasive surgery to drain intracerebral hemorrhage, or any form of resection. This disclosure is not intended to be limiting unless it is specifically stated in the context of a surgical procedure. However, it should be understood that the invention may also be applied to other forms of medical treatment, or treatment in general. be.

[0111] In some exemplary embodiments, the accessed video footage includes a view of the patient above the operating table. Located at least one of a location within a surgical body cavity, within an organ of a patient, or within the vasculature of a patient. The video image may include video footage captured via at least one image sensor. The image sensor may be any sensor capable of recording images. The image sensor is positioned external to the patient and configured to capture images from above the patient. The image sensor may include any image sensor, such as a camera, as shown in FIG. In other embodiments, the image sensor may include, for example, The device may be placed inside the patient's body, for example, in a body cavity. A surgical cavity may include any relatively empty space within the body. The space inside the patient's body where a placement or operation is being performed, or where surgical instruments are present and / or used. The surgical cavity may not be completely empty and may contain tissue, It is understood that an organ may include any self-contained part of an organism, such as blood or other fluids. Some examples of organs in a human patient include the heart or liver. The vasculature can refer to a system or group of blood vessels within an organism. and / or an image sensor located in the vasculature is included in a surgical instrument inserted into the patient. It may include a camera.

[0112] Aspects of the present disclosure analyze video footage to identify video related to operative steps of a particular surgical procedure. As used herein with respect to video footage, the term "video image" may include identifying the location of the video footage. When used, the location can refer to any specific location or area within the video footage. In some embodiments, the location may include a particular frame or range of frames of the video. Therefore, the video image position can be one or more frame numbers or can be represented as other identifiers. In other embodiments, the location is associated with the video footage. For example, a location in a video image can be referenced by a time index. timestamps, time ranges, specific start and / or end times, or video footage In other embodiments, the location may refer to at least one of It can point to at least one specific position within one frame. The video location can be one or more pixels, voxels, bounding boxes, or bounding It may be represented as a bounding polygon, a bounding shape, coordinates, etc.

[0113] For purposes of this disclosure, a stage refers to a particular period or stage in a process or series of events. Thus, a surgical phase can refer to a specific period of time or duration of a surgical procedure, as described above. For example, the surgical steps in laparoscopic cholecystectomy include trocar placement and Placement, preparation, incision of Calot's triangle, clipping and cutting of the cystic duct and artery, incision of the gallbladder, and gallbladder capsule Other examples include cataract surgery, liver bed cleansing and coagulation, and gallbladder contraction. The surgical steps of the procedure include preparation, povidone-iodine injection, corneal incision, capsulotomy, and cataract surgery. List of procedures including phacoemulsification, cortical aspiration, intraocular lens implantation, intraocular lens adjustment, and wound sealing. In yet another example, the surgical steps in pituitary surgery may include preparation, nasal incision, and nasal retractor placement. These include placement, tumor access, tumor removal, nasal replacement, suturing, and placement of nasal compression. Some other examples of surgical steps include preparation, incision, laparoscopic positioning, suturing, etc. Some examples include:

[0114] In some embodiments, identifying the location of the video footage is based on user input. User input can include any information provided by a user. When used in connection with video indexing, the user input may include identifying a location in the video footage. For example, the user can specify a specific frame number, timestamp, Identifying a time range, start and / or stop time, or location in video footage Alternatively, the user input may be used in any other way. The input may include the input or selection of a stage, event, action, or device, and the input may be associated with the video image (e.g., via a lookup table or other data structure) User input can be provided from desktop computers, laptops, tablets, etc. mobile phones, wearable devices, Internet of Things (IoT) devices, or The user interface of the user device, such as any other means for receiving input from a user. The interface can be, for example, the first stage name. One or more drop-down menus with one or more selection lists, where the user enters the stage name and / or data entry fields that suggest the stage name once a few characters are entered. field, a selection list from which you can select stage names, each relating to a different stage. A group of associated selectable icons or steps that the user can identify or select Any other mechanism for doing so may be used. For example, the user may As will be described in more detail below, a user interface similar to user interface 700 may be used. In another example, the user input can be a voice command. The user input may be received via hand and / or voice input, and the user input may be processed by a voice recognition algorithm. In yet another example, user input may be processed using gestures (hand gestures). The user input may be received via a gesture recognition algorithm. It may be processed using

[0115] In some embodiments, identifying the location of the video footage includes using computer analysis. The computer analysis may include analyzing frames of the video footage using a computer. This may include any form of electronic analysis using a computing device. In this embodiment, computer analysis is performed to identify features in one or more frames of video footage. The computer analysis may involve using one or more image recognition algorithms. It may be performed on multiple frames, for example to detect motion or other changes between frames. In some embodiments, the computation may be performed over multiple frames to achieve the desired result. Computer analysis includes Viola-Jones object detection, scale-invariant feature transformation (S IFT), Histogram of Oriented Gradients (HOG) features, Convolutional Neural Networks (C NN), or any other form of object detection algorithm. Other exemplary algorithms include video tracking algorithms, motion detection algorithms, detection algorithms, feature detection algorithms, color-based detection algorithms, texture Based detection algorithm, shape-based detection algorithm, boosting-based detection detection algorithm, face detection algorithm, or any other algorithm for analyzing video frames. Suitable algorithms can include, for example, identifying specific locations in a video. You can train a machine learning model using training examples to It uses a trained machine learning model to analyze video footage and identify the location of the video footage. An example of such a training example is a label that indicates a position within a video clip. with a bell or a label indicating that the corresponding position is not included in the video clip. Both can include video clips.

[0116] In some embodiments, the computer image analysis includes a visual analysis of the previously identified surgical steps. Use a neural network model trained using an example video frame identifying at least one of the video image position or stage tag by In other words, one or more factors known to be associated with a particular surgical step can be Using frames from the video, we can use machine learning algorithms, for example, using backpropagation Train a neural network model, for example, using gradient descent optimization Therefore, we can use the trained neural network model is used to identify whether one or more video frames are also associated with a surgical stage. Some non-limiting examples of such artificial neural networks are shallow neural networks. Light artificial neural networks, deep artificial neural networks, feedback artificial neural networks Neural networks, feedforward artificial neural networks, autoencoders Artificial neural networks, probabilistic artificial neural networks, time-delay artificial neural networks neural networks, convolutional artificial neural networks, recurrent artificial neural networks These may include neural networks, long short-term memory artificial neural networks, etc. In an embodiment, the disclosed method comprises: The method may further include updating the trained neural network model. do.

[0117] In some aspects of the present disclosure, video footage is analyzed to identify at least one surgical event or stage. Identifying video image locations associated with at least one surgical step is useful for replay. A video is used to identify at least one of the starting location or the beginning of the surgical event for playback. This may include performing computer image analysis on the video. Using the computer analysis techniques described above, the disclosed method can determine when a surgical step or event begins. For example, the method may include identifying locations within the video footage where a surgical event such as an incision is occurring. The onset of can be detected using the object and / or motion detection algorithms described above. In other embodiments, the start of the incision can be detected based on machine learning techniques. For example, a machine learning model may combine video footage and known events of an incision or other surgical event and / or procedure. and the corresponding labels that indicate the starting point of the training The generated model can then be used to identify the onset locations of similar procedures and / or events in other surgical video footage. can be identified.

[0118] Some aspects of the present disclosure may include generating a stage tag associated with a surgical stage. As used herein, a "tag" refers to information associated with or linked to a data set. In some embodiments, a tag can refer to any process or marker that is being used. It can also be a property of a data file, such as a video file. This may include writing or overwriting properties within the video file. In some embodiments, generating tags may involve, for example, storing a video file as a separate device. You can add information to files other than the video file itself by associating it with tags in the database. Tags can include writing text information, numeric identifiers, or tagged The stage tag may be expressed as any other suitable means for processing. In one embodiment, the stage tag may be a tag that identifies the stage of the surgical procedure. a marker indicating the position in the video where the surgical step begins, a marker indicating the position in the video where the surgical step ends, Markers indicating positions in the video during the surgical steps, or the range of the video encompassing the surgical steps, The tag may be a pointer to the video data itself, or A stage tag may be placed in a data structure to allow for retrieval of the stage location. The display may include computer readable information to trigger a display and identify the stage to the user. It may also contain human-readable information for the purpose of identifying the stage of surgery, for example, a stage tag associated with the stage of surgery. generating a tagged data set associated with that stage of the surgical procedure. generating a tag containing text such as "laparoscopic positioning" to indicate In another example, generating a stage tag associated with a surgical stage can include In some embodiments, generating a tag that includes a binary encoding of the stage identifier. The generation of stage tags is based on computer analysis of video footage depicting surgical stages. For example, the disclosed method may use the above-mentioned methods to determine the stage tag. This can include analyzing the video of the surgical stage using body and motion detection analysis methods. For example, a particular type of medical device or other procedure used in a specific way or in a specific sequence. If you know that a stage begins or ends with a step, you can use the via image recognition performed on video footage to identify the unique use of the means for Image recognition can be performed to identify specific stages, and stage tags can be generated. is a training method to associate one or more video frames with one or more stage tags. A trained machine learning model or neural network model that can (e.g., deep neural networks, convolutional neural networks, etc.) For example, training examples can be used to train machine learning algorithms. and associating other video footage data with one or more stage tags. An example of such a training example would be a video clip and and a label indicating the desired tag or lack of the desired tag corresponding to the video footage. Such labels may include one or more location indications within the video footage that correspond to surgical steps, It may include an indication of the type of surgical step, an indication of the characteristics of the surgical step, and the like.

[0119] A method according to the present disclosure may include associating a stage tag with a video footage position. Any suitable means can be used to associate stage tags with video footage positions. Such tags may include one or more location indications within the video footage that correspond to surgical steps, This may include an indication of the type of step, an indication of the characteristics of the surgical step, etc. In the example, the location of the video footage may be included in a tag. a time range, frame number, or other means for associating a stage tag with a position in the video footage In other embodiments, the tag is associated with the location of the video footage in the database. For example, the database may associate stage tags with video footage and specific video footage locations. The database may contain information linking to the data, as described in more detail below. It can contain a data structure.

[0120] Embodiments of the present disclosure provide a visual system for identifying the event location of a particular intraoperative surgical event within a surgical phase. The intraoperative surgical event may further include analyzing the video footage. In some embodiments, an intraoperative surgical event may be any event or action that occurs during surgery. This includes surgeons, surgical technicians, nurses, physician assistants, anesthesiologists, physicians, or any other healthcare professionals. List actions performed as part of a surgical procedure, such as those performed by a surgical professional. Intraoperative surgical events include incisions, administration of drugs, use of surgical instruments, resections, incisions, removal, ligation, grafting, stitching, suturing, or any related surgical procedure or step It may be a planned event such as any other planned event. The event may include an adverse event or a complication. Some examples of intraoperative adverse events include bleeding, intestinal bleeding, mesenteric emphysema, injury, conversion to unplanned open surgery (e.g., laparotomy), planned Some examples of intraoperative complications include: Hypertension, hypotension, bradycardia, hypoxemia, adhesions, hernias, atypical anatomical structures, dural tears, Examples of intraoperative events include peritoneal injury, arterial occlusion, etc. Intraoperative events include technical errors, communication Errors, administrative errors, judgment errors, decision-making errors, errors related to the use of medical devices, mistakes Other errors may include communication errors.

[0121] An event location may be a location or area within the video footage associated with an intraoperative surgical event. Similar to the step positions described above, event positions are specific frames (e.g., frames) of video footage. frame number or range of frame numbers) or time information (e.g., timestamp) , time range, or start and end times), or any other method for identifying a location within the video footage. In some embodiments, the event location can be identified by any other means. Analyzing video footage to obtain the desired results requires a computer to analyze the frames of the video footage. The computer analysis may include using the techniques or algorithms described above. Similar to stage identification, event identification is a method for identifying events uniquely. It can be based on the detection of actions and means used in a way that identifies the user. For example, image recognition The recognition identifies when a particular organ is dissected and allows for marking of that dissection event. In another example, image recognition can be used to note cuts in blood vessels or nerves and identify their effects. Image recognition can be used to detect bleeding or other fluids. Events can also be marked by the detection of a loss. In some embodiments, the event location can be calculated using Analyzing the video footage to identify exemplary surgical events, including previously identified surgical events, Neural network models trained using video frames (e.g., Deep neural networks, convolutional neural networks, etc.) are used to detect event locations. In one example, identifying an intraoperative surgical event in a portion of the video. Training a machine learning model using training examples to identify locations It can then use trained machine learning models to analyze the video footage (or surgical steps) Analyzing the video footage (corresponding to the video footage) to identify the event location of a specific intraoperative surgical event within a surgical phase An example of such a training example is the detection of a particular event in a video clip. The location of the event may include a video clip with a label indicating the absence of such an event. can.

[0122] Some aspects of the present disclosure relate to associating an event tag with an event location for a particular intraoperative surgical event. As mentioned above, tags are used to associate information with data or portions of data. Event tags can be used to store data or portions of data intraoperatively. Similar to stage tags, event tags can be associated with events such as surgical events. Associating with a video file writes data to the video file, e.g., to the video file's properties. In other embodiments, associating an event tag with an event location may include The event tags are stored in a file or database that associates the event tags with the video footage and / or event locations. Alternatively, associating an event tag with an event location may involve writing data The data structure may include recording the marker in a data structure, the data structure including the tag in the video footage. In some embodiments, the same file or data The database can be used to associate stage tags with video footage as event tags. In other embodiments, a separate file or database may be used.

[0123] Consistent with the present disclosure, the disclosed method stores event characteristics associated with a particular intraoperative surgical event. The event characteristic may be any property or characteristic of the event. For example, event characteristics may be patient or surgeon characteristics, characteristics or properties of surgical events or surgical stages. Examples of characteristics include excess fatty tissue, enlarged List the affected organs, tissue disruption, fractures, disc displacement, or any other physical characteristics associated with the event. Some characteristics may be discernible by computer vision. In addition, other characteristics may be discernible through human input. In the latter example, the patient's age The age or age range can be stored as an event characteristic. Similarly, the state of the patient's previous medical history can be stored as an event characteristic. The behavior can be stored as an event characteristic (e.g., diabetic). In this paper, we use the stored event characteristics to distinguish intraoperative surgical events from other similar events. For example, healthcare professionals can search video footage to identify people over 70 who have arrhythmias. One or more coronary artery bypass surgeries performed on men can be identified. Various other examples of stored event characteristics that may be obtained are provided below.

[0124] The stored event characteristics may be determined in a variety of ways. The method may include determining the stored event characteristics based on user input. For example, the user may be presented with a user interface similar to that described above in connection with the selection of a step or event. In another example, a user can input the event characteristics to be stored via the Event characteristics to be stored may be entered via voice command. Various examples are provided below. Another aspect of the disclosed method is the ability to delineate specific intraoperative surgical events. determining stored event characteristics based on computer analysis of the video footage; For example, the disclosed method can be used to recognize event characteristics based on video footage. This may involve using various image and / or video analysis techniques as described above. As an illustrative example, the video footage may include a display of one or more anatomical structures of the patient. The event features that identify anatomical structures can be based on the detection of anatomical structures in video footage. or based on detection of an interaction between a medical device and an anatomical structure. In another example, a machine can be used to determine event characteristics from a video using training examples. You can train a machine learning model and use the trained machine learning model. The video footage can be analyzed to determine stored event characteristics. An example of a training example is a video that depicts intraoperative surgical events along with labels that indicate the characteristics of the events. It may contain an octopus clip.

[0125] Some aspects of the present disclosure provide for the transfer of at least a portion of video footage of a particular surgical procedure to other external The step tags, event tags, and event characteristics in the data structure include further video footage of the surgical procedure. The data structure may also include associating one or more other surgical procedures. Also includes associated respective stage tags, respective event tags, and respective event characteristics A data structure consistent with this disclosure may include any collection of data values ​​and relationships between them. Data can be analyzed linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, Objects can be organized, multidimensionally, operationally, in an ordered or unordered manner. In a project-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner , searchable repository, sorted repository, indexed repository The data may be stored in a directory or in any manner that allows for data access. Examples of data structures include arrays, associative arrays, linked lists, binary trees, balanced trees, and chunks. ,Stack, Queue, Set, Hash Table, Record, Tagged Union, ER Model, For example, the data structure may include, for example, MongoDB, Red Hat, is, Couchbase, Datastax Enterprise Graph, E lastic Search, Splunk, Solr, Cassandra, Amaz Data on DynamoDB, Scylla, HBase, and Neo4J XML databases, RDBMS databases, SQL data for storage / retrieval The data structures of the disclosed system may include SQL-based or NoSQL alternatives. components or remote computing components (e.g., cloud-based) The data in the data structure may be stored in contiguous or non-contiguous memory. Furthermore, a data structure as used herein refers to a structure where information is co-located. The data structure does not need to be organized by the same or different entities, for example. It may be distributed across multiple servers that may be owned or operated by the organization. The data structure is used to associate video footage with stage tags, event tags, and / or event characteristics. It may include any data format that may be used.

[0126] FIG. 6 illustrates an exemplary data structure 600 consistent with disclosed embodiments. As shown in FIG. The data structure 600 includes video footage 610 and video footage 612 related to different surgical procedures. For example, the video footage 610 may include a table including a video of a laparoscopic cholecystectomy. The video footage 620 may include footage of the cataract surgery. The video footage 620 is shown at video location 620, which may correspond to a particular surgical stage of the cataract surgery. 21. The stage tag 622 can be associated with the image position 621 as described above. The associated stage (in this example, the corneal incision) can be identified. 620 can be associated with an event tag 624, which indicates the event occurring at the event location 623. It is possible to identify intraoperative surgical events (incisions in this example) within the surgical phase being performed. The video 620 may include information about intraoperative surgical events, such as the skill level of the surgeon, as described in detail above. The event may be further associated with an event characteristic 625 that may describe one or more characteristics of the event. Each video image identified in the data structure contains two or more image locations, a stage tag, an event location, For example, the video footage 610 may be associated with multiple event tags and / or event characteristics. Regarding stage tags corresponding to surgical stages (e.g., "Calot's triangle incision" and "cystic duct section") Furthermore, each surgical step in a particular video stream may be associated with multiple events. may be associated with multiple event locations, event tags, and / or event characteristics. However, in some embodiments, a particular video feed may be recorded at a single surgical stage. It is understood that the event may be associated with a floor and / or an event. In some embodiments, the event may be , including no event characteristics, a single event characteristic, two event characteristics, three or more event characteristics, etc. It will also be appreciated that an event may be associated with any number of event characteristics. Non-exhaustive examples include skill levels relevant to the event (e.g., minimum required skill level). the level of skill demonstrated, the level of skill of the healthcare providers involved in the event, the time associated with the event (e.g., start time, end time, etc.), the type of event, and the medical device associated with the event. information related to the organs, information related to the anatomical structures related to the event, and medical outcomes related to the event information related to the outcome, one or more quantities (e.g., amount of leak, amount of medication, amount of fluid, etc.), one These dimensions (e.g., dimensions of anatomical structures, dimensions of incisions, etc.) can be included. Furthermore, data structure 600 is provided as an example, and various other data structures may be used. It should be understood that this is a good thing.

[0127] An embodiment of the present disclosure provides a method for selecting a stage tag, a selected event tag, and a visual for display. The user can access the data structure by selecting the selected event characteristics of the video image. The user may provide access to the data stored in the data structure. In some embodiments, the user may be any person or entity that can provide The person in charge may be a surgeon or other health care professional. For example, a surgeon may review or train data structures and / or video footage associated with the data structures for training purposes In some embodiments, the user may access the hospital administrator, administrator, This may be a custodian, such as the surgeon, or other individual who may need access to the video footage. In some embodiments, the user may be a patient, who may view video footage of their surgery. Similarly, users may be provided with access to the patient's relatives, guardians, primary caregivers, etc. The user may be, but is not limited to, a healthcare provider, an insurance agent, or another representative. insurance companies, regulatory authorities, police or law enforcement agencies, medical organizations, or anyone providing access to video footage. may include any other entity that may be provided User selections may be made to identify specific stage tags, event tags, and / or event characteristics. In some embodiments, the user selection may include any of the following: through a graphical user interface, such as on the display of a computing device In another example, the user selection may be made via a touch screen. In a further example, the user selection may be made via voice input, The force may be processed using a voice recognition algorithm. The selection may be made via a gesture (such as a hand gesture), which may include: It may be analyzed using gesture recognition algorithms. The user can select a selected stage tag, a selected event tag, or all three of the selected event characteristics. Instead of selecting a subset of these, the user can select The user only needs to select the event characteristics, and the data structure and associated may be granted access to the information attached.

[0128] FIG. 7 illustrates a video indexed for display consistent with disclosed embodiments. FIG. 7 is a diagram of an exemplary user interface 700 for selecting a video. The interface 700 includes one or more search boxes 710, 720 and a and 730. The search box 710 can be used to select one or more surgical stages to be displayed. In some embodiments, the user interface 7 may allow the user to select a floor. 00 provides suggested surgical stages based on the stage tags contained in data structure 600 For example, when a user begins typing in search box 710, the user interface The face 700 suggests stage tag descriptions to search based on the characters entered by the user. In other embodiments, the user may select radio buttons, check boxes, drop-down lists, or A drop-down list, a touch interface, or any other suitable user interface mechanism may be used. Like step tags, users can select step tags by using the search box. 720 and 730 are used to segment the video footage based on the event tags and event characteristics, respectively. The user interface 700 also includes a drop-down list drop-down buttons 722 and 723 for accessing 32. As shown in FIG. 7, selecting the drop-down button 732 allows users to select event features based on their subcategories. For example, the user may click the drop-down list associated with the drop-down button 732. In the list, users can select "Surgeon Skill Level," which will allow them to You can search based on the skill level of surgeons in the 730. Although "Event Level" and various other event characteristic subcategories are provided as examples, the user It is understood that any feature or characteristics of the surgical procedure may be selected. The user must be a qualified surgeon, have years of experience, and / or have a high level of surgical skill, as described in more detail below. The user can refine the surgeon's skill level based on any indication of the surgical level. can search for data by clicking, tapping, or otherwise selecting the search button 740. The data structure can be accessed.

[0129] Displaying video footage involves presenting one or more frames of video footage, or portions thereof, to a user. In some embodiments, displaying may include any process that This may include electronically transmitting at least a portion of the video footage for viewing by, for example, Displaying the video image includes transmitting at least a portion of the video image over a network. In other embodiments, displaying the video footage may include By storing the video footage in a location accessible by the device you are using, Some embodiments may include making the video footage available to users. In this context, displaying a video image means displaying the video image on a computer or video screen or other device. This may include playing the video on a dual display device. and wherein the step of: It may further include presenting audio associated with the video footage.

[0130] Some aspects of the present disclosure provide a method for identifying a matching subset of stored video footage. Then, at least one selected stage tag, a selected event tag, and a selected event feature performing a lookup in a data structure of surgical video footage that matches the gender. Performing a lookup is an arbitrary process for reading data from a data structure. For example, at least one selected stage tag, event tag, and and data-structuring the corresponding video footage or portions of the video footage based on the selected event characteristics. A subset of stored video footage can be identified from its structure. It may include a video image or a plurality of identified video images related to a user selection. For example, a subset of stored video footage may be selected from the exact same stage tag. Floor tag, event tag that is exactly the same as the selected event tag, and event characteristic that is exactly the same as the selected event characteristic In another example, the event may include surgical video footage having at least one of the same event characteristics. A subset of the stored video footage is tagged with selected stage tags (e.g., selected categories). Similarity index) to the selected event tag (e.g., selected similarity index). Event tags that are similar (e.g., according to a selected event characteristic) and / or surgical videos that have at least one of the similar event characteristics (according to the similarity index determined) In some embodiments, as shown in FIG. 7, selection of search button 740 The selection can trigger a search to be performed.

[0131] In some exemplary embodiments, identifying a matching subset of stored video footage This involves using computer analysis to identify a matching subset of stored videos and select The process involves determining the degree of similarity between the event characteristics of the event. It can refer to an exact match, or it can refer to an approximate or closest match. In the example, the event characteristics are numerical (e.g., quantity, dimension, length, area, volume, etc., as discussed above). The degree of similarity can be determined by the numerical value included in the selected event characteristic and the stored It can be based on a comparison of the corresponding values ​​of the video. These include, but are not limited to, affinity functions, correlation functions, polynomial similarity functions, exponential similarity functions, and similarity functions, distance-based similarity functions, linear functions, nonlinear functions, etc. In one example, a graph matching algorithm or hyperparameter can be used to calculate the degree of Graph matching algorithms (e.g., exact match algorithms, inexact match algorithms) As another illustrative example, a "quasi-identifier" may be used to determine the degree of similarity. Video footage associated with the "preparation" stage tag is Stage tags containing "preparing", "pre-processing", "pretreatment", or other similar It is also possible to search for stage tags that refer to "preparation" but are not an exact match. Alternatively, a subset of stored video may be matched to a selected event characteristic. The degree of similarity can be expressed as a similarity ranking (e.g., 1 to 1 (on a scale of 0, 1 to 100, etc.) as a percentage of match, or how close the match is. The use of computer analysis may be used to represent the Similarity between the selected event characteristics and the event characteristics of one or more surgical procedures contained in the data structure This may include using a computer algorithm to determine the degree of sexuality. In one example, a k-nearest neighbor algorithm is used to find the most similar entries in a data structure. In one example, the entries in the data structure, as well as the user input event characteristics, can be identified. is a mathematical space (e.g., any dimensionality reduction or data embedding algorithm) ) between the entry embedding and the user input characteristics. The distance may be used to calculate the degree of similarity between two. In the example, the closest entry in the embedded mathematical space to the user-input property is may be selected as the entry that is most similar to the user-entered data.

[0132] Some aspects of the present invention involve displaying a matching subset of stored video footage to a user. thereby displaying the selected event characteristics while omitting playback of video footage lacking the selected event characteristics. The user can view surgical footage of at least one intraoperative surgical event that shares the event characteristics. The surgical video may include, as described in more detail above, , can refer to any video or video footage that captures a surgical procedure. In some embodiments, displaying a matching subset of stored video footage may be considered a video playback. For example, performing the methods described herein may include executing instructions to generate a The processing device may access a matching subset of the video footage and store the video footage. The video may be configured to be presented to the user on a screen or other display. The stored video footage may be played back using video playback, as described in more detail below with respect to FIG. It may be displayed in the video player user interface, such as in area 410. In an embodiment, as described above, a matching subset of stored video footage is displayed to the user. The causing to be displayed may include transmitting stored video footage for display. The matching subset of video footage is then transmitted over the network to a desktop computer. laptop computer, mobile phone, tablet, smart glasses, heads-up display, training equipment, or any other device capable of displaying video footage. The information may be transmitted to a computing device associated with the user, such as a .

[0133] Omitting playback means that the user is presented with a video that lacks the selected event characteristics. For example, omitting playback may be a process that This may include specifying not to show the video footage and not displaying the video footage. In embodiments where a subset is transmitted, omitting playback may be performed for events lacking selected event characteristics. This may include preventing the transmission of video footage related to the matching subset. By selectively transmitting only the parts of the video that are relevant to the matching subset, By selectively transmitting markers associated with portions of the video and / or matching This can be done by skipping parts of the video that are not relevant to the subset being processed. In an embodiment, video footage lacking the selected event characteristics may be transmitted, but the selected event characteristics may be transmitted. It may be associated with one or more instructions not to present video footage lacking the event characteristic.

[0134] According to various exemplary embodiments of the present disclosure, a selected operation lacking a selected event characteristic is At least one intraoperative procedure having selected event characteristics while omitting playback of portions of the event. Allowing the user to view surgical footage of the event may result in the loss of selected event characteristics. omitting playback of portions of a selected surgical event while replaying multiple events that share the selected event characteristics. The method may include sequentially presenting portions of the surgical footage of multiple intraoperative surgical events to the user. In other words, one or more portions of video footage may be stored in a data structure, e.g., using a lookup function. The selected event characteristic may be identified as being associated with the selected event characteristic via the selected event characteristic. allowing a user to view surgical footage of at least one intraoperative surgical event having The step of selecting the identified portion may include sequentially presenting one or more of the identified portions to the user. Unidentified portions of the video footage may not be presented. In the example, video footage is selected based on selected event tags and selected stage tags. Thus, in embodiments consistent with the present disclosure, a selected event lacking a selected event characteristic may be omitting playback of a portion of a surgical event that has been selected, while Allowing the user to view surgical footage of intraoperative surgical events allows for the user to view the selected events. Selected events that lack characteristics or are not associated with the selected event tag and selected stage tag. The selected event types are selected by sharing the selected event characteristics while omitting the playback of the selected surgical event parts. The user can then select the portion of the surgical footage of the multiple intraoperative surgical events associated with the selected stage tag. This may include presenting the information to the user sequentially.

[0135] As mentioned above, the stored event characteristics may include a wide variety of characteristics related to surgical procedures. In some exemplary embodiments, the stored event characteristics may include adverse outcomes of surgical events. For example, the stored event characteristics may include whether the event is an adverse event or not. or was associated with a comorbid condition, including the examples described in more detail above. Therefore, it is possible to display a matched subset and select adverse outcomes. Allows users to view surgical footage of selected adverse outcomes while omitting playback of missing surgical events. For example, this may include enabling a surgeon to visualize blood vessels during laparoscopic surgery. The entire operation can be displayed to the user, if they wish to see if the injury has been addressed. Instead, the user can select a vascular injury event, and the system then Only a portion of the video footage that occurred can be displayed. The stored event characteristics can be and / or outcomes, including predicted outcomes, may be identified as well. Examples of such outcomes include: The criteria for this are: full recovery by the patient, whether a leak occurred, the amount of leak that occurred, and the extent of the leak. whether the patient was readmitted after discharge; and whether the patient was hospitalized after surgery. The duration of the procedure or any other outcome that may be associated with the surgical procedure may also be mentioned. This allows users to see the long-term impact of a particular technology when viewing. In some embodiments, the stored event characteristics may include these or other outcomes. Also, displaying the matching subset allows for the reproduction of surgical events lacking the selected outcome. Allowing users to view surgical footage of selected outcomes while omitting may include:

[0136] In some embodiments, the stored event characteristics may include a surgical procedure. Thus, the stored event characteristics can identify whether a particular technique is performed. For example, there may be multiple techniques that can be applied at a particular stage of surgery, and the event The characteristics may identify which technique is being applied. In this way, learning of a particular technique Users interested in the technique can select the video result so that only procedures using the specified technique are displayed. You may be able to filter the results to show a subset of matches. The system plays back selected surgical videos while omitting playback of surgical videos that are not associated with the selected surgical procedure. This may include allowing the user to view surgical footage of the surgical procedure. For example, a user may select non-contiguous portions of video captured from either the same or different surgeries. In some embodiments, the stored event characteristics are associated with a particular surgical procedure. For example, the event characteristics may include the identity of the surgeon performing the surgical procedure. The surgeon may include his / her name, identification number (e.g., employee number, medical registration number, etc.), or any other form of identification. In some embodiments, the surgeon can recognize the surgeon's representation in the captured video. For example, as discussed above, various facial and / or audio Voice recognition technology can be used, thus allowing users to train the skills of a particular surgeon. If the user wants to investigate, he can do so. For example, to display the matching subset Showing the selected surgeon's actions while omitting playback of footage lacking actions by the selected surgeon. The method may include allowing the user to view footage showing actions by the surgeon. So, for example, if multiple surgeons are participating in the same surgical procedure, the user can You can choose to only see the behavior of a subset of the team.

[0137] In some embodiments, the event characteristics may be used to identify other healthcare providers or healers who may be involved in the procedure. In some instances, characteristics related to a healthcare provider may also be associated with a healthcare professional. The characteristics of the healthcare provider involved in the surgical procedure can include any characteristic of the healthcare provider. Some non-limiting examples of suitable healthcare providers include surgeons, anesthesiologists, nurses, and licensed practical nurses. The surgical team includes certified nurse anesthetists (CRNAs), surgical technologists, residents, medical students, and physician assistants. Further non-limiting examples of such characteristics include: These include certification, experience level (e.g., years of experience, previous experience in similar surgical procedures), , past success rates of similar surgical procedures), and demographic characteristics (e.g., age). Some examples include:

[0138] In other embodiments, the stored event characteristics may be associated with a particular surgical procedure, surgical stage, or part thereof. For example, the stored event characteristics may include the duration of the event. Displaying matching subsets can include events of different durations. Allows the user to view a video showing an event of a selected duration while omitting video playback. This may include, for example, enabling the system to complete the task faster than the standard. Users who may want to see specific actions completed within that threshold can view the specified actions. In another example, a user who wants to see more complex events can set a time threshold for the sets a time threshold to exclude treatments involving events lasting longer than the selected threshold, or In another embodiment, the treatment with the longest lasting event in the group of events can be viewed. The stored event characteristics may be the start time of the event, the end time of the event, or any other time indicator. Displaying the matching subsets may include the following: The stage associated with an event within a particular surgical procedure is displayed, while omitting the playback of the video of the event. Footage showing events from selected times within a particular surgical procedure, or within selected portions of a particular surgical procedure This may include making it visible to the user.

[0139] In another example, the stored event characteristics may include patient characteristics. The term refers to any physical, sociological, economic, demographic, or behavioral characteristic of a patient, and Refers to characteristics of a patient's medical history. Some non-limiting examples of such patient characteristics include age, , gender, weight, height, body mass index (BMI), menopausal status, typical blood pressure, patient Nom characteristics, educational status, education level, socio-economic status, income level, occupation, type of insurance, health status, self-rated health, functional status, functional impairment, duration of illness, severity of illness; Number of lesions, characteristics of the lesions (e.g., type of lesion, tumor size, histological grade, invasive risk) number of lymph nodes), health care utilization, number of health care visits, interval between health care visits, regular source of health care , family situation, marital status, number of children, family support, ethnicity, race, religion, religious tie type, native language, characteristics of previous medical tests performed on the patient (e.g., type of test, characteristics of previous medical treatments given to the patient (e.g., type of treatment, time of treatment, test results, etc.); Some of these medical tests include: Non-limiting examples of such tests include blood tests, urine tests, stool tests, medical imaging tests (e.g., ultrasound) Examination, angiography, magnetic resonance imaging (MRI), computed tomography (CT), X-ray X-ray, electromyography, positron emission tomography (PET), physical examination, electrocardiogram, Amniocentesis, Pap test, skin allergy test, endoscopy, biopsy, pathology, blood pressure measurement, oxygen Some of the non-medical procedures include saturation tests, pulmonary function tests, etc. Non-exhaustive examples include medications, dietary therapy, surgery, radiation therapy, chemotherapy, physical therapy, cardiac These may include physical therapy, blood transfusion, infusion, etc. Therefore, the matching subset is displayed. This allows for the reproduction of images of patients lacking the selected patient characteristics while omitting the reproduction of images of patients lacking the selected patient characteristics. The method may include allowing the user to view an image of the patient showing the

[0140] In some embodiments, the selected physical patient characteristics may include certain anatomical structures. As used herein, an anatomical structure is any particular part of an organism. For example, the anatomical structure may be any particular organ, tissue, cell, or other structure of the patient. In this way, for example, a user can receive a video regarding surgery on the pleural sac of a lung. If you want to observe a specific part of the video, you can present that part, while other unrelated parts are can be omitted. The stored event characteristics include patient demographics, medical condition, medical history, previous It may include various other patient characteristics, such as treatment, or any other relevant patient descriptors. This allows observers to identify individuals with very specific characteristics (e.g., 70-75 year old white people, previously biracial, etc.). We can see surgical procedures in patients with coronary heart disease (who have had coronary artery surgery) In this way, one or more patient images that match those specific criteria can be selected. The information can be presented to the user automatically.

[0141] In yet another example, the stored event characteristics may include a physiological response. As used, the term "physiological response" refers to any response that may occur in response to events within a surgical procedure. Some non-limiting examples of such physiological changes include: , changes in blood pressure, changes in oxygen saturation, changes in lung function, changes in respiratory rate, blood composition (quantitative chemistry, etc.) changes in tissue condition (e.g., color, shape, structure), bleeding, leakage, changes in blood flow to tissue, changes in tissue condition (e.g., color, shape, structure, changes in structural and functional status, changes in body temperature, changes in brain behavior, changes in sweating, or changes in surgical Any other physical change in response to the intervention may be included. Selectively view events that may occur during a surgical procedure (omitting playback of inconsistent events) This will allow you to prepare for these events.

[0142] In some examples, the event characteristics may include the skill level of the surgeon. The rule may include any indication of the relative competence of the surgeon. Kill levels depend on the surgeon's experience or skill in performing a surgical procedure or a particular technique within a surgical procedure. may include a score reflecting proficiency. In this way, users can Compare how surgeons with varying experience handle the same procedure by selecting In some embodiments, skill level can be determined by data entry (e.g., handing over the surgeon's ID). Based on surgeon identity determined either via manual entry or machine vision. For example, the disclosed method may be based on biometric analysis (e.g., Determine the surgeon's identity via (e.g., face, voice, etc.) and associate a predetermined skill level with that surgeon The predetermined skill level may include analysis of video footage to identify the By accessing a database that stores skill levels associated with specific surgeons. Skill level may be determined based on the surgeon's past performance, the surgeon's training, the type and / or level of training or education, the number of surgeries performed by the surgeon, and the number of surgeries performed by the surgeon type of medical history, the qualifications of the surgeon, the level of experience of the surgeon, and the results of medical advice from patients or other healthcare professionals. Surgeon's assessment, previous surgical outcomes, previous surgical outcomes and complications, or healthcare professional's The kill level assessment may be based on any other information relevant to the kill level assessment. Alternatively, skill level may be determined automatically based on computer analysis of video footage. For example, the disclosed embodiments may be used to perform a procedure, perform a particular technique, or perform other procedures performed by a surgeon. This may include analyzing video footage capturing the determination made or similar events. The skill level of a surgeon is then determined based on how well the surgeon performs during the event. This may be determined based on timeliness, effectiveness, adherence to preferred techniques, risk of damage or harm, etc. It may be based on the lack of action, or any other indicator of skill that can be gleaned from analysis of the video. .

[0143] In some embodiments, the skill level is determined by the global skill level assigned to each surgeon. For example, the surgeon may A person may have a first skill level with respect to one technique or procedure and a second skill level with respect to a different technique or procedure. The surgeon's skill level may also be determined by the event, technique, and / or may vary throughout the procedure. For example, the surgeon may may operate at one skill level, but at a second skill level during a second portion of the video. Thus, the skill level may be determined by the skill associated with a particular location in the video. The skill level may be multiple skill levels during an event. Alternatively, multiple skills during an event may be aggregated, such as an average, rolling average, or other form of aggregation. In some embodiments, skill level may be an aggregation of surgical, surgical, or surgical skills. The skill level typically required to perform the surgical steps and / or intraoperative surgical events and may not be tied to a particular surgeon or other healthcare professional. Levels can be expressed as a numeric scale (e.g., 1-10, 1-100, etc.), a percentage, or a text. Text-based indicators (e.g., "highly proficient," "moderately proficient," "proficient") scale (e.g., "untrained"), or any other appropriate scale for indicating the surgeon's skill. Skill levels can be expressed in a variety of ways, including in various formats. Although is described as the skill level of the surgeon, in some embodiments, the skill level Are you a surgical technologist, nurse, physician assistant, anesthesiologist, doctor, or any other healthcare professional? It may be associated with another healthcare professional, such as a home.

[0144] Embodiments of the present disclosure provide for the collection of aggregate data relating to a particular surgical procedure and multiple similar surgical procedures. The aggregated data may be collected and processed from multiple sources. Aggregated data may refer to data that is linked to a particular surgical procedure and / or combined data. For example, the surgical procedure involves the same or similar surgical steps, involves the same or similar intraoperative events, or involves the same or similar tags or characteristics (e.g., event tags, stage tags, event characteristics, or other tags) may be considered similar to certain surgical procedures when associated with .

[0145] The present disclosure further includes presenting statistical information related to the selected event characteristics to the user. Statistical information can be used to analyze any number of surgical procedures together. Statistical information can refer to the average value, data trend, standard deviation, and variance. , correlation, causality, test statistics (t-statistics, chi-squared statistics, f-statistics, or other forms of test statistics) totals), order statistics (including maximum and minimum sample values), and graphical displays (e.g. For example, a chart, graph, plot, or other visual or graphical display) or similar An illustrative example is when a user In embodiments that select an event characteristic that includes the identity of a particular surgeon, the statistical information may be The average duration of surgery (or stages or events of surgery), adverse or other effects of the surgeon This may include the rate of outcomes, the average skill level of surgeons performing intraoperative events, or similar statistics. Those skilled in the art will appreciate other forms of statistical information that may be presented in accordance with the disclosed embodiments. I can understand that.

[0146] 8A and 8B illustrate a method for video indexing consistent with disclosed embodiments. 8 is a flowchart illustrating an example process 800. The process 800 includes at least one The method may be performed by a processing device such as a processor. The processor is an application specific integrated circuit (ASIC), a microchip, or a microcontroller. processor, microprocessor, central processing unit (CPU), graphics processing unit (GPU) ), Digital Signal Processor (DSP), Field Programmable Gate Array (FP GA), server, virtual server, or other circuit suitable for executing instructions or performing logical operations. The circuit may include one or more integrated circuits (ICs) that contain all or part of the circuit. The instructions executed by the processor may be, for example, integrated with or controlled by a controller. The data may be preloaded into a memory incorporated in the controller or may be stored in a separate memory. The memory may be a random access memory (RAM), a read-only memory (ROM), or (ROM), hard disk, optical disk, magnetic media, flash memory, other permanent, It may be a fixed or volatile memory or any other mechanism capable of storing instructions. In some embodiments, at least one processor may include two or more Each processor may have a similar structure, or the processors may The processors may be of different structures that are electrically connected or disconnected from each other. The processors may be separate circuits or may be integrated into a single circuit. When processors are used, the processors may be configured to operate independently or cooperatively. The processor may be electrically, magnetically, optically, acoustically, mechanically, or by any of the methods described above. They may also be coupled by other means that allow them to interact.

[0147] In some embodiments, the non-transitory computer-readable medium is executable by a processor. Step 8 may include instructions that, when executed, cause a processor to perform process 800. At 02, the process 800 accesses the video footage to be indexed. and the indexed video footage may include footage of a particular surgical procedure. Video footage is accessed from local memory, such as a local hard drive. It may also be accessible from a remote source, for example via a network connection. In this example, video footage is captured using one or more image sensors or a separate process. In step 804, the process 800 analyzes the video footage. and identifying locations in the video footage that are related to the operative steps of a particular surgical procedure. As mentioned above, the position can be a specific frame, a range of frames, or a time index. The location may be associated with a time range, a time period, or any other location identifier.

[0148] The process 800 generates a stage tag associated with the surgical stage, as shown in step 806. This can include, for example, video motion detection, video tracking, shape recognition, object recognition, etc. body detection, fluid flow detection, equipment identification, behavior analysis, or other forms of computer-aided situational awareness This can be done by video content analysis (VCA) using techniques such as one or more of When a learned feature associated with a stage is identified in the video, a tag defining that stage is generated. The tag can include, for example, a predetermined name for the step. At 808, the process 800 includes associating a stage tag with a location in the video footage. A stage tag may be used to indicate, for example, that an identified video footage location corresponds to a surgical stage of a particular surgical procedure. In step 810, the process 8 00 uses the VCA described above to identify the event location of a specific intraoperative surgical event within a surgical phase. The process may include analyzing the video footage using one or more of the following techniques: The process 800 may then assign an event tag to the event location of a particular intraoperative surgical event, as shown in step 812. The event tag can include, for example, associating the video footage with the surgical location. Like stage tags, event tags can indicate that they are associated with an event. In step 814, in FIG. 8B, The process 800 can include storing event characteristics associated with a particular intraoperative surgical event. As described in more detail above, event characteristics include adverse outcomes of surgical events, surgical procedures, surgeons, and skill level, patient characteristics, specific surgeon identity, physiological response, duration of event, or any other characteristic or property associated with the event. Event characteristics may include manual It may be determined (e.g., entered by the viewer), for example, as described above, This may be determined automatically by artificial intelligence applied to thin vision. Characteristics include skill level (e.g., minimum skill level required, skill level demonstrated during the event, etc.). training to determine such skill level from the video. You can train a machine learning model using training examples, and It can analyze video footage and determine skill level using a machine learning model developed Examples of such training examples are presented together with labels indicating the corresponding skill level. In another example, the event characteristics may include the time of the event. It may also include associated characteristics (e.g., start time, end time, duration, etc.), and such The correlation properties may be calculated by analyzing intervals in the video footage that correspond to events. In yet another example, the event characteristics may include an event type, and the event type may be determined from the video. You can train a machine learning model using training examples to determine It also uses trained machine learning models to analyze video footage and identify incident types. An example of such a training example is a label indicating the event type. In a further example, the event characteristics may include a video clip that together depict the event. Information related to the medical device involved in the incident (e.g., type of medical device, how the medical device is used, etc.) The machine learning model can extract such information related to the medical device from the video. It can be trained using training examples to identify information and The trained machine learning model analyzes the video footage and identifies the medical devices involved in the incident. An example of such a training example is a medical device Video clips depicting events involving the use of the medical device, along with labels showing relevant information about the device. In yet another example, the event characteristics may include a characteristic associated with an anatomical structure involved in the event. Relevant information (e.g., type of anatomy, condition of anatomy, and the cause of the event) The machine learning model can then extract anatomy from the video. training examples to identify such information related to the structural The trained machine learning model can then analyze the video footage and , can be used to determine information related to the anatomical structures involved in the event. An example of a suitable training example is an anatomical structure with labels showing information related to the anatomy. A video clip depicting the event including the structure may be included. The data may include information related to medical outcomes associated with the event, and the machine learning model may To identify such information related to medical outcomes from the data, training examples are used. The trained machine learning model can then be used to analyze the video footage. can be analyzed to determine information related to medical outcomes associated with the event An example of such a training example is a set of visuals that show medical outcomes, along with labels that show the medical outcomes. A deoclip may be included.

[0149] In step 816, the process 800 begins by extracting at least one of the video footage of the particular surgical procedure. and assigning at least some of the data to the stage tag, event tag, and / or event characteristics in the data structure. This step involves associating tags with the video. Various tags are associated with the video footage to enable it to be used for Various data structures are used to store related data in an associated manner, such as It is possible.

[0150] In step 818, the process 800 selects the selected stage tag, the selected event tag, and and selecting at least one of selected event characteristics of the video footage for display. This may include allowing users to access the data structure. In an embodiment, a user may access a computing device such as the user interface 700 shown in FIG. Selected stage tags, selected event tags, etc., are input via the user interface of the device. and selected event characteristics. For example, data entry fields, Provide drop-down menus, icons, or other selectable items to facilitate surgical procedures, treatments, and Allowing users to select stages of a procedure, events within a procedure, and procedure and patient characteristics In step 820, the process 800 searches for a matching sub-segment of the stored video footage. To identify a subset, at least one selected stage tag, a selected event tag, , and lookup in the data structure for surgical video footage that matches the selected event characteristics. In step 822, the process 800 may include performing displaying a matching subset of the selected video footage to the user, thereby The event characteristics are selected based on the video quality of the video. This may include allowing the user to view surgical footage of an intraoperative surgical event. This filtering allows users to filter out large amounts of video data that are not relevant to their interests. The user can quickly view only the video segments that correspond to their interest while skipping the playback of the data. This can be done.

[0151] When preparing for a surgical procedure, the surgeon may view video footage of a surgical procedure with similar surgical events. Reviewing the images can be beneficial; however, it can be time-consuming for surgeons to view the entire video. It takes too long or it takes too long to skip around and find the relevant part of the surgery video. Therefore, it is important to record images of relevant surgical events while omitting other unrelated images. Enabling surgeons to efficiently and effectively view surgical video summaries that aggregate images Non-traditional approaches are needed.

[0152] Aspects of the present disclosure include methods, systems, devices, and computer-readable media. It can involve the generation of an overview video, for example, a video of a surgical procedure previously analyzed. By comparing the recorded surgical footage, relevant intraoperative surgical events can be identified and tagged. The surgeon uses a surgical video that aggregates intraoperative surgical events while omitting many of the other irrelevant images. For ease of explanation, the method aspects may be summarized as systems, devices, The methods are described below with the understanding that they apply equally to computer readable media. For example, some aspects of such methods may involve wired, wireless, or both networks. Other aspects of such methods may be performed using non-electronic means. In its broadest sense, the method is not limited to specific physical and / or electronic means. The method is not fixed, but rather can be achieved using many different means.

[0153] Consistent with disclosed embodiments, the method may include: The method may include accessing a particular surgical video including a first group of frames. Surgical footage can be any video, group of video frames, or display of a surgical procedure. For example, surgical footage may refer to one or more videos captured during a surgical procedure. The surgical video may include a video frame of a memory device. The surgical video may include retrieving the video from a storage location such as a local hardware device. It may be accessed from local memory, such as a hard drive, or over a network, for example. may be accessed from a remote source via a connection. Generally, video refers to any form of recorded visual media containing recorded images and / or sound. Video can include Audio-Video Interleaved (AVI) files, Flash Flash Video Format (FLV) files, QuickTime file formats MOV, MPEG (MPG, MP4, M4P, etc.), Windows Media ia Video (WMV) files, Material Exchange Format (M XF) file, or any other suitable video file format Additionally or alternatively, in some instances, a particular surgical image may be stored as a file. Accessing the surgical system involves capturing specific surgical footage using one or more image sensors. may include:

[0154] As noted above, an intraoperative surgical event is any event or action associated with a surgical procedure or step. A frame can refer to one of several still images that make up a video. The first group of frames may include frames captured during an intraoperative surgical event. For example, a particular surgical procedure may be performed on a patient and recorded in at least one operating room. The surgical procedure can be depicted as being captured by an image sensor, e.g. For example, cameras 115, 121, 123, and / or 125 may be located in the operating room 101. In some embodiments, the at least one image sensor may include a handheld device. At least one of above the operating table and within the patient. For example, the image sensor may be The surgical instrument may be positioned above the patient, such as above the patient, or within the patient's surgical cavity, organ, or vasculature. The first group of frames may be placed within the anatomy, surgical instruments, and intraoperative hands. Intraoperative procedures, including healthcare professionals performing surgical events or other visual indications of intraoperative surgical events However, in some embodiments, a portion of a frame or All of these may not include an indication of an intraoperative surgical event, but may be otherwise associated with the event. The event may be captured (e.g., caught while it is being executed).

[0155] Consistent with this disclosure, certain surgical footage may include a second group of frames not related to the surgical action. For example, a surgical procedure may involve the use of a surgical procedure in which no significant surgical action is being performed and the video This can involve lengthy downtime without significant reason to review the surgical procedure. may refer to any action taken in connection with a surgical procedure. The surgical behaviors include preoperative behaviors, perioperative behaviors, intraoperative behaviors, and / or postoperative behaviors. It can broadly refer to any action related to treatment. A group may contain frames that are not associated with such an action. In the present context, surgical actions include physical manipulation of a patient's organs or tissues performed by a surgeon. The second group of frames can refer to a narrower set of actions. preparing, providing anesthesia, monitoring vital signs, collecting or preparing surgical instruments, and discussion of various actions related to surgical procedures, or other actions that may not be considered surgical actions. It is possible.

[0156] According to the present disclosure, the method provides for the generation of historical data based on historical surgical footage of previous surgical procedures. The historical data may include accessing previously recorded and / or stored data. In some embodiments, historical data may refer to data in any format. The data may be one or more video files containing historical surgical footage. For example, historical data The data may include a series of frames captured during a previous surgical procedure. Therefore, this historical data is not limited to video files. For example, the historical data may be The surgical video may include information stored as text representing at least one aspect of the surgical video. For example, historical data may include a database of information summarizing or referencing historical surgical footage. In another example, the historical data may include a number representing at least one aspect of the historical surgical footage. In a further example, the historical data may include information stored as values. In yet another example, the method may include statistical information and / or statistical models based on analysis of the image. Historical data includes machine learning models that have been trained using training examples. Training examples can be based on historical surgical footage. The method includes receiving the historical data via electronic transmission, storing the historical data in a storage device (e.g., a memory device). reading historical data from the device, or any other program to access the data In some embodiments, the historical data may include the specific surgical process described above. In another embodiment, the historical data can be accessed from the same resources as the video. Additionally or alternatively, historical data may be accessed from a separate resource. Accessing the surgical procedure may be accomplished, for example, by analyzing historical surgical footage of previous surgical procedures or By analyzing data based on historical surgical footage of previous surgical procedures, The method may include generating data.

[0157] According to an embodiment of the present disclosure, the historical data associates portions of the surgical footage with intraoperative surgical events. It may contain information that distinguishes frames from frames that are not related to surgical activities. There are various ways to distinguish between parts of a surgical video. For example, In this context, even if frames associated with surgical and non-surgical actions have already been distinguished, This can be done, for example, by manual flagging of surgical actions or by separating surgical and non-surgical actions. This may already be done by training an artificial intelligence engine to distinguish between The history information may be, for example, a set of frames of a surgical video (e.g., starting frame number, The information can be identified using the frame number, end frame number, etc. Start timestamp, end timestamp, duration, timestamp range, or surgical footage The time information may include other information about the timing of the images. The data may include various indicators and / or rules that distinguish surgical behavior from non-surgical behavior. Some non-limiting examples of such indicators and / or rules are described below. The historical data may be, for example, based on historical surgical videos, and may include a portion of the video corresponding to a surgical action and and / or a machine learning model trained to identify portions of the video that correspond to non-surgical activities. It may include Dell.

[0158] Using a variety of indicators, the analysis can be performed manually, semi-manually, or automatically (e.g., via machine learning). For example, in some embodiments, surgical actions can be distinguished from non-surgical actions. The information that distinguishes a portion of the historical surgical footage into frames related to intraoperative surgical events is The surgical instrument may include an indication of at least one of the presence or movement of the surgical instrument. The device may be any instrument or device that can be used during a surgical procedure, and such an instrument or device A vice may include cutting instruments (e.g., scalpels, scissors, saws, etc.), gripping and / or holding tools. Holding instruments (e.g., Billroth forceps, hemostatic "mosquito" forceps, atraumatic hemostatic forceps, deshamp needles, Hoppner hemostatic forceps, etc.), retractors (e.g., Farabef C-shaped lamellar hooks, blunt Toothed hooks, sharp toothed hooks, grooved probes, tamp forceps, etc.), tissue integration instruments and / or materials (e.g., needle holders, surgical needles, staplers, clips, adhesive tape, metal protective equipment (e.g., face and / or respiratory protection, hats, footwear, gloves, etc.) These include, but are not limited to, laparoscopes, endoscopes, patient monitoring devices, and the like. Using video or image analysis algorithms such as those described above for video indexing. can be used to detect the presence and / or movement of surgical instruments in the image. The surgical instrument movement index can be calculated and the calculated movement index can be compared to a selected threshold. The threshold can be compared to a value to distinguish surgical from non-surgical behavior. based on the type of surgical procedure, based on the time of the surgical procedure or within the surgical procedure, The parameters were determined by analyzing video footage of the surgical procedure based on the stages of the procedure. based on parameters determined by analyzing historical data, etc. In some instances, signal processing algorithms can be used to Analyze calculated movement measurements at various times within the procedural video footage and compare surgical and non-surgical behaviors. Some non-limiting examples of such signal processing algorithms are: For example, training examples are used to distinguish surgical from non-surgical behaviors. A trained machine learning-based signal processing algorithm to distinguish surgical from non-surgical behaviors. an artificial neural network (e.g., a recurrent neural network) configured to , long short-term memory neural networks, deep neural networks, etc.), Markov Examples include the Viterbi model.

[0159] In some exemplary embodiments, a portion of the historical surgical footage may be included in a frame associated with an intraoperative surgical event. The frame distinguishing information includes the detected instruments and anatomical features in the associated frame. For example, the disclosed methods may utilize image and / or video analysis algorithms. The method may include detecting instruments and anatomical features using a hologram. The anatomical features may include the anatomical structure of a living body. The surgical procedure may involve the anatomy (defined in more detail above) or other parts. The surgical instruments interacting with the physical structures are included in the hands detected in one or more relevant frames. The presence of both surgical instruments and anatomical structures can serve as indicators of surgical action. In response to detecting the first instrument in the group of frames, the group of frames is while the first fixture in the group of frames may be determined to be associated with In response to the absence of detection, the group of frames is not associated with an intraoperative surgical event. In another example, a first anatomical feature may be identified in a group of frames. Accordingly, the group of frames may be determined to be associated with an intraoperative surgical event. On the other hand, in response to the absence of the first anatomical feature detection in a group of frames, A group of systems may be identified as not being associated with an intraoperative surgical event. In this study, we further analyzed the video footage to examine the interactions between the detected instruments and anatomical features. and distinguishing surgical from non-surgical behaviors is a key factor in the interactions detected. For example, upon detection of a first interaction in a group of frames. Thus, a group of frames may be determined to be associated with an intraoperative surgical event, On the other hand, in response to the absence of a first interaction detection within a group of frames, the group of frames The video may be identified as not being associated with an intraoperative surgical event. The video footage can be further analyzed to detect the actions performed by the detected device. Also, distinguishing surgical from non-surgical behaviors can be based on the detected behaviors. For example, in response to detecting a first motion within a group of frames, the group of frames may , may be determined to be associated with an intraoperative surgical event, while within a group of frames In response to the absence of a first motion detection, the group of frames is associated with an intraoperative surgical event. In some instances, the video footage may be further analyzed to identify anatomical features. detects changes in the state of critical features and distinguishes surgical from non-surgical behaviors based on the detected changes For example, in response to detecting the first change in a group of frames, may be determined to be associated with an intraoperative surgical event, while a group of frames In response to the absence of the first change in the group, the group of frames was may be identified as not being associated with elephants.

[0160] Some aspects of the present invention provide for the determination of frame rate in a particular surgical video based on information from historical data. and distinguishing a first group of frames from a second group of frames. For example, information may be provided regarding which frames of a particular surgical video are associated with intraoperative events and / or surgical actions. This can provide useful context for determining what is being attached. In an embodiment, a first group of frames is divided into a second group of frames in a particular surgical video. Differentiating from the group may involve the use of machine learning algorithms. The learning model uses training examples based on information from historical data to identify intraoperative events and / or It can be trained to identify surgical actions.

[0161] According to the present disclosure, the first and second groups of frames are analyzed to generate historical data. The first frame of the frame can be distinguished by identifying the information similar to that of the first frame. 9 illustrates an example process 900 for distinguishing a group from a second group of frames. It should be understood that process 900 is provided as an example. Those skilled in the art will understand, consistent with this disclosure, how to convert a first group of frames into a second group of frames. Various other processes for distinguishing between the The method 900 may include analyzing a particular surgical image to detect a medical device. A medical device, as described above, is any instrument or device used in the treatment of a patient, including a surgical instrument. In addition to the surgical instruments mentioned above, medical equipment includes stethoscopes, gadgets, and other surgical instruments. Sponges, catheters, cannulas, defibrillators, needles, trays, lighting, thermometers, pipettes or drippers, oxygen masks and tubing, or any other medical equipment. For example, but not limited to, a machine learning model can be used to identify medical devices in images and / or videos. It may be trained using training examples to detect the The resulting machine learning model is then used to analyze specific surgical footage and detect medical devices. An example of such a training example is one or more along with labels indicating the presence of specific medical devices on the may include video and / or images of the surgical procedure along with a label indicating the absence of medical equipment. do.

[0162] In step 920, the process 900 analyzes the particular surgical image to identify anatomical structures. The anatomical structure may include any organ, organ, or the like, as described above. As mentioned above, one or more video and and / or use image recognition algorithms to detect medical devices and / or anatomical structures. For example, machine learning models can detect anatomical structures in images and / or videos. It may be trained using training examples to The machine learning model developed is used to analyze specific surgical footage and detect anatomical structures. An example of such a training example is one or more Along with labels indicating the presence of specific anatomical structures or specific solutions within the video and / or images May include videos and / or images of surgical procedures, along with labels indicating the absence of anatomical structures. can.

[0163] In step 930, the process 900 performs a comparison of the detected medical devices and the detected anatomy. The method may include analyzing the video to detect relative motion between the object and the object. Pairwise motion can be, for example, pixel change between frames, optical flow, or other forms of motion. Based on the motion detection algorithm, the motion detection algorithm can be used to detect For example, a motion detection algorithm is used to estimate the movement of medical equipment in the video and The movement of anatomical structures within the device can be estimated, and the estimated movement of the medical device can be analyzed. The relative motion can be determined by comparing with the estimated movement of the anatomical structures. At 940, the process 900 filters the first group of frames based on the relative motion. This may include distinguishing the first group of frames from the second group of frames, in which case The first group of frames contains surgical action frames, and the second group of frames contains non-surgical action frames. For example, a group of frames may be A group may be determined to include a surgical action, and a second group of frames may be determined to include a surgical action. In response to the detection of relative motion patterns, the group of frames is divided into groups that do not include non-surgical action frames. Therefore, by presenting an aggregation of the first group of frames, Therefore, surgeons preparing for surgery should be able to observe non-surgical behaviors during video review of the summarized presentation. In some embodiments, non-surgical action frames may be omitted. This may involve omitting most of the frames that capture non-surgical actions. Frames immediately before or after intraoperative events, non-surgical actions that provide context to intraoperative surgical events frames capturing non-surgical actions, such as frames capturing a surgical procedure, or any other frames that may be relevant to the user. In some cases, not all captured frames may be omitted.

[0164] In some exemplary embodiments of the present disclosure, the first group of frames is divided into the second group of frames. The key to distinguishing these groups is the detected relative position between the medical device and the anatomical structure. The relative position can be based on the distance between the medical device and the anatomical structure, the anatomical structure, It can refer to the orientation of a medical device relative to anatomy or the location of a medical device relative to anatomy. For example, the relative positions can be determined by comparing detected medical devices and The relative position may be estimated based on the relative position of the anatomical structures. For example, the relative position may be determined based on the distance (e.g., It can include dimensions (in pixels, real-world measurements, etc.), directions, vectors, etc. uses object detection algorithms to determine the location of medical devices and locate anatomical structures. The two determined positions can be compared to determine the relative position. In one example, a group of frames can be sorted according to a first relative position within the group of frames. A group may be determined to include a surgical action, while a second relative position within a group of frames may be determined to include a surgical action. In response to the detection of a surgical position, a group of frames may be identified as non-surgical action frames. In another example, the distance between the medical device and the anatomical structure may be compared to a selected threshold. Often, it is also possible to distinguish a first group of frames from a second group of frames. The threshold may be further based on the results of the comparison. For example, the threshold may be based on the type of medical device, the anatomical structure, The selection may be based on the type of implant, the type of surgical procedure, etc. Distinguishing the first group of frames from the second group of frames is the difference between medical devices and The interaction can be further based on detected interactions between anatomical structures. This may include any action by a medical device that may affect a physical structure, or vice versa. For example, interactions can include contact between a medical device and an anatomical structure, Action by the medical device on the mechanical structure (e.g., cutting, clamping, applying pressure, scraping, etc.) (e.g., muscle movements), reactions by anatomical structures (e.g., reflexes), or any other form of interaction. For example, machine learning models can detect the relationship between medical devices and anatomical structures from videos. may be trained using training examples to detect interactions, and The trained machine learning model analyzes video footage and identifies medical devices and anatomical structures. An example of such a training example is Labels indicating the presence of specific interactions between medical devices and anatomical structures within the Deoclip together, or the absence of specific interactions between medical devices and anatomical structures in the video clip. A video clip of the surgical procedure can be included along with indicative labels.

[0165] Some aspects of the present disclosure may include, upon user request, providing a second group of frames to the user. presenting a first group of frames of a particular surgical video to the user while omitting the presentation to the The aggregation of the first group of frames may be presented in various forms. In some embodiments, the aggregation of the first group of frames may be included in a video file. The video file may include a video clip including a first group of frames. In some embodiments, the user may select a video clip. Each may be presented separately or as a single compiled video. In some embodiments, a separate video file is generated for the aggregation of the first group of frames. In another embodiment, the aggregation of the first group of frames may be performed by: instructions to identify frames to be included and frames to be omitted for presentation The execution of the instructions may be presented to the user as if a continuous video were being generated. Various other methods may be used, including presenting the first group of frames as a still image. Formats can also be used.

[0166] Presentation may include any process for providing an aggregation to a user. So this is a computer screen or monitor, projector, mobile phone display , tablet, smart device, or other device capable of displaying images and / or audio The method may include playing the aggregation on a display of any device, and , presenting includes transmitting an aggregate of a first group of frames to a user, or This may include making the content accessible to users otherwise. The first group of systems aggregates data from the user's computing device via the network. As another example, the location of the aggregation of the first group of frames may be The second group of frames may be aggregated and shared with For example, if the aggregation is presented as a video, The video clip comprising the second group of frames is not included in the video file. The first group of frames may be presented in any order, including chronological order. Optionally, present at least some of the first group of frames in a non-chronological order. In some embodiments, the aggregation of the first group of frames may be: For example, a user may be able to view multiple intraoperative surgical events within a particular surgical video. A user may request to view multiple intraoperative surgical events. Presenting the frames to the user is done in chronological order, omitting the frames from the second group. The display may include displaying a group frame.

[0167] A user is any person or entity that may require access to surgical summary videos. In some embodiments, the user may be a surgeon or other healthcare professional. For example, a surgeon may request a surgical summary video for review or training purposes. In some embodiments, the user may be an administrator, a supervisor, a surgeon, an insurance company official, a regulatory authority, or a regulatory authorities, police or investigative authorities, or any other entity that may require access to surgical footage. Various other examples of user behavior in connection with video indexing techniques include: is given above. Users can use laptops, desktop computers, mobile phones, Tablet, smart glasses, or any other form of computer that can submit a request. You may submit a request via a computing device, such as a In some embodiments, the request may be received electronically over a network. Alternatively, aggregations may be presented based on receiving a request.

[0168] In some exemplary embodiments, the user request includes at least one type of technique of interest. and a first group of frames including a representation of an intraoperative event, and the first group of frames including at least one type of The method may depict at least one intraoperative surgical event of the subject. The type of event can be any category into which an intraoperative surgical event can be classified. For example, type These include the type of procedure being performed, the stage of the procedure, and whether the intraoperative surgical event was harmful. , whether the intraoperative surgical event is part of a planned procedure, the surgeon's Identification information, purpose of the intraoperative surgical event, medical condition related to the intraoperative surgical event, or any other category A category or classification can be mentioned.

[0169] An embodiment of the present disclosure encodes a first group of frames for storage in a patient's medical record. As described above, the specific surgical image may be exported to a patient. The disclosed methods can be used to depict surgical procedures performed on at least A first group of frames associated with at least one intraoperative surgical event is sent to the patient's physician. As used herein, a medical record may be associated with a diagnosis, treatment, and and / or may include any form of documentation of information relating to the patient's health, including care. Medical records may be stored in a digital format, such as an electronic medical record (EMR). Exporting the first group of frames means transmitting or otherwise made available for storage in the medical record or otherwise filed. This may include associating the first group of frames with a medical record. For example, the first group of frames (or a copy of the first group of frames) is added to the database. In some embodiments, the disclosed The method includes associating a first group of frames with a unique patient identifier; and and updating the medical record to include the unique patient identifier. The identifier may be any indicator, such as an alphanumeric string, that uniquely identifies a patient. The string can anonymize the patient, which may be required for privacy purposes. Where privacy is not an issue, the unique patient identifier may be the patient's name and / or social security number. It may contain a fault number.

[0170] In some exemplary embodiments, the disclosed method comprises: As described above, the index may further include generating an index. It can refer to a form of data storage that allows retrieval of video frames. Indexing is a more efficient and / or effective way of doing things than not indexing. The index is shown in the first group of frames. Include a list or other itemization of intraoperative surgical events that are relevant or otherwise related. Exporting the first group of frames means exporting the first group of frames. generating a compilation of the loop, the compilation including the index viewing at least one intraoperative surgical event based on the selection of one or more index items; For example, by selecting "incision" through the index, This allows the user to be presented with a compilation of surgical footage depicting the incision. In some embodiments, the compilation may include a continuous video For example, the user may allows you to select multiple intraoperative events via indexes, and Related frames can be compiled into a single video.

[0171] Embodiments of the present disclosure may further include generating a causal relationship summary. The summary provides a clear overview of the clips related to the underlying stage of the surgical procedure without the need to view intermediate clips or images. A clip or image of the outcome and associated outcome steps can be viewed by the user. As used herein, "cause" refers to something that brings about a particular result, phenomenon, or condition. An "outcome" refers to a phenomenon or condition that can be attributed to a cause. In this embodiment, the outcome can be an adverse outcome. For example, the outcome can be bleeding, pneumomesenteric, , injury, conversion to unplanned open surgery (e.g., laparotomy), more severe than planned The causes include adverse events such as surgeon error. For example, an error may be an action that has a negative or adverse effect on the errors, communication errors, administrative errors, judgment errors, decision-making errors, and errors in the use of medical equipment. The outcome may include errors made during surgery, or other forms of errors that may occur. It may include a positive or predicted outcome of a treatment, or stage, etc.

[0172] In embodiments where a causal summary is generated, the historical data includes historical surgical outcome data and Each of the historical cause data may further be included. The historical causal data can be used to show the portion of historical surgical footage associated with the cause. The portion of the historical surgical footage associated with each cause of outcome can be shown. In such an embodiment, the first group of frames is a causal set of frames and a The second group of frames may include an intermediate set of frames. may include:

[0173] FIG. 10 illustrates an exemplary process for generating a causal summary consistent with disclosed embodiments. 1 is a flowchart illustrating process 1000. Process 1000 is given as an example. Those skilled in the art will also appreciate that various other processes for generating causal summaries consistent with this disclosure are possible. In step 1010, the process 1000 analyzes the particular surgical video. and identifying the surgical outcome and each cause of the surgical outcome, The analysis is based on historical outcome data and respective historical causal data. This can be done using image and / or video processing algorithms, such as: In this embodiment, step 1010 uses historical data to analyze a particular surgical video. A machine trained to identify surgical outcomes and their respective causes using For example, a machine learning model may be used to identify known or unknown Training can be based on historical data with specific surgical outcomes and their respective causes. The trained model can then be used to analyze other videos, such as specific surgical videos. Such machine learning models can identify surgical outcomes and their respective causes. An example of a training example used for training corresponds to a video clip Surgical procedures, along with labels indicating the surgical outcomes and, in some cases, the causes of each surgical outcome Such training examples may include video clips of the user's position. Historical data including video clips from historical data, including outcomes determined based on the data It can be data-based.

[0174] In step 1020, the process 1000 determines whether or not a particular surgical video is being used based on the analysis. The method may include detecting an outcome set for the frame, the outcome set for the frame being a surgical outcome set. The outcome stage is within the surgical procedure outcome stage. The outcome stage is the time of the surgical procedure related to the outcome as described above. In step 1030, the process 1000 determines whether the This may include detecting a causal set of frames in a particular surgical video, The causal set of the system is within the causal stage of the surgical procedure, which is temporally separated from the outcome stage. In embodiments, the outcome step may include a surgical step in which the outcome is observable; and , the outcome set of frames can be a subset of the frames in the outcome stage. A stage may be a time period or portion of a surgical procedure that is related to the cause of the outcome in the outcome stage. In some embodiments, the cause stage may include the surgical stage at which the cause occurred, The causal set of a frame can be a subset of the frames in the causal stage. The set is an intermediate step between the cause set of the frame and the outcome set of the frame. In step 1040, the process 1000 generates a causal profile of the surgical video. and the causal summary may include generating a causal set of frames and a causal set of frames. In some embodiments, the causal relationships include the outcome set of frames and omit the intermediate set of frames. The association summary may be similar to the aggregation of the first group of frames, as described above. Thus, the causal summary is associated with the cause set of the frame and the outcome set of the frame. As mentioned above, the frames presented to the user may include a compilation of video clips. The first group of aggregations of the system may include a causal summary.

[0175] FIG. 11 illustrates an exemplary process for generating a surgical overview video consistent with disclosed embodiments. 11 is a flowchart illustrating a process 1100 performed by a processing device. In some embodiments, the non-transitory computer-readable medium may be executed by a processor. 11. The method of claim 11, further comprising: In step 1110, the process 1100 performs at least one intraoperative procedure. The first group of frames associated with events and frames not associated with surgical actions. and a second group of surgical images. The first group of frames is associated with multiple intraoperative surgical events, as detailed in Furthermore, in some embodiments, As described above with respect to process 1000, the first group of frames is A second group of frames may be included, which may include a cause set and an outcome set of frames. A group can contain an intermediate set of frames.

[0176] In step 1120, the process 1100 begins by recording historical surgical footage of previous surgical procedures. The method may include accessing historical data based on the images, where a portion of the surgical video is The frames are divided into frames associated with intraoperative surgical events and frames not associated with surgical actions. In some embodiments, a portion of the historical surgical footage may be associated with an intraoperative surgical event. The information that distinguishes the associated frames is the presence or movement of surgical instruments and / or anatomical features. In step 1130, the process 1 100, based on information from historical data, in a particular surgical video, a first group of frames The method may include distinguishing the first group from the second group of frames.

[0177] In step 1140, the process 1100 generates a frame based on the user's request. a first group of frames of a particular surgical video while omitting presentation to the user of a second group of frames of the particular surgical video. The user's request may include presenting the aggregation of the loop to the user. A computing device that can include a user interface that enables In some embodiments, the user may receive a specific Further frames associated with a type or category can be requested. Process 1 Based on the steps described in 100, the user can create a frame containing frames related to intraoperative events. A summary may be presented that omits frames that are not relevant to the surgical action, along with the , a training video that summarizes intraoperative surgical events while omitting much of the other irrelevant footage. It can be used by the surgeon as

[0178] When preparing for a surgical procedure, the surgeon may choose to perform several surgical procedures with similar surgical events. Reviewing video footage can be beneficial. Traditionally, surgeons have been known to Video footage of surgical procedures with elephants may not be readily accessible. Even if the footage is accessed, it may be difficult to watch the entire video or find relevant parts of the video. Therefore, it may take too long for a surgeon to Efficiently and effectively view video compilations of surgical events from surgeries performed There is a need for non-traditional approaches that make this possible.

[0179] Aspects of the present disclosure include surgical procedures, including methods, systems, devices, and computer-readable media. In particular, different events in procedures performed on different patients may be related to the The compilation video may be presented to a surgeon or other user. Surgical video excerpts of different intraoperative events from similar surgical procedures that can be automatically aggregated in a format The surgeon may choose to include a compilation of selected video segments from similar surgeries on different patients. You can enter case-specific information to search for the compilation. The study included one intraoperative event from one surgery and one or more other different intraoperative events from a second surgery. For example, different complications that occur when operating on different patients can be Videos of multiple surgical procedures may share common characteristics (e.g., In situations involving the same event with similar technology used, the system Imaging may be omitted from one or more surgical procedures to facilitate the

[0180] For ease of description, aspects of the method may be described as systems, devices, and computer-readable media. The methods are described below with the understanding that they apply equally to the body. For example, such methods Some aspects of the present invention may be performed electronically over wired, wireless, or both networks. Other aspects of such methods may be performed using non-electronic means. In this regard, the method is not limited to any particular physical and / or electronic means, but rather may be implemented using many This can be achieved using different means.

[0181] Consistent with disclosed embodiments, a method may include: As used herein, a repository can involve accessing a video Refers to any storage location or set of storage locations where video images can be electronically stored. For example, the repository can be stored on a hard drive and / or a flash drive. In some embodiments, the repository may include a server, a client, a storage device, or a network. A cloud storage location, a shared network drive, or over a network It may also be a network location such as any other form of accessible storage. The repository contains a database of surgical video footage captured at various times and / or locations. In some embodiments, the repository may contain additional data beyond surgical video footage. The data can be stored.

[0182] As mentioned above, surgical video footage may be any video, group of video frames, or It can refer to video footage that includes a display of a surgical procedure. For example, surgical footage can include footage of a surgical procedure being performed during surgery. The set of surgical video footage may include one or more captured video frames. It can refer to a grouping of one or more surgical videos or surgical video clips. The videos may be stored in the same location or may be selected from multiple storage locations. The footage in the set may, but need not, be related in some way. For example: For example, video footage on set may be from the same facility, recorded by the same capture device. recorded at the same time or within the same time frame, for the same patient or group of patients Describes surgical procedures performed in conjunction with other procedures; Describes the same or similar surgical procedures; Common characteristics (For example, similar events involving the use of similar technology, including the use of similar medical devices, etc.) surgical procedures that share the same level of complexity, etc., or any other characteristic or features Depicts, may include video.

[0183] Multiple surgical video sets reflect multiple surgical procedures performed on different patients For example, patients who have undergone the same or similar surgical procedures or who have used similar techniques may be A number of different individuals who have undergone a given surgical procedure are included in a common set or sets. Alternatively or additionally, one or more sets may be collected from a single patient at different times. Multiple surgical procedures may include, for example, all appendectomies. They may be of the same type, including the same In this embodiment, the multiple surgical procedures share common features, such as the same or similar stages or intraoperative events. Sex can be shared.

[0184] Multiple surgical video sets were analyzed to assess intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and and intraoperative surgical event characteristics. Examples of such events, outcomes, and characteristics include: As described throughout this disclosure, surgical outcome refers to the outcome of the entire surgical procedure (e.g., patient whether the patient recovered or fully recovered, whether the patient was readmitted after discharge, whether the surgery successful or not), or the outcome of individual steps or events within a surgical procedure (e.g., whether complications occurred This may include whether a technology has been developed or not, or whether the technology has been successful.

[0185] Some aspects of the present disclosure provide a method for a surgeon preparing for a planned surgical procedure to This may include allowing for the entry of case-specific information corresponding to the surgical procedure. The surgical procedure that has been performed may include any surgical procedure that has not yet been performed. In this context, a surgical procedure is a planned surgical procedure that a surgeon intends to perform on a patient. In other embodiments, the proposed surgical procedure may be a virtual procedure. , may not necessarily be associated with a particular patient. Surgical procedures may be experimental or not widely practiced. does not include any characteristic or features of the proposed surgical procedure or the proposed or hypothetical patient. For example, case-specific information may include characteristics of the patient on whom the procedure is being performed, the the characteristics of the surgeon performing the procedure, the characteristics of other healthcare professionals involved in the procedure, and the characteristics of the procedure being performed. type, specific details or aspects of the procedure, type of equipment or instrumentation involved, type of technology involved type, complicating factors of the procedure, location of the procedure, type or identification of the medical condition being treated Any aspect of the procedure, surgical outcome, intraoperative event outcome, or intended surgical procedure that may define or describe the procedure. Other information that may be included, but is not limited to, case-specific information and The patient's age, weight, medical condition, vital signs, other physical characteristics, past medical history, family history, etc. history of the disease, or any other factors that may have any direct or indirect relationship to the potential outcome. Case-specific information includes the patient-related information of the patient group. The skill level of the surgeon, the surgical techniques used, the complications encountered, or the nature of the surgeon, procedure, and use An indication of the equipment used or any other information about the facility may also be included.

[0186] Case-specific information can be entered in a variety of ways. In some embodiments, the surgeon Case-specific information can be entered through a graphical user interface. The interface consists of one or more text fields for entering information, prompts, and may include a drop-down list, checkbox, or other field or mechanism. In some embodiments, the graphical user interface may include the disclosed It may be associated with a computing device or processor that executes the method. In an embodiment, the graphical user interface is laptops, desktop computers, computer terminals, wearable devices (smartphones) This includes smartwatches, smart glasses, smart jewelry, head-mounted displays, etc. ), or any other electronic device capable of receiving user input. In some embodiments, case-specific information can be associated with a calling device. may occur at an earlier time point or over a period of time (e.g., days, months, years, or longer). Some or all of the case-specific information may be entered in a hospital or other healthcare facility database. a database, electronic medical record, or any other device capable of storing patient data and / or other medical data. In some embodiments, case characteristics corresponding to the intended surgical procedure may be extracted from the location. The relevant information may be received from an external device. For example, the case-specific information may be received from an external computer. computing devices, servers, cloud computing services, network devices or any other device external to the system performing the disclosed method. Alternatively, case-specific information corresponding to the proposed surgical procedure may be received. At least a portion of the data is collected from an electronic medical record (EMR) or from a system that handles EMRs (e.g., the EMR of the specific patient for whom the procedure is to be performed, the EMR related to the intended surgical procedure, etc.) The scheduling system then responds to the medical professionals associated with the proposed surgical procedure. The information may be received from an electronic record that contains the information, or from a system that handles electronic records.

[0187] In some exemplary embodiments, the case-specific information includes patient information related to the intended treatment. For example, as previously mentioned, case-specific information may include characteristics of the intended patient. The patient characteristics may include the patient's gender, age, weight, height, physical strength, heart rate, and blood pressure. , body temperature, medical condition or disease, medical history, previous treatment, or any other relevant characteristics. Other exemplary patient characteristics include, but are not limited to, those listed throughout this disclosure. In some embodiments, patient characteristics may be entered directly by the surgeon. For example, as described above, patient characteristics may be displayed via a graphical user interface. In other embodiments, the patient characteristics can be entered in a database or other electronic record. In some embodiments, the patient characteristics may be retrieved from a patient's medical record. For example, the patient characteristics may be received from an identifier entered by a surgeon or other This information can be retrieved from medical records or other sources. The patient may enter a patient identifier and the patient's medical record and / or patient characteristics may be As described herein, the patient identifier may be used to search for the patient. It may be a name (e.g., an alphanumeric code or machine-readable code) or in an identifiable way. The patient may be identified (e.g., patient name or social security number). In some instances, case-specific This information may be used to identify characteristics of two or more patients that are relevant to the proposed procedure (e.g., This may include any contemplated surgical procedures, including surgical procedures involving the removal of tissue from the body, such as implantation.

[0188] According to the present disclosure, the case-specific information may include information regarding a surgical instrument. It can be any device or instrument used as part of a surgical procedure. Some typical surgical procedures Instruments are described throughout this disclosure. In some embodiments, the present disclosure relates to surgical instruments. The information may include at least one of an appliance type or an appliance model. A type can refer to any classification of an instrument. For example, an instrument type can refer to the type of instrument being used. Type of instrument (e.g., scalpel, scissors, forceps, retractor, or other type) The type of device can refer to whether the device is electronic or not, Whether the instrument is to be used in invasive surgery, the material from which the instrument is constructed, the size of the instrument, or any other A device model can include various other classifications, such as identifying characteristics. and / or manufacturer (e.g., "15921 Halsted Mosquito Forceps"). This can be done.

[0189] Embodiments of the present disclosure provide for the collection of case-specific information in data associated with multiple sets of surgical video footage. Identify groups of intraoperative events that are more likely to be encountered during a planned surgical procedure compared with The data associated with the set of multiple surgical videos may further include: The data may include any stored information related to the surgical video footage. Identifying intraoperative surgical events, surgical stages, or surgical event characteristics depicted in or associated with video footage The data may include information about patient or surgeon characteristics, video characteristics (e.g., date of recording, file size, information about the imaging device, location of recording, etc.), or surgical video footage The data may include other information such as tags or any other information about the video. In other embodiments, the data may be stored separately as other data in the file. In some embodiments, the surgical video footage may be stored in a file. The data may then be indexed to associate it with the video footage. The case-specific data structure 600 may be stored in a data structure such as the data structure 600 described above. The information is then combined with data associated with one or more surgical video footage (e.g., multiple sets of surgical procedures). Comparison of the images with surgical video footage can be performed using, for example, one or more similarity functions to obtain case-specific one or more similarity measures between the information and data associated with one or more surgical video footages Further, in one example, the k most similar surgical video images may be calculated. to predict commonly occurring events likely to be encountered during a planned procedure. ,e.g., using a k-nearest neighbor algorithm, events occurring in one or more surgical videos can be,detected. A calculated risk assessment was performed to determine whether the risk is likely to occur during the proposed surgical procedure. The similarity metric can be compared to a selected threshold. In some examples, the machine learning model ,Information related to a specific surgical procedure can be used to identify intraoperative symptoms likely to be encountered during a specific surgical procedure. It may be trained using training examples to identify events, and The trained machine learning model analyzes case-specific information of the proposed surgical procedure and Used to identify groups of intraoperative events likely to be encountered during the illustrated surgical procedure An example of such a training example would be to examine the potential risks that may be encountered during a particular surgical procedure. It may contain information about specific surgical procedures, along with labels indicating potentially important intraoperative events. do.

[0190] The group of intraoperative events likely to be encountered during a planned surgical procedure is based on data. For example, case-specific information can be associated with a set of multiple surgical videos. This may be compared with data collected from the patient, based on the characteristics of the intended surgical procedure (case-specific information) associated with surgical procedures with the same or similar characteristics. For example, case-specific information may include identifying the intended procedure in the surgical video footage. related to a surgical procedure on a patient with the same or similar medical condition, Another example is a set of surgical video footage that can be used to identify a patient with diabetes, high cholesterol, or other conditions. A 73-year-old man with steroids, hypertension, and a family history of heart disease underwent a catheterization study. Surgeons preparing to perform the procedure should consider patients who share similar characteristics (or who are similar to a particular patient). case-specific to depict video footage for review of the patient (anticipated to present) Information can be entered about intraoperative events likely to be encountered during the intended surgical procedure. The group reported on intraoperative surgical incidents encountered during surgical procedures related to the identified surgical video footage. In some embodiments, surgical video footage and / or possible encounters may be included. Several factors can be considered when identifying groups of intraoperative events that are more likely to occur.

[0191] Whether an intraoperative event is considered likely to be encountered during a planned surgical procedure The frequency with which intraoperative events occur in similar surgical procedures may depend on the intended surgical procedure. For example, intraoperative events can be measured by the number of times they occurred during similar procedures, the percentage of times they occurred during similar procedures, or other statistical information based on multiple surgical video footage sets. In this embodiment, intraoperative events may be identified based on comparing the likelihood to a threshold. For example, an intraoperative event is recognized if it occurs in more than 50% of similar surgical procedures, or any other proportion. In some embodiments, groups of intraoperative events may be classified into groups based on the likelihood of occurrence. For example, a group may be created for intraoperative events with a high probability of occurring. The hierarchy may include one or more hierarchies of unlikely intraoperative events.

[0192] According to some embodiments of the present disclosure, machine learning or other artificial intelligence techniques are used to Groups of events can be identified, and case-specific information can then be extracted from surgical video footage. Comparing multiple sets of data with associated data is an artificial neural network to identify groups of intraoperative events likely to be encountered during a planned surgical procedure. In one example, the artificial neural network may be manually configured. It may also be generated from a combination of two or more other artificial neural networks. In one example, an artificial neural network may be confronted with a variety of case-specific information. In some embodiments, the system may be provided with training data that correlates with intraoperative events with high likelihood of progression. The training data is associated with surgical videos and surgical footage contained in the repository. The training data may include one or more sets of data collected from patients. This may include non-image related data such as the characteristics of the patient's eyes or past medical history. Use the network to generate a trained model based on the training data. Therefore, using an artificial neural network is a This may include providing case-specific information to the artificial neural network. The output of the model is a group of intraoperative events likely to be encountered during the intended surgical procedure. It can identify logistic regression, linear regression, regression, random forest, k-nearest neighbor (KNN) models (e.g., the models mentioned above), k-means models, decision trees, Cox proportional hazards regression model, Naive Bayes model, Support Vector Machine (S VM) model, gradient boosting algorithm, or any other form of machine learning model Alternatively, various other machine learning algorithms may be used, including the algorithm.

[0193] Some aspects of the present disclosure provide case-specific information and identification of intraoperative events likely to be encountered. The identified groups were used to replicate the surgical video footage corresponding to the identified groups of intraoperative events. It may further include identifying a particular frame within a particular set of sets of numbers. A specific frame within a particular set of multiple surgical video footage is the video in which an intraoperative event occurs. For example, if a group of intraoperative events includes a complication, the specific The frames may include video footage depicting or associated with the complication. In some embodiments, a particular frame may be used to provide context for an intraoperative event, for example. To achieve this, some surgical video footage may be included before or after the intraoperative event occurred. Also, the specific frames do not necessarily have to be consecutive. For example, if an intraoperative event is an adverse event, If the adverse event or outcome is a specific frame, the frame corresponding to the adverse outcome and the cause of the adverse outcome is A frame may contain multiple frames, and these frames do not have to be consecutive. , frame number (e.g., frame number, start frame number and end frame number, start Based on the frame number and the number of subsequent frames, time information (e.g., start time and stop times, durations, etc.) or to identify specific frames of video footage The identification may be based on any other aspect of the above.

[0194] In some embodiments, a particular frame may be indexed across multiple surgical video feeds. For example, as described above, the video image may be identified based on the image position. may be indexed to associate with tags, event tags, and / or event characteristics Therefore, identifying a specific frame within a particular set of multiple surgical video footage The intraoperative event may be performed using a data structure such as data structure 600 as described in connection with FIG. This may include performing a lookup or search for the elephant.

[0195] According to the present disclosure, the particular frame identified may be used in multiple scans performed on different patients. It may include frames from a surgical procedure. Therefore, the specific frame identified The data are collected from intraoperative events from surgical procedures performed on different patients that can be used for surgical preparation. A compilation of related footage can be created, e.g. examples of the best video clips (e.g., with regard to video quality, clarity, representativeness, compatibility with the intended surgical procedure) may vary. The procedures may be selected from different procedures performed on different patients and may be related to each other, This means, for example, that the preparatory surgeon would not have to review each video separately. You can watch the best of a group of video clips without having to click a button.

[0196] Embodiments of the present disclosure may, for example, reduce the resulting compilation to avoid redundancy. To shorten the time, to remove parts that are less relevant or less informative, This may further include omitting portions of certain frames that have been selected. In some embodiments, the first and second sets of video footage from different patients are shared. This can include determining to include frames related to intraoperative events that share a characteristic. The first and second sets of video footage correspond to identified groups of intraoperative events. The common characteristics include the first and second frames identified as frames of interest. determining whether frames from both the first set and the second set should be included; The common characteristic may be any characteristic of the intraoperative events related to the first set and the second set. For example, intraoperative events can be used to determine whether a surgical procedure is redundant. It may be a complication that occurs during treatment, and the common characteristic may be a type of complication. If the complications in the first set of frames and the second set of frames are of the same type, In this case, the surgeon preparing for surgery will It may not be efficient or useful to view both the Only one set can be selected for this purpose, and the other sets are omitted. In embodiments, the common characteristics may include characteristics of the different patients. For example, the common characteristics may include: This may include the patient's age, weight, height, or other demographics, and may include the patient's condition, etc. Various other patient characteristics described throughout this disclosure may also be shared. In other embodiments, the common characteristics may include intraoperative surgical event characteristics of the intended surgical procedure. Intraoperative surgical event characteristics may include any trait or characteristic of an intraoperative event. For example: Adverse outcomes of surgical events, surgical techniques, surgeon skill levels, specific surgeon identities, The physiological response, duration of the event, or any other characteristic or property associated with the event.

[0197] According to various exemplary embodiments of the present disclosure, a first set of video footage from a different patient is and a second set contains frames related to intraoperative events that share a common characteristic. This may involve identifying common characteristics using an implementation of a machine learning model. In one example, a system can be trained to identify frames of video footage that have particular characteristics. You can train a machine learning model using examples, and The machine learning model was used to analyze a first set of video footage from a different patient and a second set of video footage from a different patient. The frames can be analyzed to identify frames related to intraoperative events that share common characteristics. An example of such a training example is a specific feature of a specific frame of a video clip. It can contain video clips with labels indicating their gender. Various machine learning models have been developed as described above. Such machine learning models include logistic regression models and linear regression models. Models, regression models, random forest models, k-nearest neighbor (KNN) models, k-means Models, decision trees, Cox proportional hazards regression models, naive Bayes models, support base models Support Vector Machine (SVM) models, gradient boosting algorithms, deep learning models, may include any other form of machine learning model or algorithm. Some embodiments of the present invention determine whether two sets of video footage share a common characteristic. Further, the present invention uses video examples to train machine learning models to determine In this case, implementing the machine learning model may include: In one example, example video footage is classified into two groups that share common characteristics. training footage, which may include a set of pairs of video footage known to The trained machine learning model can then be used to identify common features between the two sets of video footage. The device may be configured to determine whether to share a gender.

[0198] The disclosed embodiment omits the inclusion of the second set from the compilation presented to the surgeon. and including the first set in a compilation presented to the surgeon. As used herein, compilation can refer to continuous playback and / or continuous playback. In some embodiments, the compilation may include a series of frames that may be presented for In other embodiments, the compilation may be performed in a separate video file, e.g. For example, we present a sequence of frames from each surgical video stored in a repository. The compilation may be stored as instructions for other programs from the identified particular frame. The first set may include additional frames other than those included in the first set, including frames can.

[0199] Some aspects of the present disclosure involve the collection of frames from different surgical procedures performed on different patients. and making the presentation including the compilation including the program available for viewing by the surgeon. The presentation may be any form of visual display, including a compilation of frames. In some embodiments, the presentation may be a compilation video. In addition to compiling, you can also create menus, controls, indexes, timelines, or other components. In some embodiments, the surgeon may view the presentation and other elements such as content. This allows for the display of information on a screen (e.g., , OLED, QLED LCD, Plasma, CRT, DLPT, e-Paper, or similar display technology), optical projectors (e.g., movie projectors, slide projectors, projector), 3D display, smart glasses, or any other visual presentation mechanism outputting the data for displaying the presentation using a display device. In other embodiments, allowing the surgeon to view the presentation may involve one or more other storing the presentation in a location accessible by the computing device of Such storage locations include local storage (such as flash memory) hard drive), network location (server or database), cloud computing a computing platform, or any other accessible storage location. Therefore, the presentation can be accessed from and displayed on an external device. In some embodiments, outputting the video may include outputting the video to an external device. For example, allowing a surgeon to view the presentation. This means that the content can be transmitted to the user's device or other devices via a network for playback on an external device. The present invention can include transmitting the presentation to an external device.

[0200] The presentation involves stitching different clips from different procedures together and then stitching them together. The clips can be presented to the surgeon in the chronological order that they might occur during the procedure. The clip may be presented to the surgeon as a clip, or the surgeon may actively take action to play subsequent clips. The surgeon may be required to see multiple alternative techniques or In some instances where it may be beneficial to look at different responses to adverse events, different surgical procedures may be used. Multiple separate clips from different locations may be presented in sequence.

[0201] Some embodiments of the present disclosure may include selecting one or more of the identified specific frames along the presentation. Allows for the display of a common surgical timeline with one or more corresponding chronological markers on top For example, as described above, a common surgical timeline may be provided. An exemplary surgical timeline 420 including chronological markers may be overlaid on the display. As shown in FIG. 4, the time series markers correspond to markers 432, 434, and / or 436. Therefore, time series markers can be used to track surgical stages, intraoperative surgical events, and decision-making processes. correspond to a particular transition or other significant occurrence of the identified particular frame along the presentation. The markers can be displayed as shapes along the timeline, as explained in more detail above. , icons, or other graphical representations. It may be presented with frames relating to procedures performed on a single patient, or multiple patients. The information may be presented along with a compilation of video clips from surgeries performed on the patient.

[0202] According to some embodiments of the present disclosure, allowing the surgeon to view the presentation sequentially captures separate sets of video footage of different surgical procedures performed on different patients. Each separate set of video footage may be displayed to a different patient. In some embodiments, the device may accommodate different surgical procedures performed on a patient. Displaying separate sets of video images sequentially provides a surgeon or other user with a continuous video In other embodiments, playback may be stopped or paused between separate sets of video footage. The surgeon or other user can stop the next set of video footage in the sequence. You can start the event manually.

[0203] According to some embodiments of the present disclosure, the presentation includes identifying intraoperative events that are likely to be encountered. Multiple surgical video clips corresponding to identified groups of intraoperative events and / or intraoperative events Simulated surgery based on identified specific frames within a specific set of sets It may include representations of machine learning algorithms (generative adversarial networks, etc.). training examples to develop a machine learning model (e.g., artificial neural network) training neural networks, deep learning models, convolutional neural networks, etc. based on groups of intraoperative events and / or frames of surgical video footage. It is possible to generate simulations of surgical procedures and also to use trained mechanics The learning model is used to identify groups and / or characteristics of intraoperative events likely to be encountered. Identified intraoperative events within a specific set of multiple surgical video footage corresponding to a group of identified intraoperative events. The specific frames captured can be analyzed to generate a simulated surgical procedure.

[0204] In some embodiments, sequentially displaying the separate sets of video footage includes sequentially displaying the separate sets of video footage. A video system that allows a surgeon or other user to select one or more of the distinct sets of images. The method may include displaying an index of a separate set of video images. For example, the link may include intraoperative events, surgical stages, or other indicators for different, separate sets of video footage. In other embodiments, the index may be a text-based index that lists The index can be a graphical display such as the timeline mentioned above, or a combination of graphical information and text. For example, the index may be a combination of the distinct sets. The timeline can include parsing into surgical stages and textual stage indicators. In such an embodiment, the separate sets may correspond to different operative stages of a surgical procedure. The distinct sets may be different colors, different shading, bounding boxes or separators, or distinct Other visual indicators for distinguishing the sets may be used. The indicators may describe or identify the corresponding surgical stage. To identify separate sets, In some embodiments, the timeline may be displayed in list format. The text stage indicator may be included in the list and may be displayed as a formula.

[0205] According to the present disclosure, the timeline includes intraoperative surgical event markers corresponding to intraoperative surgical events. Intraoperative surgical event markers can be used to record intraoperative procedures associated with locations within the surgical video footage. The surgeon can respond to surgical events by clicking on the intraoperative surgical event marker. The system may be enabled to display at least one frame depicting an intraoperative surgical event. For example, clicking on an intraoperative surgical event will change the display of the compilation video to the selected marker. In some embodiments, the surgeon can skip to the associated location. As explained in more detail below, information summarizing aspects of a procedure or derived from previous surgical procedures may be used. The information provided by the marker may include additional information about the event or occurrence associated with the marker. The features mentioned above with respect to the timeline overlay on the surgical video or Any of the features may also be applied to the compilation video described herein.

[0206] Embodiments of the present disclosure provide a method for analyzing intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical outcomes. Train machine learning models to generate repository indexes based on event characteristics. and generating an index of the repository. Comparing case-specific information with data associated with multiple sets is an indexing technique. This may include searching for logistic regression models, linear regression models, and regression models. , Random Forest model, K-nearest neighbor (KNN) model, K-means model, decision tree, proportional hazards regression model, naive Bayes model, support vector machine (SV M) model, gradient boosting algorithm, deep learning model, or any other form of Use any of the machine learning models or algorithms described above. Known intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical events We train the model using a training dataset of surgical video footage with morphological features. The trained model can be trained on the Based on additional surgical video footage, intraoperative events, surgical outcomes, patient characteristics, and surgeon characteristics were analyzed. The repository can be configured to determine the characteristics of the surgical procedure and the characteristics of the intraoperative surgical events. When applied to video footage, the video footage may be tagged based on the identified characteristics. For example, video footage may be indexed by video location, as described above with respect to video indexing. , stage tags, event locations, and / or event tags. The repository may be stored as a data structure such as data structure 600 described above.

[0207] FIG. 12 illustrates an exemplary process 120 for surgical preparation consistent with disclosed embodiments. 12. The process 1200 is a flowchart illustrating one of the methods described herein. The present invention may be implemented by a processing device such as a plurality of collocated or distributed processors. In some embodiments, the non-transitory computer-readable medium may be executed by a processor. The process may include instructions that, when executed, cause a processor to perform the process 1200. Process 1200 is not necessarily limited to the steps shown in diagram 1200, but may be used throughout this disclosure. Any step or process of the various embodiments described below may be included in process 1200. In step 1210, the process 1200 may be performed for a different patient. Access a repository of multiple surgical video footage sets reflecting multiple surgical procedures The plurality of sets of surgical video footage may include intraoperative surgical events, surgical outcomes, These may include patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics. The repository is indexed using process 800, for example, and multiple surgical procedures are performed. Searching and identifying sets of video footage can be facilitated.

[0208] In step 1220, the process 1200 begins by preparing for the intended surgical procedure. Allows the attending surgeon to enter case-specific information corresponding to the intended surgical procedure As mentioned above, the intended surgical procedure may include a planned procedure, a virtual This may be a treatment, an experimental treatment, or another treatment that has not yet been performed. For example, it may be entered manually by a surgeon via a user interface. In some embodiments, some or all of the case-specific information may be received from the patient's medical record. The useful information may include patient characteristics relevant to the intended procedure, and the information may include surgical instruments. information about the device (e.g., device type, device model, device manufacturer, etc.) or related procedures It may include any other information that may be used to identify the surgical video footage.

[0209] In step 1230, the process 1200 extracts case-specific information from the surgical video footage. Comparing data associated with multiple sets allows for a better understanding of the risks encountered during a proposed surgical procedure. This may involve identifying groups of intraoperative events that are likely to be encountered. Groups of intraoperative events may include, for example, historical video footage, historical data other than video data, and machine-generated data, or any other form of data from which predictions can be derived The determination may be based on learning analysis. 0 uses case-specific information and identified groups of intraoperative events likely to be encountered. The specific sections of multiple sets of surgical video footage corresponding to identified groups of intraoperative events were then analyzed. The identified specific frame may include identifying a specific frame within the frame. As mentioned previously, frames from multiple surgical procedures performed on different patients are collected. It can include.

[0210] In step 1250, the process 1200 performs a step of collecting the data from different patients, as previously described. The first and second sets of video footage are associated with intraoperative events that share common characteristics. In step 1260, determining that the frame includes a frame that has been The process 1200, as previously described, generates a second set of images from the compilation that are presented to the surgeon. and including the first set in the compilation presented to the surgeon. may include:

[0211] In step 1270, the process 1200 determines whether different tests are performed on different patients. A presentation containing a compilation of frames from a surgical procedure for the surgeon to view. As mentioned above, the presentation may be made available for viewing by the surgeon. This allows the presentation to be displayed on a screen or other display device. outputting the data and recording the presentation in a location accessible to another computing device. storing, transmitting, or viewing presentations and / or compilations. The present invention may include any other process or method that can cause the

[0212] When preparing for a surgical procedure, it is common for surgeons to review video footage of previous surgical procedures. However, in some cases, only a particularly complicated part of the surgical procedure Using conventional techniques, surgeons may assign surgical procedures based on the complexity of the procedure. Identifying parts of a video can be difficult and time-consuming. Efficiently and effectively analyzes the complexity of surgical videos, allowing surgeons to quickly find the relevant parts of the surgical video There is a need for non-traditional methods to enable this review.

[0213] Aspects of the present disclosure include surgical procedures, including methods, systems, devices, and computer-readable media. In particular, when preparing for a surgical procedure, the surgeon must consider the specific complications that may arise. You may want to see portions of a surgical video with different levels of detail. For example, a typical routine surgical procedure In a video, a highly skilled surgeon may only want to see a single, highly complex event. However, finding the right video and the right position within the video can be challenging for the surgeon. Therefore, in some embodiments, the complexity of the surgical images can be A method and system for analyzing surgical videos based on complexity is provided. The clip viewing process automatically tags portions of the surgical video with a complexity score. This accelerates the process, allowing the surgeon to quickly select frames of interest based on complexity. It will be possible to find it.

[0214] For ease of description, aspects of the method may be described as systems, devices, and computer-readable media. The methods are described below with the understanding that they apply equally to the body. For example, such methods Some aspects of the present invention may be performed electronically over wired, wireless, or both networks. Other aspects of such methods may be performed using non-electronic means. In this regard, the method is not limited to any particular physical and / or electronic means, but rather may be implemented using many This can be achieved using different means.

[0215] Consistent with disclosed embodiments, a method may analyze frames of surgical footage to determine the Identifying the anatomical structures in the first set. ,Surgical footage includes any video, group of video frames, or display of a surgical procedure. For example, surgical footage can refer to one or more videos captured during a surgical procedure. The first set of frames may include one or more video frames included in the surgical video. In some embodiments, the first of the frames may be a group of more than one frame. The set may be consecutive frames, but this is not necessarily the case. For example, the first set of frames may include multiple groups of consecutive frames. .

[0216] As mentioned above, an anatomical structure may be, for example, an organ, tissue, vessel, artery, cell, or other anatomical structure. The first set of frames may be any particular part of a living body, including a biological part. For example, as described above, the image can be analyzed to identify anatomical structures using various techniques. In some embodiments, as described above, frames of surgical video may be subjected to object detection. For example, object detection algorithms can be used to analyze the appearance, image characteristics, The detected object may be based on one or more of the following: a feature, a template, etc. In some embodiments, identifying the anatomical structure in the first set of frames may include, for example, As mentioned above, we used a machine learning model trained to detect anatomical structures. For example, images and / or videos may be used to Machine learning was used as training data, along with the identification of anatomical structures known to The resulting trained model is The method may be used to analyze surgical images to identify anatomical structures in a set of For example, artificial neural networks configured to identify anatomical structures in images and / or videos. The surgical video is analyzed using a neural network to identify anatomical features within the first set of frames. Able to identify structures. Logistic regression, linear regression, regression, random forest models, k-nearest neighbor (KNN) models, k-means models, decision trees, Cox proportional hazards regression models models, Naive Bayes models, Support Vector Machine (SVM) models, Gradient Boosting models machine learning model, deep learning model, or any other form of machine learning model or algorithm A variety of other machine learning algorithms may be used, including the .

[0217] Some aspects of the present disclosure may further include accessing the first historical data. The first historical data is a first frame captured from a first group of previous surgical procedures. Generally, the frame data is based on an analysis of the surgical procedure described herein. The first historical data and / or the second historical data may include any image or video data that captures the first historical data and / or the second historical data. A frame of data can be stored in one or more memory locations. accessing the historical data includes reading the historical data from a storage location; In other embodiments, accessing the first historical data can be performed, for example, at the time of image capture. and receiving first historical data and / or first frame from the capturing device or computing device. Consistent with embodiments of the present disclosure, the first history data may include receiving the history data. Accessing the historical data includes retrieving or receiving a first frame of data; and The method can include analyzing the first frame data to identify first historical data.

[0218] Historical data may be any information relating to a previous surgical procedure. Some non-limiting examples of data are described above. In some embodiments, the first history The data can include complexity information associated with a first group of previous surgical procedures. Complexity information may include any data indicative of the complexity level of a procedure, as further described below. The first historical data may be collected from a first frame of data. The first group of surgical procedures may include any other information related to the first group of surgical procedures. One frame of data includes the anatomical structures involved, the medical instruments used, and the surgical procedure performed. type of procedure, intraoperative events (including adverse events) occurring during the procedure, and medical conditions presented by the patient. the patient's condition, the characteristics of the surgeon, the expertise of the surgeon or other health care professionals involved ,timing information (e.g., duration of interaction between medical instruments and anatomical structures, surgical Duration of the stage or intraoperative event, appearance of the medical device and initial contact between the medical device and the anatomical structure time between interactions, or other relevant duration or timing information), anatomical structures the condition of the patient, the number of surgeons or other health care professionals involved, or any other factors related to previous surgical procedures may include or show information related to previous surgical procedures, including any other information related to .

[0219] In embodiments where the first historical data includes complexity information, such information may be used to may indicate or be associated with the complexity of the part. For example, the first historical data may include the display of statistical relationships between specific anatomical structures and specific levels of surgical complexity. A statistical relationship can be established between a particular level of surgical complexity and a particular anatomical structure. For example, the information may be any information that can show the correlation between a particular blood vessel and a surgical site. When exposed during a procedure, when a particular part of an organ is torn, or when a particular amount of blood is detected When issued, such events may be statistically correlated with the level of surgical complexity. The amount of fat or the detection of organ malconditioning may also be correlated to the complexity level. These are just a few examples. Any condition or event that correlates with surgical complexity may serve as an indicator of surgical complexity. obtain.

[0220] In some embodiments, the first historical data is used to determine whether an object detection algorithm and / or a motion detection algorithm is being used. and detecting the first frame using one or more image or video analysis algorithms, including a detection algorithm. In other embodiments, the history can be identified based on the frame data. First frame is identified using a machine learning model trained to identify historical data. For example, the machine learning model may identify the first historical data from the Use the training examples to distinguish historical data (as described above) from frame data. The trained machine learning model may be trained using the first historical data. may be used to analyze the first frame data to determine An example of a training example is a label indicating the complexity level of a surgical procedure or part of a surgical procedure. along with images and / or video showing the surgical procedure or a portion of the surgical procedure. For example, such labels may be generated manually or by different processes. It may be generated, read from memory, etc.

[0221] An embodiment of the present disclosure uses the first historical data and uses the identified anatomical structure to and analyzing the first set of frames using a first As used herein, complexity can involve determining a surgical complexity level. The severity level is a value or other classifier that indicates the relative complexity of a surgical procedure or part of a surgical procedure. For example, complexity may be based on the difficulty of a surgical procedure relative to other surgical procedures. The level of difficulty required to perform one or more of the techniques involved in a surgical procedure Surgeon skill level, adverse events (e.g., tears, bleeding, injury, or other adverse events), It may be based on the likelihood of occurrence, the success rate of the surgical procedure, or any other indicator of the difficulty of the procedure. Surgical procedures with higher relative difficulty levels are associated with higher complexity levels. It can be done.

[0222] As another illustrative example, the complexity level may be the duration required to complete a surgical procedure or portion thereof. It can be based on duration or time requirements, for example, procedures that require longer execution times. Or a technique can be considered more complex and is associated with a higher complexity level. As another example, the complexity level may be the number of steps required to perform a surgical procedure or portion thereof. It may also be based on the number of steps, e.g., a procedure or technique requiring more steps. can be considered more complex and can be associated with a higher complexity level. In some embodiments, the complexity level is based on the type of surgical procedure or procedure being performed. A particular technique or procedure may have a predetermined complexity level. can be based on the complexity of the technique or procedure involved. For example, a cholecystectomy is a total It can be considered more complex than a cholecystectomy and therefore requires surgical procedures including cholecystectomy. Other factors that may be related to the complexity level include: The factors considered are disease severity, complicating factors, anatomical structures involved, type of medical device used, The type of surgical procedure performed, intraoperative events (including adverse events) occurring during the procedure, and patient outcomes The physical response, the medical condition presented by the patient, the patient's characteristics, the surgeon's...

Claims

1. 1. A computer-implemented method for reviewing surgical videos, comprising: accessing at least one video of the surgical procedure; outputting said at least one video for display; Overlaying the at least one video output to display a surgical timeline. laying out the surgical timeline, the surgical timeline including surgical stages, intraoperative surgical events, and a marker identifying at least one of the decision junctions; A surgeon views a playback of the at least one video while viewing one of the surgical timelines. and allowing the user to select one or more of the markers, thereby shifting the display of the video to the selected marker. skipping to a position associated with the marker; 23. A computer-implemented method comprising:

2. The markers are coded by at least one of a color or a critical level. The method according to claim 1.

3. The surgical timeline includes text information identifying parts of the surgical procedure. Item 1. The method according to item 1.

4. The at least one video is from a plurality of surgical procedures arranged in a procedural chronological order. a compilation of images, the compilation of images comprising a complexity summary from the plurality of surgical procedures; the one or more markers are associated with the plurality of surgical procedures to represent a common tag. The method of claim 1 , wherein the method is displayed on a timeline.

5. The one or more markers may be decision junctions corresponding to decision junctions of the surgical procedure. and the selection of the decision junction marker comprises: The surgeon views two or more separate video clips from two or more corresponding other surgical procedures. wherein the two or more video clips depict different activities.

2. The method according to claim 1.

6. The one or more markers may be decision junctions corresponding to decision junctions of the surgical procedure. and the selection of the decision junction marker comprises: a table of one or more alternative possible decisions associated with the selected decision junction marker; The method of claim 1 , further comprising:

7. One or more predicted outcomes associated with the one or more alternative possible decisions are 7. The method of claim 6, wherein the display is displayed in conjunction with the display of one or more other possible decisions. 。

8. The one or more predicted outcomes are each a set of events involving similar decision junctions.

8. The method of claim 7, wherein the method is the result of analysis of multiple videos of previous surgical procedures.

9. The distribution of past decisions made at each similar past decision junction is 7. The method of claim 6, wherein related information is displayed in conjunction with the display of the alternative possible decisions. Method of posting.

10. The decision junction of the surgical procedure is associated with a first patient, Each similar past decision junction is a set of patients with similar characteristics to the first patient.

10. The method of claim 9, wherein the surgical procedure is selected from a past surgical procedure associated with the patient.

11. The decision junction of the surgical procedure is associated with a first medical professional, Each of the similar past decision junctions has the same characteristics as the first medical expert.

10. The method of claim 9, wherein the surgical procedure is selected from a past surgical procedure associated with a medical professional having the same medical condition. Method of posting.

12. The decision junction of the surgical procedure is a first prior event in the surgical procedure. and the similar past decision junction is associated with the first prior event 10. The method of claim 9, wherein the prior event is selected from a previous surgical procedure comprising a similar prior event.

13. The markers include intraoperative surgical event markers, and the selection of the intraoperative surgical event markers is performed by the surgeon. Allowing different video clips from different surgical procedures to be viewed, Oclip presents different ways in which selected intraoperative surgical events have been handled. The method described.

14. The overlay on the video output may include a plurality of pixels arranged to overlap the external video signal. The method of claim 1 , wherein the indication is displayed prior to completion of a therapeutic procedure.

15. The analysis may include analyzing one or more electronic medical records associated with the plurality of videos of past surgical procedures. The method of claim 8 based on medical records.

16. Each of the similar decision junctions is compared to the surgical junctions according to a similarity metric. The method of claim 8, wherein the decision junction is similar to a disposition.

17. 10. The method of claim 8, wherein the analysis includes using an implementation of a computer vision algorithm. How to.

18. The markers relate to intraoperative surgical events, and the selection of intraoperative surgical event markers is performed by the surgeon 2. The method of claim 1, which allows the user to view different video clips from different surgical procedures. The method described.

19. 1. A system for reviewing surgical videos, the system comprising: Accessing at least one video of a surgical procedure; outputting said at least one video for display; Overlaying the at least one video output to display a surgical timeline. The surgical timeline includes surgical stages, intraoperative surgical events, and decision junctions. a marker identifying at least one of the A surgeon views a playback of the at least one video while viewing one of the surgical timelines. and allowing the user to select one or more of the markers, thereby shifting the display of the video to the selected marker. skip to the position associated with the 23. A system comprising: at least one processor configured to:

20. When executed by at least one processor, enables review of surgical video instructions for causing said at least one processor to perform the operations of: A readable medium, the operation comprising: accessing at least one video of the surgical procedure; outputting said at least one video for display; Overlaying the at least one video output to display a surgical timeline. laying out the surgical timeline, the surgical timeline including surgical stages, intraoperative surgical events, and a marker identifying at least one of the decision junctions; A surgeon views a playback of the at least one video while viewing one of the surgical timelines. and allowing the user to select one or more of the markers, thereby shifting the display of the video to the selected marker. skipping to a position associated with the marker; 13. A non-transitory computer readable medium comprising:

21. 1. A computer-implemented method for video indexing, comprising: Accessing the video footage to be indexed, the video footage to be tagged includes footage of a particular surgical procedure; The video footage is analyzed to generate video sequences associated with operative steps of the particular surgical procedure. identifying an image location; generating a stage tag associated with the surgical stage; associating the stage tag with the video image location; Interpreting the video footage to identify event locations of specific intraoperative surgical events within the surgical phase. and associating an event tag with the event location of the particular intra-operative surgical event; storing event characteristics associated with the particular intra-operative surgical event; Comparing at least a portion of the video footage of the particular surgical procedure with further video footage of other surgical procedures. The stage tag, the event tag, and the event characteristic in a data structure including video footage. and associating said data structure with one or more of said other surgical procedures. Each stage tag, each event tag, and each event feature associated with the above Steps, including sex, Selected stage tags, selected event tags, and selected allowing a user to access the data structure by selection of selected event characteristics. and, The at least one selected stage tag, the selected event tag, and the selected event and performing a lookup within said data structure for surgical video footage that matches the subject characteristics and storing the identifying a matching subset of the selected video footage; displaying a matching subset of the stored video footage to the user; by omitting playback of video footage lacking the selected event characteristics, The user may view the surgical footage of at least one intraoperative surgical event that shares an event characteristic. and 4. A computer-implemented method comprising:

22. omitting playback of portions of the selected surgical event that lack the selected event characteristics. The user views surgical footage of at least one intraoperative surgical event having the selected event characteristics. The step of enabling comprises: detecting selected surgical events that lack the selected event characteristics. The present invention relates to a method for recording a plurality of intraoperative surgical events that share the selected event characteristics while omitting playback of a portion of the selected event characteristics.

22. The method of claim 21, comprising presenting portions of surgical video sequentially to the user.

23. The stored event characteristics include an adverse outcome of the surgical event, and the matched subset is The displaying step may allow the user to view surgical images of the selected adverse outcomes. while omitting replay of surgical events lacking the selected adverse outcome.

22. The method of claim 21 , comprising:

24. the stored event characteristics include a surgical procedure, and a screen for displaying the matching subset is selected. Step 2 allows the user to view surgical footage of a selected surgical procedure. On the other hand, a step of omitting playback of the surgical video not associated with the selected surgical procedure. The method of claim 21 , further comprising:

25. the stored event characteristics include a surgeon skill level, and displaying the matching subset. The step of allowing the user to view a video indicative of the selected surgeon skill level may include: while omitting playback of videos lacking the selected surgeon skill level.

22. The method of claim 21, comprising the step of:

26. the stored event characteristics include physical patient characteristics, and displaying the matching subset. The step of selecting includes allowing the user to view an image showing the selected physical patient characteristic. while omitting playback of videos lacking the selected physical patient characteristics.

22. The method of claim 21 , comprising:

27. The stored event characteristics include an identity of a particular surgeon, and the matching subset is The displaying step may include allowing the user to view a video showing an action by a selected surgeon. While allowing the surgeon to select a desired action, playback of the selected action is omitted.

22. The method of claim 21, comprising the step of:

28. The stored event characteristics include a physiological response, and a screen for displaying the matching subset is provided. Step 2 allows the user to view an image showing the selected physiological response. On the other hand, the method includes a step of omitting playback of the video lacking the selected physiological response.

22. The method according to claim 21.

29. The video footage is analyzed to identify associated information for at least one of the surgical events or the surgical stages. The step of identifying the associated video image location includes identifying the opening of the surgical step for playback. The video is then used to identify at least one of the starting location of a surgical event for regeneration or the beginning of a surgical event for regeneration.

22. The method of claim 21, further comprising the step of performing computer image analysis on the video footage. method.

30. Accessing aggregate data relating to multiple surgical procedures similar to the particular surgical procedure. and presenting to the user statistical information associated with the selected event characteristics. The method of claim 21 , further comprising the steps of:

31. The accessed video footage may include images of areas above the operating table, within the patient's surgical cavity, within the patient's organs, or at least one imaging device positioned at at least one of the locations within the patient's vasculature.

22. The method of claim 21, comprising video footage captured via an image sensor.

32. 22. The method of claim 21, wherein identifying the video image location is based on user input. Law.

33. The step of identifying a location of the video image may include computing a location of the video image using computer analysis.

22. The method of claim 21, comprising analysing image frames.

34. The computer image analysis uses example video frames containing previously identified surgical steps. By using a neural network model trained on 30. The method of claim 29, further comprising identifying at least one of an image location or a stage tag. The method described.

35. determining the stored event characteristics based on user input.

22. The method according to claim 21.

36. The memory is based on a computer analysis of video footage depicting the particular intraoperative surgical event.

22. The method of claim 21, further comprising determining a characteristic of the event.

37. The step of generating the stage tags may include computer generated video footage depicting the surgical stages. The method of claim 21 , based on data analysis.

38. The step of identifying a matching subset of the stored video footage includes computer analysis. using a matching subset of the stored videos and the selected event characteristics.

22. The method of claim 21, comprising determining a degree of similarity between the.

39. Accessing video footage to be indexed; the video footage includes footage of a particular surgical procedure; Analyzing the video footage to generate stage tags; identifying video image locations associated with operative stages of the surgical procedure; Associating the stage tag with the video image location; analyzing the video footage to identify event locations of specific intra-operative surgical events; Associating an event tag with the event location of the particular intra-operative surgical event; storing event characteristics of the particular intraoperative surgical event; Comparing at least a portion of the video footage of the particular surgical procedure with further video footage of other surgical procedures. The stage tag, the event tag, and the event characteristic in a data structure including video footage. and the data structure is associated with at least one other surgical procedure. Each step tag, each event tag, and each event characteristic are also included. Selected stage tags, selected event tags, and selected allowing a user to access said data structure by selection of a selected event characteristic; The at least one selected stage tag, the selected event tag, and the selected event and performing a lookup within said data structure for surgical video footage that matches the subject characteristics and storing the identifying a matching subset of the selected video footage; displaying a matching subset of the stored video footage to the user; by omitting playback of video footage lacking the selected event characteristics, The user is able to view surgical footage of at least one intraoperative surgical event that shares an event characteristic. To make it possible, A surgical video indexing system, comprising at least one processor configured to: Tem.

40. When executed by at least one processor, the method enables video indexing. instructions for causing said at least one processor to perform operations that enable said persistent computing device to a data-readable medium, the operation comprising: Accessing the video footage to be indexed, the video footage to be tagged includes footage of a particular surgical procedure; analyzing the video footage to generate stage tags; identifying video image locations associated with operative stages of the surgical procedure; associating the stage tag with the video image location; analyzing the video footage to identify event locations of specific intra-operative surgical events; 、 associating an event tag with the event location of the particular intra-operative surgical event; storing event characteristics of the particular intra-operative surgical event; Comparing at least a portion of the video footage of the particular surgical procedure with further video footage of other surgical procedures. The stage tag, the event tag, and the event characteristic in a data structure including video footage. wherein the data structure is associated with one or more other surgical procedures. Each step tag, each event tag, and each event characteristic that is included in the Steps and Selected stage tags, selected event tags, and selected allowing a user to access the data structure by selection of selected event characteristics. and, The at least one selected stage tag, the selected event tag, and the selected event and performing a lookup within said data structure for surgical video footage that matches the subject characteristics and storing the identifying a matching subset of the selected video footage; displaying the matching subset of the stored video footage to the user; Thereby, the selected event characteristics are reproduced while omitting the reproduction of video footage that lacks the selected event characteristics. The user may view surgical footage of at least one intraoperative surgical event that shares the event characteristics. and 13. A non-transitory computer readable medium comprising:

41. 1. A computer-implemented method for generating a surgical overview video, comprising: a first group of frames associated with at least one intraoperative surgical event and a surgical action; and a second group of frames that are not associated with the first group of frames. Tep and accessing historical data based on historical surgical footage of previous surgical procedures; wherein the historical data is a frame that associates a portion of the surgical video with an intraoperative surgical event. including information distinguishing between frames associated with surgical actions and frames not associated with surgical actions; Based on the information of the history data, in the particular surgical image, the first distinguishing a group from said second group of frames; Based on a user request, omitting presentation of said second group of frames to the user. and presenting to the user an aggregation of the first group of frames of the particular surgical video. Steps and 4. A computer-implemented method comprising:

42. classifying a portion of the historical surgical footage into frames associated with intraoperative surgical events; 42. The method of claim 41, wherein the information includes an indication of at least one of the presence or movement of a surgical instrument. Method of posting.

43. The information distinguishing a portion of the historical surgical footage into frames associated with intraoperative surgical events.

42. The method of claim 41, further comprising: Law.

44. The request of the user includes an indication of at least one type of intraoperative surgical event of interest. , the first group of frames includes at least one type of subject intraoperative surgical event.

42. The method of claim 41, wherein the method depicts at least one intraoperative surgical event of:

45. The request of the user includes a request to view a number of intraoperative surgical events within the particular surgical video. and presenting an aggregation of the first group of frames to the user includes: omitting the time-sequential frames of the first group of frames and displaying the first group of frames in chronological order 42. The method of claim 41, comprising the step of:

46. The historical data further includes historical surgical outcome data and respective historical cause data. Including, The first group of frames includes a set of causes of frames and a set of outcomes of frames. Including, the second group of frames includes an intermediate set of frames; The method comprises: Analyzing the particular surgical image to identify surgical outcomes and respective causes of the surgical outcomes. a step of: identifying said historical outcome data and each of said historical outcome data; Steps based on the actual cause data, Based on the analysis, a sequence for detecting the outcome set for frames within the particular surgical video is provided. wherein the outcome set of frames is within an outcome stage of the surgical procedure. Top and detecting a causal set of frames in the particular surgical video based on the analysis; wherein the cause set of frames is the surgical stage temporally separated from the outcome stage. Within the causal phase of treatment, the intermediate set of frames is in phase with the causal set of frames. in an intermediate stage interposed between said outcome set of frames; generating a causal summary of the surgical video, the causal summary being the cause set of a frame and the outcome set of a frame, and the intermediate set of a frame Eliminate steps and Including, The aggregation of the first group of frames presented to the user is 42. The method of claim 41 , comprising:

47. The cause stage includes the surgical stage in which the cause occurred, and the cause set of a frame includes:

47. The method of claim 46, which is a subset of the frames in the causal phase.

48. The outcome stage includes a surgical stage at which the outcome is observable, and the outcome set of frames includes , a subset of frames in the outcome stage.

49. The method further comprises: determining the surgical outcome and a reason for each of the surgical outcomes using the historical data; The particular surgical video is then subjected to a machine learning model trained to identify the cause of the defect.

47. The method of claim 46, further comprising the step of analyzing:

50. The particular surgical image is performed on a patient and is imaged by at least one image sensor in the operating room. and a surgical procedure captured by said method for storing in said patient's medical record.

52. The method of claim 51, further comprising exporting the first group of frames for The method described above.

51. The method further comprises generating an index of the at least one intra-operative surgical event. and wherein the step of exporting the first group of frames includes exporting a first group of frames before the first group of frames. generating a compilation of the first group, the compilation comprising: and based on a selection of one or more index items, 51. The method of claim 50 configured to enable viewing of one intra-operative surgical event.

52. The compilation includes a sequence of frames of different intraoperative events stored as a continuous video. The method of claim 51 .

53. The method further comprises associating the first group of frames with a unique patient identifier to generate the unique patient identifier.

51. The method of claim 50, further comprising updating the medical record to include the valid patient identifier. 。

54. The position of the at least one image sensor may be located above an operating table in the operating room or above the patient.

52. The method of claim 51, wherein the at least one of

55. The first group of frames in the particular surgical image are divided into the second group of frames. The steps of distinguishing from loops are: analyzing the specific surgical image to detect a medical device; analyzing the particular surgical image to detect anatomical structures; Analyzing the video to identify a relationship between the detected medical device and the detected anatomical structure. detecting relative motion of the The first group of frames is moved to the second group of frames based on the relative motion. a step of distinguishing the first group of frames from a group of frames, the first group of frames being surgical action frames; and wherein the second group of frames includes non-surgical action frames and presents an aggregation. The present invention provides a method for allowing a surgeon preparing for surgery to review the non-surgical presentation during a video review of the presentation to be summarized. A structural frame may be omitted; 52. The method of claim 51 , comprising:

56. Distinguishing said first group of frames from said second group of frames. is further based on a detected relative position between the medical instrument and the anatomical structure. The method described in 55.

57. Distinguishing said first group of frames from said second group of frames. is further based on a detected interaction between the medical device and the anatomical structure. The method described in 55.

58. The step of omitting non-surgical action frames may include omitting a majority of frames capturing non-surgical actions.

56. The method of claim 55, comprising omitting minutes.

59. 1. A system for generating a surgical summary video, comprising: a first group of frames associated with at least one intraoperative surgical event and a surgical action; and a second group of unassociated frames; accessing historical data associated with historical surgical footage of previous surgical procedures; The historical data may be a frame that associates a portion of the historical surgical footage with an intraoperative surgical event. Including information that distinguishes between frames associated with surgical actions and frames that are not associated with surgical actions, Based on the information of the history data, in the particular surgical image, distinguishing one group of frames from the second group of frames; omitting presentation of the second group of frames to the user based on a user request. and presenting to the user an aggregation of the first group of frames of the particular surgical video. do, 23. A system comprising: at least one processor configured to:

60. When executed by at least one processor, the method is capable of generating a surgical overview video. A persistent computer includes instructions for causing the at least one processor to perform operations such as a data-readable medium, the operation comprising: a first group of frames associated with at least one intraoperative surgical event and a surgical action; and a second group of frames that are not associated with the first group of frames. Tep and A system that accesses historical data associated with historical operative footage of previous surgical procedures. the historical data associating a portion of the historical surgical footage with an intraoperative surgical event; A step-by-step procedure that includes information to distinguish between frames that are associated with a surgical action and frames that are not. Top and Based on the information of the history data, in the particular surgical image, distinguishing one group of frames from said second group of frames; omitting presentation of the second group of frames to the user based on a user request. and presenting to the user an aggregation of the first group of frames of the particular surgical video. and 13. A non-transitory computer readable medium comprising:

61. 1. A computer-implemented method for surgical preparation, comprising: It reflects multiple surgical procedures performed on different patients and includes intraoperative events, procedural Multiple analyses of surgical video footage including surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative event characteristics. accessing a repository of sets; A surgeon preparing for a proposed surgical procedure may enabling entry of case specific information; The case-specific information is compared with data associated with the plurality of sets of surgical video footage. In comparison, a group of intraoperative events likely to be encountered during the intended surgical procedure is identified. and Using the case-specific information and the identified groups of intraoperative events likely to be encountered, using the plurality of sets of the surgical video footage corresponding to the identified groups of intraoperative events. identifying a particular frame within a particular set of frames, The particular frame selected may be a frame from the plurality of surgical procedures performed on different patients. and The first and second sets of video footage from different patients share a common characteristic. determining that the frame includes a frame associated with an intraoperative event; omitting to include the second set from the compilation to be presented to the surgeon; including the first set in the compilation to be presented to the surgeon; The comparison includes frames from the different surgical procedures performed on different patients. making a presentation including the image available to the surgeon; 4. A computer-implemented method comprising:

62. one or more corresponding to one or more of the identified particular frames along the presentation. The method further includes the step of enabling display of a common surgical timeline including a time series marker of 62. The method of claim 61.

63. The step of allowing the surgeon to view the presentation may include the step of: sequentially displaying separate sets of video images of the different surgical procedures being performed.

62. The method of claim 61 , comprising:

64. The step of sequentially displaying the separate sets of video images includes sequentially displaying the separate sets of video images. a separate set of said video images that allows said surgeon to select one or more of said 64. The method of claim 63, further comprising the step of displaying an index of.

65. The index converts the separate sets into corresponding surgical stage and text stage indicators.

65. The method of claim 64, including parsing a timeline.

66. The timeline includes intraoperative surgical event markers corresponding to intraoperative surgical events, click on the intraoperative surgical event marker to display a small image depicting the corresponding intraoperative surgical event.

66. The method of claim 65, wherein at least one frame is allowed to be displayed.

67. The case-specific information corresponding to the intended surgical procedure is received from an external device. The method of claim 61 .

68. The case-specific information is compared with data associated with the plurality of sets of surgical video footage. The comparing step may include using an artificial neural network to compare the intended surgical procedure.

7. The method of claim 6, further comprising identifying a group of intraoperative events that are likely to be encountered during the procedure.

2. The method according to claim 1.

69. The step of using the artificial neural network includes inputting the case-specific information and 69. The method of claim 68, comprising the step of: 。

70. The case-specific information includes patient characteristics associated with the proposed treatment. Item 62. The method according to item 61.

71. 71. The method of claim 70, wherein the patient characteristics are received from a medical record of the patient.

72. 72. The method of claim 71, wherein the case specific information includes information related to a surgical instrument.

73. The information related to the surgical instrument may include at least one of an instrument type and an instrument model.

73. The method of claim 72, comprising:

74. 72. The method of claim 71, wherein the common characteristics include characteristics of the different patients.

75. 62. The method of claim 61, wherein the common characteristics include intra-operative surgical event characteristics of the intended surgical procedures. The method described above.

76. The first and second sets of video footage from different patients share a common characteristic. The step of determining that the frames are associated with an intraoperative event may include implementing a machine learning model.

62. The method of claim 61, comprising identifying the common characteristic using

77. The method further comprises determining whether two sets of video footage share the common characteristic. using example video footage to train the machine learning model to The step of implementing the machine learning model further comprises:

77. The method of claim 76, comprising the step of implementing the model.

78. The method comprises: determining a patient's intraoperative surgical events, a surgical outcome, a patient characteristic, a surgeon characteristic, and a machine learning system for generating an index of the repository based on the intraoperative surgical event characteristics; training a learning model and generating the index of the repository. and further comprising: a step of: associating the case-specific information with the plurality of sets of data; 62. The method of claim 61, wherein the step of comparing includes searching the index. How to.

79. It reflects multiple surgical procedures performed on different patients and includes intraoperative events, procedural Multiple analyses of surgical video footage including surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative event characteristics. Access the set repository, A surgeon preparing for a proposed surgical procedure may Allows you to enter case-specific information, The case-specific information is compared with data associated with the plurality of sets of surgical video footage. In comparison, a group of intraoperative events likely to be encountered during the intended surgical procedure is identified. death, Using the case-specific information and the identified groups of intraoperative events likely to be encountered, using the plurality of sets of the surgical video footage corresponding to the identified groups of intraoperative events. identifying a particular frame within a particular set of frames, , including frames from the plurality of surgical procedures performed on different patients; The first and second sets of video footage from different patients share a common characteristic. determining that the frame includes a frame associated with an intraoperative event; omitting to include the second set from the compilation to be presented to the surgeon; including the first set in the compilation to be presented to the surgeon; The compilation includes a set of different surgical procedures performed on different patients. and making the presentation including the frames of the images available to the surgeon viewable by the surgeon.

16. A surgical preparation system comprising: at least one processor configured to:

80. The method includes the steps of: when executed by at least one processor, performing operations to enable surgical preparation; A non-transitory computer-readable medium containing instructions for causing said at least one processor to execute The operation includes: It reflects multiple surgical procedures performed on different patients and includes intraoperative events, procedural Multiple analyses of surgical video footage including surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative event characteristics. accessing a repository of sets; A surgeon preparing for a proposed surgical procedure may enabling entry of case specific information; The case-specific information is compared with data associated with the plurality of sets of surgical video footage. In comparison, a group of intraoperative events likely to be encountered during the intended surgical procedure is identified. and Using the case-specific information and the identified groups of intraoperative events likely to be encountered, using the plurality of sets of the surgical video footage corresponding to the identified groups of intraoperative events. identifying a particular frame within a particular set of frames, The particular frame selected may be a frame from the plurality of surgical procedures performed on different patients. and The first and second sets of video footage from different patients share a common characteristic. determining that the frame includes a frame associated with an intraoperative event; omitting to include the second set from the compilation to be presented to the surgeon; including the first set in the compilation to be presented to the surgeon; The comparison includes frames from the different surgical procedures performed on different patients. making a presentation including the image available to the surgeon; 13. A non-transitory computer readable medium comprising:

81. 1. A computer-implemented method for analyzing complexity of a surgical video, comprising: Analyzing frames of the surgical video and identifying anatomical structures in a first set of frames. and Accessing first historical data, the first historical data being a previous A stage based on an analysis of a first frame of data captured from a first group of surgical procedures. Top and a frame using the first historical data and using the identified anatomical structure; analyzing the first set of frames and determining a first surgical procedure associated with the first set of frames; determining a complexity level; Analyzing the frames of the surgical video to identify a medical instrument, a front end, and a second set of frames. A sequence for identifying the anatomical structure and the interaction between the medical instrument and the anatomical structure. Tep and Accessing second historical data, the second historical data being a previous A stage based on analysis of a second frame of data captured from a second group of surgical procedures. Top and using the second historical data and using the identified interactions to analyzing the second set of frames and determining a second surgical complexity associated with the second set of frames; determining a level of intensity; 23. A computer-implemented method comprising:

82. The step of determining the first surgical complexity level includes determining a first surgical complexity level for the first set of frames.

82. The method of claim 81, further comprising identifying the medical device according to the method.

83. The step of determining the second surgical complexity level comprises:

82. The method of claim 81, based on elapsed time to the second set of frames.

84. The step of determining the first complexity level or the step of determining the second complexity level 82. The method of claim 81 , wherein at least one of the steps is based on a physiological response.

85. The method further comprises: determining a level of skill demonstrated by a health care provider in the surgical video; determining the first complexity level or At least one of the steps of determining the two complexity levels is performed by the healthcare provider.

82. The method of claim 81, based on a determined level of skill demonstrated by:

86. determining that the first surgical complexity level is below a selected threshold; determining that a second surgical complexity level exceeds the selected threshold; determining that the surgical complexity level of the first surgical procedure is less than the selected threshold; and determining that the intensity level exceeds the selected threshold. storing the first set of frames in a data structure while extracting the first set of frames from the data structure; and a. omitting.

87. The step of identifying the anatomical structure in the first set of frames comprises: and a first interaction between the medical instrument and the anatomical structure.

82. The method according to claim 81.

88. tagging the first set of frames with the first surgical complexity level; 、 tagging the second set of frames with the second surgical complexity level; 、 and a second surgical complexity level associated with the first tag to allow a surgeon to select the second surgical complexity level. the first set of frames with the second tag and the second set of frames with the second tag. and generating a data structure including: displaying said second set of frames; 82. The method of claim 81, further comprising:

89. At least one of the first surgical complexity level or the second surgical complexity level. Use frame data captured from previous surgical procedures to determine surgical complications using a machine learning model trained to identify severity levels.

82. The method of claim 81 , comprising:

90. The step of determining the second surgical complexity level includes comparing the first set of frames with the first set of frames.

82. The method of claim 81 , based on events occurring between the second set of frames.

91. At least one of the first surgical complexity level or the second surgical complexity level.

82. The method of claim 81 , wherein the step of determining is based on a state of the anatomical structure.

92. At least one of the first surgical complexity level or the second surgical complexity level.

82. The method of claim 81, wherein the step of determining is based on analysis of electronic medical records.

93. The step of determining the first surgical complexity level includes determining a first surgical complexity level after the first set of frames.

82. The method of claim 81, based on an event occurring in

94. At least one of the first surgical complexity level or the second surgical complexity level. determining a skill level of a surgeon associated with the surgical video; 82. The method of claim 81.

95. The step of determining the second surgical complexity level includes determining a second surgical complexity level after the first set of frames.

82. The method of claim 81, based on an indication that a further surgeon has been paged.

96. The step of determining the second surgical complexity level includes determining a second surgical complexity level after the first set of frames.

82. The method of claim 81, based on an indication that a particular medication has been administered to.

97. The first historical data is captured from the first group of previous surgical procedures.

82. The method of claim 81, further comprising: The method described above.

98. The first historical data includes statistics between particular anatomical structures and particular surgical complexity levels.

82. The method of claim 81, further comprising displaying a correlation.

99. 1. A system for analyzing complexity of a surgical video, comprising: Analyzing frames of the surgical video to identify anatomical structures in a first set of frames. Identify, First historical data is accessed, the first historical data being a first historical data of a previous surgical procedure. based on an analysis of a first frame of data captured from the group; a frame using the first historical data and using the identified anatomical structure; analyzing the first set of frames to identify a first hand associated with the first set of frames; Determine the surgical complexity level, Analyzing the frames of the surgical video to identify a medical instrument, a front end, and a second set of frames. identifying the anatomical structure and an interaction between the medical device and the anatomical structure; Accessing second historical data, the second historical data including a second history of a previous surgical procedure. based on an analysis of a second frame of data captured from the group; using the second historical data and using the identified interactions to analyzing the second set of frames and determining a second surgical complexity associated with the second set of frames; Determine the degree level, 23. A system comprising at least one processor configured to:

100. When executed by at least one processor, enables review of surgical video instructions for causing said at least one processor to perform the operations of: A readable medium, the operation comprising: Analyzing frames of the surgical video and identifying anatomical structures in a first set of frames. and Accessing first historical data, the first historical data being a previous A stage based on an analysis of a first frame of data captured from a first group of surgical procedures. Top and a frame using the first historical data and using the identified anatomical structure; analyzing the first set of frames and determining a first surgical procedure associated with the first set of frames; determining a complexity level; Analyzing the frames of the surgical video to identify a medical instrument in a second set of frames. identifying anatomical structures and interactions between said medical instrument and said anatomical structures; P and Accessing second historical data, the second historical data being a previous A stage based on analysis of a second frame of data captured from a second group of surgical procedures. Top and the second historical data being based on an analysis of frame data captured from a previous surgical procedure; and analyzing the second set of frames using the identified interactions. determining a second surgical complexity level associated with said second set of frames; Top and 13. A non-transitory computer readable medium comprising:

101. When executed by at least one processor, adjusting the operating room schedule. A persistent computer comprising instructions for causing said at least one processor to perform a method for enabling said persistent computer. A computer-readable medium, the method comprising: Image sensors located in the operating room receive visual data that tracks the ongoing surgical procedure. The step of believing, accessing a data structure containing information based on historical surgical data; A method for determining an estimated time of completion of the ongoing surgical procedure using the data structure. analyzing the visual data of the ongoing surgical procedure; a time schedule associated with the completion of the ongoing surgical procedure; accessing a schedule for the operating room; Based on the estimated completion time of the ongoing surgical procedure, a predicted completion time is associated with the completion. whether it is likely to result in deviation from the scheduled time associated with A calculating step; By outputting a notification upon the calculation of the variance, subsequent users of the operating room may be notified accordingly. Steps to help you adjust your schedule accordingly; 13. A non-transitory computer readable medium comprising:

102. 102. The persistent control system of claim 101, wherein the notification includes an updated operating room schedule. Computer readable media.

103. The updated operating room schedule is updated to allow queued healthcare professionals to continue. The persistent computer of claim 101 enables preparation for a surgical procedure. Computer-readable medium.

104. The method further comprises: providing a device associated with a subsequent scheduled user of the operating room; 102. The method of claim 101, further comprising the step of electronically transmitting said notification to Computer-readable medium.

105. The method comprises: determining a range of the variance from the scheduled time associated with the completion; and outputting said notification in response to a first determined range; refraining from outputting the notification in response to the second determined range; 102. The non-transitory computer readable medium of claim 101, further comprising:

106. The method comprises: The predicted completion time is calculated from the scheduled time associated with the completion. A step of determining whether a delay of at least a selected threshold time is likely to result Top and the predicted completion time is likely to result in a delay of at least the selected threshold time outputting said notification in response to said determination; the predicted completion time is unlikely to result in a delay of at least the selected threshold time; in response to the determination, not outputting the notification; 102. The non-transitory computer readable medium of claim 101, further comprising:

107. The step of determining the estimated completion time may be performed by a healthcare provider performing the ongoing surgical procedure.

102. The method of claim 101, wherein the method is based on one or more stored characteristics associated with a professional. A persistent computer readable medium.

108. updating the historical average time to completion based on the determined actual time; 102. The method of claim 101, further comprising the step of completing a surgical procedure. Data-readable medium.

109. 102. The persistent computer of claim 101, wherein the image sensor is positioned above a patient. Readable medium.

110. 102. The persistent computer of claim 101, wherein the image sensor is located on a surgical instrument. Readable medium.

111. The analyzing step includes detecting characteristic events within the received visual data; Evaluating information based on historical surgical data to identify the characteristic event within the historical surgical data. determining a predicted time to complete the surgical procedure after occurrence of an elephant; and determining the estimated completion time based on a forecasted completion time.

13. The non-transitory computer readable medium of claim 1.

112. The method includes using historical visual data to train a machine learning model; 112. The persistent computer readable method of claim 111, further comprising the step of detecting a characteristic event. Medium.

113. The method includes: historically analyzing the data to train a machine learning model to estimate completion time; using visual data such as the time to complete the route, Implementing the trained machine learning model. A non-transitory computer readable medium.

114. The method includes determining the estimated completion time using an average historical completion time.

102. The non-transitory computer readable medium of claim 101, further comprising:

115. The method further includes detecting a medical instrument within the visual data, 102. The method of claim 101, wherein calculating a completion time is based on the detected medical device. A non-transitory computer readable medium.

116. The analyzing step further includes detecting anatomical structures within the visual data. The step of calculating an estimated time of completion may include calculating a claim based on the detected anatomical structure.

102. The non-transitory computer readable medium of claim 101.

117. The analyzing step includes analyzing interactions between anatomical structures and medical instruments in the visual data. The step of calculating the estimated completion time further includes the step of detecting the detected 102. The non-transitory computer readable medium of claim 101, wherein the non-transitory computer readable medium is based on an interaction.

118. The analyzing step includes determining a skill level of a surgeon in the visual data. and wherein the step of calculating the estimated completion time is based on the determined skill level.

102. The non-transitory computer-readable medium of claim 101 .

119. 1. A system for enabling adjustment of operating room schedules, comprising: Image sensors located in the operating room receive visual data that tracks the ongoing surgical procedure. Believe, accessing a data structure containing information based on historical surgical data; A method for determining an estimated time of completion of the ongoing surgical procedure using the data structure. analyzing the visual data of the ongoing surgical procedure; a time schedule associated with the completion of the ongoing surgical procedure; Access operating room schedules, The predicted completion time is determined based on the estimated completion time of the ongoing surgical procedure. whether it is likely to result in variance from the scheduled time associated with Calculate, By outputting a notification upon the calculation of the variance, subsequent users of the operating room may be notified accordingly. Be able to adjust your schedule accordingly 23. A system comprising at least one processor configured to:

120. 1. A computer-implemented method for enabling adjustments to operating room schedules, comprising: Image sensors located in the operating room receive visual data that tracks the ongoing surgical procedure. The step of believing, accessing a data structure containing information based on historical surgical data; A method for determining an estimated time of completion of the ongoing surgical procedure using the data structure. analyzing the visual data of the ongoing surgical procedure; a time schedule associated with the completion of the ongoing surgical procedure; accessing a schedule for the operating room; The predicted completion time is determined based on the estimated completion time of the ongoing surgical procedure. whether it is likely to result in variance from the scheduled time associated with A calculating step; By outputting a notification upon the calculation of the variance, subsequent users of the operating room may be notified accordingly. Steps to help you adjust your schedule accordingly; 23. A computer-implemented method comprising:

121. A computer-implemented method for analyzing surgical images to determine insurance reimbursement. hand, accessing video frames captured during a surgical procedure on a patient; Analyzing the video frames captured during the surgical procedure to identify at least one medical device, at least one anatomical structure, and At least one interaction between a medical device and the at least one anatomical structure. and identifying Correlated to medical devices, anatomical structures, and interactions between medical devices and anatomical structures accessing a database of refund codes for The discrimination between the at least one medical device and the at least one anatomical structure. comparing at least one of the interactions to information in said database of refund codes; determining at least one reimbursement code associated with said surgical procedure. P and said at least one payment method for use in obtaining insurance reimbursement for said surgical procedure; outputting a return code; 23. A computer-implemented method comprising:

122. The at least one outputted refund code may be a plurality of outputted refund codes.

122. The method of claim 121, comprising:

123. At least two of the plurality of output reimbursement codes share a common anatomical structure.

123. The method of claim 122, wherein the interaction is based on differential interactions with

124. The at least two output rebate codes are associated with the detection of two different medical devices. The method of claim 123, wherein the method is based on partitioning.

125. The step of determining at least one reimbursement code may also include analyzing post-operative reports. The method of claim 121 based on

126. The video frames are captured from an image sensor positioned above the patient. Item 122. The method according to item 121.

127. The video frames are captured from an image sensor associated with a medical device. The method of claim 121.

128. by associating said at least one reimbursement code with said surgical procedure; 122. The method of claim 121, further comprising the step of updating the database.

129. A plurality of medical devices in a historical video footage, the medical devices being processed according to a reimbursement code, A plurality of anatomical structures in a video image, or a medical device and an anatomical structure in the historical video image. A process for generating a correlation between at least one of a plurality of interactions between the biological structure and the and updating the database based on the generated correlations.

122. The method of claim 121, further comprising:

130. 13. The method of claim 12, wherein generating the correlation comprises implementing a statistical model.

9. The method according to claim 9.

131. Using a machine learning model, a plurality of medical devices, a plurality of solutions, and a plurality of medical devices are detected within the historical video footage. At least one of the anatomical structures or interactions between the medical device and the anatomical structures 130. The method of claim 129, further comprising the step of detecting one.

132. The video frames captured during the surgical procedure are analyzed to identify the patient's anatomy. determining a state of the anatomical structure; and determining the state of the anatomical structure based on the determined state of the anatomical structure. determining the at least one reimbursement code associated with the surgical procedure; 122. The method of claim 121, further comprising:

133. The video frames captured during the surgical procedure are analyzed to determine a pre-operative determining a change in the state of the patient's anatomy; determining the at least one payment associated with the surgical procedure based on the determined change; and determining a return code.

134. The video captured during the surgical procedure to determine the use of a particular medical device. analyzing the frame and determining based on the determined use of the particular medical device. determining said at least one reimbursement code associated with said surgical procedure using said at least one reimbursement code; 122. The method of claim 121, further comprising:

135. 2. A method for determining a type of use of a particular medical device, comprising: analyzing the video frames according to the first determined usage type; determining at least a first reimbursement code associated with the surgical procedure; In response to the second determined use type, at least a second determining a refund code for the at least a first refund code, different from the at least second refund code. The method described above.

136. receiving a processed reimbursement code associated with the surgical procedure; 13. The method of claim 12, further comprising the step of updating the database based on a refund code.

2. The method according to claim 1.

137. The processed refund code is a corresponding refund code of the at least one refund code.

137. The method of claim 136, wherein the return code is different from the return code.

138. The video frames captured during the surgical procedure are analyzed for use in the surgical procedure. determining an amount of a particular type of medical supply to be dispensed; and determining said at least one reimbursement code associated with said surgical procedure; 122. The method of claim 121, further comprising:

139. 1. A surgical image analysis system for determining insurance reimbursement, comprising: Accessing video frames captured during a surgical procedure on a patient; Analyzing the video frames captured during the surgical procedure to identify at least one medical device, at least one anatomical structure, and At least one interaction between a medical device and the at least one anatomical structure. Identify Correlated to medical devices, anatomical structures, and interactions between medical devices and anatomical structures Access a database of refund codes that The discrimination between the at least one medical device and the at least one anatomical structure. comparing at least one of the interactions to information in said database of refund codes; determining at least one reimbursement code associated with the surgical procedure; said at least one payment method for use in obtaining insurance reimbursement for said surgical procedure; Print the return code, 16. A surgical image analysis system comprising: at least one processor configured to:

140. When executed by at least one processor, enables insurance reimbursement decisions instructions for causing said at least one processor to perform the operations of: A readable medium, the operation comprising: accessing video frames captured during a surgical procedure on a patient; Analyzing the video frames captured during the surgical procedure to identify at least one medical device, at least one anatomical structure, and At least one interaction between a medical device and the at least one anatomical structure. and identifying Correlated to medical devices, anatomical structures, and interactions between medical devices and anatomical structures accessing a database of refund codes for The discrimination between the at least one medical device and the at least one anatomical structure. comparing at least one of the interactions to information in said database of refund codes; determining at least one reimbursement code associated with said surgical procedure. P and said at least one payment method for use in obtaining insurance reimbursement for said surgical procedure; outputting a return code; 13. A non-transitory computer readable medium comprising:

141. When executed by at least one processor, the method automates post-operative reporting of a surgical procedure. instructions for causing the at least one processor to perform operations to enable a a non-transitory computer readable medium comprising: receiving an input of a patient identifier; receiving an input of a healthcare provider identifier; inputting operative footage of a surgical procedure performed on the patient by the healthcare provider; receiving the To analyze a plurality of frames of said surgical video to generate a post-operative report of said surgical procedure. deriving image-based information of The derived image-based information is adapted to generate the post-operative report of the surgical procedure. and 13. A non-transitory computer readable medium comprising:

142. The operation includes identifying one or more stages of the surgical procedure and selecting one of the identified stages. The method further includes analyzing the surgical video to identify a characteristic of at least one stage. The derived image-based information is based on the at least one identified stage and a previous and the identified characteristic of at least one stage. Computer-readable medium.

143. The operation includes analyzing the surgical video and associating a name with the at least one step. and wherein the derived image-based information is 143. The non-transitory computer readable medium of claim 142, further comprising the name associated with

144. The operations further include determining at least a start of the at least one stage.

142. The method of claim 141, wherein the derived image-based information is based on the determined onset. A non-transitory computer readable medium as described above.

145. The operations further include associating a time marker with the at least one stage. before the derived image-based information is associated with the at least one stage.

143. The non-transitory computer readable medium of claim 142, comprising the time marker.

146. The action includes transmitting data to the health care provider, the data to be collected includes the patient identifier and the derived image-based information.

142. The non-transitory computer-readable medium of claim 141, further comprising:

147. The operation includes analyzing the surgical video to identify at least one recommendation for post-operative treatment. and providing the identified at least one recommendation.

142. The non-transitory computer readable medium of claim 141.

148. The triggered creation of the post-operative report of the surgical procedure includes the derived The health care provider may modify at least a portion of the image-based information stored in the health care provider.

142. The non-transitory computer-readable medium of claim 141, configured to

149. The generated post-operative report of the surgical procedure is generated based on the derived image. identifying at least a portion of the information in said postoperative report as automatically generated data.

142. The non-transitory computer-readable medium of claim 141 configured to

150. The operations include analyzing the surgical video to identify surgical events within the surgical video; and further comprising identifying a characteristic of the identified surgical event, and 15. The method of claim 14, wherein the information based on the identified surgical event and the identified characteristics is based on the identified surgical event.

13. The non-transitory computer readable medium of claim 1.

151. The operations include analyzing the surgical video to determine an event name for the identified surgical event. and wherein the derived image-based information includes the determined event name.

151. The non-transitory computer readable medium of claim 150.

152. The operations further include associating a time marker with the identified surgical event; 151. The sustained imaging system of claim 150, wherein the derived image-based information includes the time markers. A computer readable medium.

153. The operation includes: updating the derived image-based information in a manner that enables updating an electronic medical record.

142. The non-transitory computer readable medium of claim 141, further comprising the step of providing in a formula:

154. 142. The method of claim 141, wherein the derived image-based information is based in part on user input.

20. The non-transitory computer readable medium of claim 19 .

155. The derived image-based information is associated with a first portion of the surgical procedure. a first portion and a second portion associated with a second portion of the surgical procedure; The operation includes: receiving a preliminary post-operative report; Analyzing the preliminary post-operative report to select a first location and a second location within the preliminary post-operative report. wherein the first location is associated with the first portion of the surgical procedure. and the second location is associated with the second portion of the surgical procedure. inserting the first portion of the derived image-based information into the selected first location. inputting the second portion of the derived image-based information into the selected second position; and inserting the Further comprising:

142. The non-transitory computer readable medium of claim 141.

156. The operation includes analyzing the surgical video to determine a minimum of one frame of the surgical video. selecting at least a portion of the at least one frame of the surgical video; and including at least a portion of the selected features in the post-operative report of the surgical procedure.

142. The non-transitory computer-readable medium of claim 141.

157. The operation includes: receiving a preliminary post-operative report; The preliminary post-operative report and the surgical video are analyzed to identify at least one frame of the surgical video. selecting at least a portion of the group; 157. The non-transitory computer readable medium of claim 156, further comprising:

158. The operation includes: receiving a preliminary post-operative report; The preliminary post-operative report and the surgical video are analyzed, and a correlation between the preliminary post-operative report and the surgical video is obtained. identifying at least one inconsistency between the providing an indication of the identified at least one inconsistency; 142. The non-transitory computer readable medium of claim 141, further comprising:

159. 1. A computer-implemented method for generating a post-operative report of a surgical procedure, comprising: receiving an input of a patient identifier; receiving an input of a healthcare provider identifier; inputting surgical footage of a surgical procedure performed on the patient by the healthcare provider; a step of receiving a force; A plurality of frames of the surgical video are analyzed to detect a relationship between the medical device and the biological structure. identifying a stage of the surgical procedure based on the interaction; and associating a name with each identified stage. determining at least the start of each identified stage; associating a time marker with the start of each identified stage; transmitting data to the health care provider, the transmitted data , the patient identifier, the name of the identified stage of the surgical procedure, and the identified a step of: The transmitted data is used to generate a post-operative report, and the health care provider reports to the post-operative report. creating a step in such a manner that the stage name can be changed; 23. A computer-implemented method comprising:

160. 1. A system for automatically generating a post-operative report of a surgical procedure, comprising: receiving an input of a patient identifier; receiving an input of a healthcare provider identifier; inputting surgical footage of a surgical procedure performed on the patient by the healthcare provider; a step of receiving a force; A plurality of frames of the surgical video are analyzed to detect a relationship between the medical device and the biological structure. identifying a stage of the surgical procedure based on the interaction; and associating a name with each identified stage. determining at least the start of each identified stage; associating a time marker with the start of each identified stage; transmitting data to the health care provider, the transmitted data , the patient identifier, the name of the identified stage of the surgical procedure, and the identified including the time markers of the selected stage; The transmitted data is used to generate a post-operative report, and the health care provider reports to the post-operative report. creating a step in such a manner that the stage name can be changed; A system including:

161. When executed by at least one processor, the method includes: instructions for causing said at least one processor to perform operations enabling event determination and notification; A non-transitory computer readable medium containing instructions, the operations comprising: accessing frames of video captured during a particular surgical procedure; Accessing stored data identifying a recommended course of events for said surgical procedure. and comparing the accessed frames to the recommended sequence of events to determine the specific surgical procedure; identifying indications of deviation between the proposed surgical procedure and the recommended course of events for the surgical procedure; and determining a name of the intraoperative surgical event associated with the deviation; providing a notice of the deviation including the name of the intraoperative surgical event associated with the deviation. Steps and 13. A non-transitory computer readable medium comprising:

162. The steps of identifying the indication of the deviation and providing the notification may include 162. The non-transitory computer readable medium of claim 161, which occurs in real time during a procedure.

163. The operation includes: Receive an indication that a particular action is about to occur in said particular surgical procedure. and Using the recommended sequence of events, identify preliminary actions for the particular action. Steps and Based on an analysis of the accessed frames, the identified preliminary action has not yet occurred. determining that the In response to determining that the identified preparatory action has not yet occurred, identifying a representation; 162. The non-transitory computer readable medium of claim 161, further comprising:

164. 162. The persistent computer program of claim 161, wherein the particular surgical procedure is a cholecystectomy. Readable medium.

165. 162. The method of claim 161, wherein the recommended course of events is based on a safety hazard perspective. A persistent computer readable medium.

166. 162. The persistent computer-implemented method of claim 161, wherein the particular surgical procedure is an appendectomy. Reading medium.

167. 162. The persistent computing device of claim 161, wherein the particular surgical procedure is hernia repair. Data-readable medium.

168. 162. The persistent computer-implemented method of claim 161, wherein the particular surgical procedure is a hysterectomy. Reading medium.

169. The sustained release conjugate of claim 161, wherein the specific surgical procedure is a radical prostatectomy. Computer readable media.

170. The particular surgical procedure is a partial nephrectomy, and the deviation is without identifying the renal hilum.

162. The non-transitory computer-readable medium of claim 161, further comprising:

171. The particular surgical procedure is a thyroidectomy, and the deviation identifies a recurrent laryngeal nerve.

162. The non-transitory computer-readable medium of claim 161, comprising ignoring

172. The operations further include identifying a set of frames associated with the deviation. and the step of providing notification comprises:

162. The non-transitory computer readable medium of claim 161, further comprising the step of displaying the set.

173. The indication that the particular action is about to occur may be 162. The non-transitory computer readable medium of claim 161, based on input from a surgeon performing the steps.

174. The indication that the particular action is about to occur may include a 164. The non-transitory computer readable medium of claim 163, which is the entry of a particular medical device.

175. The step of identifying the deviation may include a step of determining that the surgical instrument is within a particular anatomical region.

162. The non-transitory computer readable medium of claim 161, comprising the steps of:

176. 162. The persistent computing device of claim 161, wherein the particular surgical procedure is a hemicolectomy. Data-readable medium.

177. 177. The method of claim 176, wherein the deviation includes ignoring performing an anastomosis. Computer readable media.

178. The step of identifying the indication of deviation may include determining whether the indication is based on an elapsed time associated with the intraoperative surgical procedure.

162. The non-transitory computer-readable medium of claim 161.

179. COMPUTER IMPLEMENTATION TO ENABLE DETERMINATION AND NOTIFICATION OF OMITTED EVENTS IN SURGICAL PROCEDURES - Patent application 1. A method comprising: accessing frames of video captured during a particular surgical procedure; Accessing stored data identifying a recommended course of events for said surgical procedure. and comparing the accessed frames to the recommended sequence of events to determine the specific surgical procedure; identifying indications of deviation between the proposed surgical procedure and the recommended course of events for the surgical procedure; and determining a name of the intraoperative surgical event associated with the deviation; providing a notice of the deviation including the name of the intraoperative surgical event associated with the deviation. Steps and 23. A computer-implemented method comprising:

180. A system for enabling the determination and notification of omitted events in a surgical procedure. 、 Accessing frames of video captured during a particular surgical procedure; Accessing stored data identifying a recommended course of events for said surgical procedure. death, comparing the accessed frames to the recommended sequence of events to determine the specific surgical procedure; and identifying an indication of deviation between the proposed procedure and the recommended course of events for the surgical procedure. 、 determining a name of the intraoperative surgical event associated with the deviation; providing a notice of the deviation including the name of the intraoperative surgical event associated with the deviation. 、 23. A system comprising at least one processor configured to:

181. When executed by at least one processor, the method provides decision support regarding a surgical procedure. A persistent computer comprising instructions for causing the at least one processor to perform operations to a data-readable medium, the operation comprising: A stepper that receives video footage of a surgical procedure being performed by a surgeon on a patient in an operating room. P and Accessing at least one data structure containing image-related data characterizing the surgical procedure. and The image-related data is used to analyze the received video footage and to inform surgical decision making. determining the presence of a connection; In the at least one data structure, an outcome and a decision action taken at the decision junction are accessing a correlation between the specific action being performed; the determined existence of the decision junction and the accessed correlations; outputting a recommendation to the user to perform the particular action based on the recommendation; 13. A non-transitory computer readable medium comprising:

182. The instructions cause the at least one processor to perform the operations realistically during the surgical procedure.

182. The method of claim 181, wherein the method is configured to run at a specific time and the user is the surgeon. A non-transitory computer readable medium.

183. The decision junctions are multiple cases where different action courses have taken place following a common surgical situation. The persistent comparator of claim 181, determined by analysis of a number of different historical treatments. Computer-readable medium.

184. the video footage includes images from at least one of an endoscope and an intra-body camera; 182. The non-transitory computer readable medium of claim 181.

185. 182. The persistent computing device of claim 181, wherein the recommendation includes a recommendation to perform a medical test. Data-readable medium.

186. The operation includes: receiving the results of said medical test; The determined existence of the decision junction, the accessed correlations, and and a second recommendation to take a particular action based on the received results of the medical test. to the user; 186. The non-transitory computer readable medium of claim 185, further comprising:

187. 181. The method of claim 180, wherein the specific action includes bringing an additional surgeon into the operating room. A non-transitory computer readable medium as described above.

188. The decision junction may be due to inadequate access or exposure, retraction of anatomy, or resolution.

181. The method of claim 181, comprising at least one of: misinterpretation of anatomical structures; or fluid leakage. A non-transitory computer readable medium as described above.

189. The recommendation may include a degree of confidence that a desired surgical outcome will occur if the particular action is taken.

182. The non-transitory computer readable medium of claim 181, comprising a level.

190. The recommendation may include a confidence that a desired outcome will not occur if the particular action is not taken.

182. The non-transitory computer-readable medium of claim 181, further comprising a degree level.

191. The recommendation is based on the time elapsed since a particular point in the surgical procedure.

181. The non-transitory computer readable medium of claim 181.

192. The recommendation may be a measure of the likelihood of an undesirable surgical procedure occurring if the particular action is not taken.

182. The non-transitory computer readable medium of claim 181, comprising an indication of an outcome.

193. 182. The method of claim 181, wherein the recommendation is based on a skill level of the surgeon. Computer-readable medium.

194. The recommendation is based on a surgical event that occurred in the surgical procedure prior to the decision junction.

182. The non-transitory computer-readable medium of claim 181,

195. 182. The persistent computer-implemented method of claim 181, wherein the particular operation comprises a plurality of steps. Reading medium.

196. The determination of the presence of the surgical decision junction may include determining a location of the surgical decision junction based on a detected anatomy.

18. The method of claim 17, further comprising:

13. The non-transitory computer readable medium of claim 1.

197. The operations further include receiving vital signs of the patient, and the recommendation includes:

182. The method of claim 181, based on the accessed correlations and the vital signs. A persistent computer readable medium.

198. The surgeon is a surgical robot, and the recommendation is in the form of a command to the surgical robot.

182. The non-transitory computer readable medium of claim 181, wherein:

199. The persistent computing device of claim 181, wherein the recommendation is based on a tissue condition of the patient. Data-readable medium.

200. 182. The persistent computing device of claim 181, wherein the recommendation of a particular action includes forming a fistula. Computer-readable medium.

201. 1. A computer-implemented method for providing decision support regarding a surgical procedure, comprising: A stepper that receives video footage of a surgical procedure being performed by a surgeon on a patient in an operating room. P and Accessing at least one data structure containing image-related data characterizing the surgical procedure. and The image-related data is used to analyze the received video footage and to inform surgical decision making. determining the presence of a connection; In the at least one data structure, an outcome and a decision action taken at the decision junction are accessing a correlation between the specific action being performed; the determined existence of the decision junction and the accessed correlations; Based on the result, the surgeon is instructed to perform the particular action or to avoid the particular action. outputting the recommendations to 23. A computer-implemented method comprising:

202. 1. A system for providing decision support regarding a surgical procedure, comprising: receiving video footage of a surgical procedure being performed by a surgeon on a patient in an operating room; Accessing at least one data structure containing image-related data characterizing the surgical procedure. death, The image-related data is used to analyze the received video footage and to inform surgical decision making. Determine the existence of a connection, In the at least one data structure, an outcome and a decision action taken at the decision junction are Access the correlation between the specific actions that can be taken, the determined existence of the decision junction and the accessed correlations; Based on the result, the surgeon is instructed to perform the particular action or to avoid the particular action. output the recommendations to 23. A system comprising at least one processor configured to:

203. When executed by at least one processor, the method comprises: and causing the at least one processor to perform an operation to estimate a contact force acting on the a non-transitory computer readable medium including instructions for causing a user to: A stage for receiving image data of a surgical procedure from at least one image sensor in the operating room. Top and analyzing the received image data to determine an identity of an anatomical structure; determining a state of the anatomical structure as reflected in the image data; selecting a contact force threshold associated with the anatomical structure, the determined contact force threshold is based on the determined state of the anatomical structure; and receiving an indication of actual contact forces acting on the anatomical structure; comparing the indication of actual contact force with the selected contact force threshold; Based on a determination that the indication of actual contact force exceeds the selected contact force threshold, outputting the knowledge; 13. A non-transitory computer readable medium comprising:

204. 204. The persistent computer of claim 203, wherein the contact force threshold is associated with a tension level. Data-readable medium.

205. 204. The persistent computer of claim 203, wherein the contact force threshold is associated with a compression level. Data-readable medium.

206. The actual contact force is associated with contact between a medical device and the anatomical structure. The non-transitory computer-readable medium of claim 203.

207. 4. The method of claim 3, wherein the indication of actual contact force is estimated based on image analysis of the image data.

203. The non-transitory computer readable medium of claim 203.

208. The step of outputting the notification provides a real-time alert to a surgeon performing the surgical procedure.

204. The non-transitory computer-readable medium of claim 203, comprising the step of:

209. 204. The persistent computing device of claim 203, wherein the notification is a command to a surgical robot. Computer-readable medium.

210. The operations include determining from the image data that the surgical procedure is in a combat mode.

204. The method of claim 203, further comprising: A computer readable medium.

211. The operation determines from the image data that the surgeon is operating in a mode that ignores contact force notifications. determining that the surgeon is operating in the mode that ignores contact force notifications; at least temporarily withholding further contact force notifications based on the determination that the contact force notification has been received.

204. The non-transitory computer readable medium of claim 203, further comprising:

212. The step of selecting the contact force threshold may include determining the location of contact between the anatomical structure and a medical device.

204. The non-transitory computer-readable medium of claim 203, wherein the non-transitory computer-readable medium is based on a configuration.

213. The step of selecting the contact force threshold is based on a contact angle between the anatomical structure and a medical device.

204. The non-transitory computer-readable medium of claim 203.

214. The step of selecting the contact force threshold comprises: and selecting the contact force threshold based on an output of the regression model.

204. The non-transitory computer readable medium of claim 203, comprising:

215. The step of selecting the contact force thresholds may include selecting a table of anatomical structures including corresponding contact force thresholds.

204. The non-transitory computer-readable medium of claim 203, based on a program.

216. The step of selecting the contact force threshold is based on an action performed by a surgeon.

204. The non-transitory computer readable medium of claim 203.

217. 204. The persistent contact device of claim 203, wherein the indication of actual contact force is received from a surgical instrument. Computer readable media.

218. 204. The surgical device of claim 203, wherein the indication of actual contact force is received from a surgical robot. A computer readable medium.

219. The operations include: using a machine learning model trained using training examples; 21. The method of claim 20, further comprising determining a state of the anatomical structure in the image data.

4. The non-transitory computer readable medium of claim 3.

220. The operation includes: performing a preprocessing operation using a machine learning model trained using training examples; 204. The persistent computer of claim 203, further comprising the step of selecting said contact force threshold. Readable medium.

221. Computer-implemented method for estimating contact forces acting on anatomical structures during surgical procedures - Patents.com And, A stage for receiving image data of a surgical procedure from at least one image sensor in the operating room. Top and analyzing the received image data to determine an identity of an anatomical structure; determining a state of the anatomical structure as reflected in the image data; selecting a contact force threshold associated with the anatomical structure, the determined contact force threshold is based on the determined state of the anatomical structure; and receiving an indication of actual contact forces acting on the anatomical structure; comparing the indication of actual contact force with the selected contact force threshold; Based on a determination that the indication of actual contact force exceeds the selected contact force threshold, outputting the knowledge; 23. A computer-implemented method comprising:

222. 1. A system for estimating contact forces acting on an anatomical structure during a surgical procedure, comprising: receiving image data of the surgical procedure from at least one image sensor in the operating room; analyzing the received image data to determine an identity of an anatomical structure; determining a state of the anatomical structure reflected in the image data; selecting a contact force threshold associated with the anatomical structure, the selected contact force threshold being Based on the determined state of the anatomical structure, receiving an indication of actual contact forces acting on said anatomical structure; comparing the indication of actual contact force with the selected contact force threshold; Based on a determination that the indication of actual contact force exceeds the selected contact force threshold, Outputting knowledge, 23. A system comprising at least one processor configured to:

223. When executed by the at least one processor, the method updates the predicted outcome during the surgical procedure. instructions for causing the at least one processor to perform operations to enable updating A computer-readable medium, the operations comprising: from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a first event during a surgical procedure; determining whether the surgical procedure is performed based on the received image data associated with the first event; determining a predicted outcome associated with the placement; from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a second event during the surgical procedure; determining the predicted outcome based on the received image data associated with the second event; determining a change in the predicted outcome, whereby the predicted outcome falls below a threshold; accessing a data structure of image-related data based on a previous surgical procedure; identifying a recommended corrective action based on the accessed image-related data. P and outputting said recommended corrective action; 13. A non-transitory computer readable medium comprising:

224. The recommended corrective action may include a recommendation for the surgeon to stop the surgical procedure and take a break.

224. The non-transitory computer readable medium of claim 223.

225. 23. The method of claim 22, wherein the recommended corrective action includes a recommendation to request assistance from another surgeon.

4. The non-transitory computer readable medium of claim 3.

226. 224. The method of claim 223, wherein the recommended corrective action comprises a modification to the surgical procedure. A non-transitory computer readable medium.

227. The non-transitory computer readable medium of claim 223, wherein the predicted outcome comprises the likelihood of hospitalization. body.

228. 223. The method of claim 223, wherein the step of determining the change in predicted outcome is based on a magnitude of bleeding.

20. The non-transitory computer readable medium of claim 19 .

229. The step of identifying a corrective action comprises determining whether the corrective action increases the predicted outcome above a threshold.

224. The persistent computer readable memory of claim 223, Medium.

230. The step of identifying a corrective action may include identifying the corrective action using historical examples of the corrective action and surgical outcomes. using a machine learning model trained to identify corrective actions; 224. The non-transitory computer readable medium of claim 223.

231. The step of determining a predicted outcome may include comparing the historical surgical video with the historical surgical video. A machine trained to determine a predicted outcome based on information indicating the corresponding surgical outcome.

224. The persistent computer readable method of claim 223, comprising using a machine learning model. Medium.

232. The step of determining a predicted outcome may include identifying interactions between a surgical instrument and an anatomical structure. determining the predicted outcome based on the identified interactions.

224. The non-transitory computer readable medium of claim 223, comprising:

233. The step of determining the predicted outcome may include determining a skill level of a surgeon as depicted in the image data.

224. The non-transitory computer-readable medium of claim 223, wherein the non-transitory computer-readable medium is based on a rule.

234. The operations include determining a skill level of a surgeon as represented in the image data. and determining the change in the predicted outcome is based on the skill level. The non-transitory computer-readable medium of claim 223.

235. The action is to discontinue a procedure associated with the surgical procedure in response to the predicted outcome falling below a threshold.

224. The method of claim 223, further comprising the step of updating a schedule record associated with the operating room.

20. The non-transitory computer readable medium of claim 19 .

236. The step of determining the change in the predicted outcome may include determining a change in the predicted outcome at a particular point in the surgical procedure.

224. The persistent computing device of claim 223, wherein the determination is based on an elapsed time between the first event and the second event. Data-readable medium.

237. The step of determining the predicted outcome may include determining the state of an anatomical structure depicted in the image data.

224. The non-transitory computer-readable medium of claim 223, based on

238. 238. The method of claim 237, wherein the operations further include determining a state of the anatomical structure.

20. The non-transitory computer readable medium of claim 19 .

239. The step of determining the change in the predicted outcome may include determining a change in the predicted outcome by measuring at least a portion of the anatomical structure.

224. The non-transitory computer readable medium of claim 223 based on a change in color.

240. The step of determining the change in the predicted outcome may include determining a change in the predicted outcome by measuring at least a portion of the anatomical structure.

224. The non-transitory computer readable medium of claim 223 based on a change in appearance.

241. 1. A computer-implemented method for updating a predicted outcome during a surgical procedure, comprising: from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a first event during a surgical procedure; determining whether the surgical procedure is performed based on the received image data associated with the first event; determining a predicted outcome associated with the placement; from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a second event during the surgical procedure; determining the predicted outcome based on the received image data associated with the second event; determining a change in the predicted outcome, whereby the predicted outcome falls below a threshold; accessing a data structure of image-related data based on a previous surgical procedure; identifying a recommended corrective action based on the data structure; outputting said recommended corrective action; 23. A computer-implemented method comprising:

242. 1. A system for updating a predicted outcome during a surgical procedure, comprising: from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a first event during a surgical procedure; determining whether the surgical procedure is performed based on the received image data associated with the first event; determining a predicted outcome associated with the placement; from at least one image sensor positioned to capture images of the surgical procedure; receiving image data associated with a second event during the surgical procedure; determining the predicted outcome based on the received image data associated with the second event; determining a change in, thereby reducing said predicted outcome below a threshold; Accessing a data structure of image-related data based on a previous surgical procedure; Identifying a recommended corrective action based on said data structure; outputting said recommended corrective action; 23. A system comprising at least one processor configured to:

243. 1. A computer-implemented method for analyzing fluid leaks during surgery, comprising: receiving real-time endoluminal video of the surgical procedure; Analyzing frames of the endoluminal video to determine abnormal fluid leak conditions within the endoluminal video. and developing a corrective action when the abnormal fluid leakage condition is determined; 23. A computer-implemented method comprising:

244. 244. The method of claim 243, wherein the fluid comprises at least one of blood, bile, or urine. method.

245. The analyzing step includes analyzing the frames of the endoluminal video to identify blood splatter and and at least one characteristic of the liquid splash, and selecting the corrective action includes:

244. The method of claim 243, wherein the method is dependent on the at least one characteristic of the identified blood spatter. Law.

246. 246. The method of claim 245, wherein the at least one characteristic is associated with a source of blood spatter. Law.

247. 246. The method of claim 245, wherein the at least one characteristic is associated with blood splatter strength. How to.

248. 246. The method of claim 245, wherein the at least one characteristic is associated with an amount of blood splatter. method.

249. The step of analyzing the frames of the endoluminal video includes characterizing the abnormal fluid leak condition. wherein the selection of the corrective action depends on the determined characteristic. The method of claim 243.

250. 250. The method of claim 249, wherein the characteristic is associated with an amount of fluid leakage.

251. 250. The method of claim 249, wherein the characteristic is associated with a color of the fluid leak.

252. 250. The method of claim 249, wherein the characteristic is associated with a type of fluid associated with the fluid leak. The method described.

253. 250. The method of claim 249, wherein the characteristic is associated with a fluid leakage rate.

254. The method further comprises the steps of storing the endoluminal video and determining the abnormal leak condition. analyzing previous frames of the stored endoluminal video to determine the source of the leak. The method of claim 243, further comprising:

255. 3. The method of claim 2, wherein the step of developing corrective action includes providing notification of the source of the leak.

43. The method according to claim 43.

256. and determining the source of the leak includes identifying a ruptured anatomical organ. The method of claim 255.

257. The method further includes determining a flow rate associated with the fluid leak condition; 244. The method of claim 243, wherein the step of formulating the corrective action is based on the flow rate.

258. The method further comprises determining an amount of fluid loss associated with the fluid leak condition. and wherein said developing corrective action is based on said amount of fluid loss. The method described.

259. Analyzing frames of the endoluminal video to determine abnormal fluid leakage conditions in the endoluminal video. The step includes determining whether the determined fluid leak condition is an abnormal fluid leak condition. said method comprising: In response to determining that the determined fluid leak condition is an abnormal fluid leak condition, formulating corrective measures; In response to determining that the determined fluid leak condition is a normal fluid leak condition, Steps to avoid developing corrective measures; The method of claim 243, further comprising:

260. the endoluminal video depicts a surgical robot performing the surgical procedure, and the corrective action is 244. The method of claim 243, comprising sending commands to the robot.

261. 1. A surgical system for analyzing a fluid leak, comprising: Receive real-time intraluminal video of the surgical procedure; Analyzing frames of the endoluminal video to determine abnormal fluid leak conditions within the endoluminal video. death, developing a corrective action when the abnormal fluid leakage condition is determined; 16. A surgical system comprising: at least one processor configured to:

262. A dynamic system that, when executed by at least one processor, enables fluid leak detection. a non-transitory computer readable medium comprising instructions for causing said at least one processor to execute an operation. a body, the action comprising: receiving real-time endoluminal video of the surgical procedure; Analyzing frames of the endoluminal video to determine abnormal fluid leak conditions within the endoluminal video. and developing a corrective action when the abnormal fluid leakage condition is determined; 13. A non-transitory computer readable medium comprising:

263. 1. A computer-implemented method for predicting post-discharge risk, comprising: Accessing frames of video captured during a particular surgical procedure on a patient. P and accessing stored historical data identifying intraoperative events and associated outcomes; 、 Analyzing the accessed frames and obtaining information from the history data and identifying at least one particular intraoperative event in the accessed frames based on the and Based on information obtained from the historical data and the identified at least one intraoperative event. determining a predicted outcome associated with the particular surgical procedure based on the predicted outcome; outputting the predicted outcome in a manner that associates the predicted outcome with the patient; 23. A computer-implemented method comprising:

264. The step of identifying at least one particular intraoperative event comprises: a detected surgical instrument in said accessed frame; a detected anatomical structure in said accessed frame; , an interaction in the accessed frame between a surgical instrument and an anatomical structure, or At least one of the abnormal fluid leakage conditions detected in the accessed frames is The method of claim 263, based on

265. Identifying the at least one particular intraoperative event in the accessed frame. A machine learning model is used to The method of claim 263, wherein the method is trained using the

266. The step of determining a predicted outcome may be based on characteristics of the patient, electronic medical records, or post-operative 264. The method of claim 263 based on at least one of the reports.

267. The machine learning model associates the prediction with the particular surgical procedure based on intraoperative events. The machine learning model is trained using training examples to determine a clinical outcome. The method of claim 263, wherein the

268. The step of determining a predicted outcome includes using the trained machine learning model to: A method for predicting the surgical outcome based on the identified intraoperative events and the identified characteristics of the patient. The method of claim 267, further comprising the steps of:

269. The method includes receiving information identifying an achieved surgical outcome following the surgical procedure. and training the machine learning model using the received information. and updating the machine learning model according to the method of claim 267.

270. The method further includes identifying a characteristic of the patient, and the predicted outcome is determined based on the identified characteristic. The method of claim 263, further determined based on identified patient characteristics.

271. 271. The method of claim 270, wherein the patient characteristics are derived from electronic medical records.

272. The step of identifying the patient characteristics includes using a machine learning model to analyze the accessed field. the machine learning model includes analyzing a frame based on a historical surgical procedure history; The system is trained to identify patient characteristics using the training examples and corresponding historical patient characteristics. The method of claim 270, wherein

273. The predicted outcomes are post-discharge events, post-discharge adverse events, post-discharge complications, or readmission.

264. The method of claim 263, further comprising at least one of: estimating risk.

274. accessing a data structure containing a recommended sequence of surgical events; The step of identifying at least one particular intraoperative event includes identifying in the data structure A recommended sequence of events for the surgical procedure and a sequence of events for the accessed frame. The method of claim 263, based on identifying deviations between an actual sequence of events detected in a Law.

275. The identification of the deviation may be based on the detected surgical instrument in the accessed frame, Detected anatomical structures in the accessed frames or the relationship between surgical instruments and anatomical structures based on at least one of the interactions within the accessed frames between 274. The method according to claim 274.

276. The identification of the deviation may be based on historical surgical video footage, historical recommended courses of events, and and identifying deviations from the historical recommended sequence of events in the historical video footage. trained to identify deviations from a recommended course of events based on 275. The method of claim 274, comprising using a machine learning model.

277. The step of identifying deviations comprises comparing the accessed frames with the recommended sequence.

275. The method of claim 274, comprising the step of comparing the event to a reference frame that depicts the event. 。

278. The step of outputting the predicted outcome may include updating an electronic medical record associated with the patient.

264. The method of claim 263, comprising the step of:

279. Outputting the predicted outcome may include associating the predicted outcome with a health care provider.

264. The method of claim 263, comprising the step of transmitting to a data receiving device.

280. The method may further include improving the predicted outcome based on the accessed frames. determining at least one movement that is high; and and providing a recommendation based on the result.

281. A system for predicting post-discharge risk, comprising: accessing frames of video captured during a particular surgical procedure on a patient; Accessing stored historical data identifying intraoperative events and associated outcomes; Analyzing the accessed frames and obtaining information from the history data and identifying at least one particular intraoperative event in the accessed frames based on the Separately, Based on information obtained from the historical data and the identified at least one intraoperative event. determining a predicted outcome associated with the particular surgical procedure based on the outputting the predicted outcome in a manner that associates the predicted outcome with the patient; 23. A system comprising at least one processor configured to:

282. When executed by at least one processor, the method enables prediction of post-discharge risk. instructions for causing said at least one processor to perform the operations of: A readable medium, the operation comprising: Accessing frames of video captured during a particular surgical procedure on a patient. P and accessing stored historical data identifying intraoperative events and associated outcomes; 、 Analyzing the accessed frames and obtaining information from the history data and identifying at least one particular intraoperative event in the accessed frames based on the and Based on information obtained from the historical data and the identified at least one intraoperative event. determining a predicted outcome associated with the particular surgical procedure based on the predicted outcome; outputting the predicted outcome in a manner that associates the predicted outcome with the patient; 13. A non-transitory computer readable medium comprising:

Images