Systems and methods for surgical data review

Abstract

Various embodiments disclosed relate to systems and methods for processing surgical data to facilitate further downstream operations. For example, some embodiments can include a machine learning system trained to identify whether video from a surgical visualization tool, such as an endoscope, depicts a field of view inside or outside a patient’s body. The system can crop or whiten video frames that appear outside the patient to remove potentially leaking personal information, such as the identity of surgical team members, the identity of the patient, the configuration of the surgical room, etc. Proper removal of such non-surgical data can facilitate downstream processing, for example, by complying with regulatory requirements and by removing irrelevant data that can be detrimental to further downstream processing, such as training downstream classifiers.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 116,906, filed November 24, 2020, entitled “SYSTEMS AND METHODS FOR SURGICAL DATA CENSORSHIP,” the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] Various disclosed embodiments relate to systems and methods for removing non-surgical information from surgical data. Background Technology

[0004] Recent advances in data processing technologies, such as new deep learning systems, have enabled numerous new applications and improvements across various disciplines, including finance, advertising, and product management. The success of these technologies often hinges on the availability of large amounts of data, for example, for training and optimizing machine learning systems. With the increasing integration of data acquisition sensors (such as those in surgical robot systems and laparoscopic tools) into the operating room, corresponding improvements in surgical systems, methods, and outcomes can be optimistically anticipated.

[0005] Unfortunately, unlike stock market or warehouse inventory data, surgical data is often subject to considerable regulatory, commercial, and cultural constraints. For example, requirements under the Health Insurance Portability and Accountability Act (HIPAA) restrict the time, place, and manner in which patient data can be distributed and used. Hospitals and physicians may also be reluctant to share data that could involve personal liability or privacy issues. In addition to these already significant challenges, variations in sensor availability and data acquisition capabilities between different operating rooms can further complicate the uniform acquisition of surgical data. This uniform acquisition can be particularly important if one wants to avoid unwanted overfitting or bias in subsequent data processing. Such challenges can be even more severe and disruptive when data is acquired, analyzed, and acted upon in real time during surgical procedures. Unacceptable delays in the processing pipeline can impact downstream analytics and, in fact, may even completely halt certain downstream operations.

[0006] Therefore, there is a need for improved surgical data acquisition systems and methods that, despite these many challenges, can still acquire useful data. Attached Figure Description

[0007] The various embodiments described herein can be better understood by referring to the following specific embodiments in conjunction with the accompanying drawings, wherein the same reference numerals indicate the same or similarly functional elements:

[0008] Figure 1A These are schematic views of various elements that appear in the operating room during a surgical procedure, as described in some embodiments.

[0009] Figure 1B These are schematic views of various components that appear in the operating room during a surgical procedure employing a surgical robot, as can be seen in some embodiments.

[0010] Figure 2A It is a schematic Euler diagram depicting the general grouping of machine learning models and methodologies;

[0011] Figure 2B It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example unsupervised learning method;

[0012] Figure 2C It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example supervised learning method;

[0013] Figure 2D It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example semi-supervised learning method;

[0014] Figure 2E It is a description based on Figure 2A A schematic diagram illustrating various operations of reinforcement learning methods, using examples of conventional partitioning;

[0015] Figure 2F It is a schematic diagram depicting the relationship between machine learning models, machine learning model architectures, machine learning methodologies, machine learning methods, and machine learning implementation methods;

[0016] Figure 3A It is a schematic depiction of the operation of various aspects of an example support vector machine (SVM) machine learning model architecture;

[0017] Figure 3B It is a schematic depiction of various aspects of the operation of an example random forest machine learning model architecture;

[0018] Figure 3C It is a schematic depiction of various aspects of the operation of an example neural network machine learning model architecture;

[0019] Figure 3D yes Figure 3C A schematic depiction of the possible relationships between inputs and outputs in nodes of an example neural network architecture;

[0020] Figure 3EIt is a schematic depiction of example input-output relationship changes that can occur in a Bayesian neural network;

[0021] Figure 3F It is a schematic depiction of various aspects of the operation of an example deep learning architecture;

[0022] Figure 3G It is a schematic depiction of various aspects of the operation of the example integrated architecture;

[0023] Figure 3H It is a schematic block diagram depicting the various operations of the example pipeline architecture;

[0024] Figure 4A It is a schematic flowchart depicting the various operations common to different machine learning model training methods;

[0025] Figure 4B It is a schematic flowchart depicting the various operations common to inference methods of various machine learning models;

[0026] Figure 4C It is a schematic flowchart depicting the various iterative training operations that occur in box 405b in some architectures and training methods.

[0027] Figure 4D It is a schematic block diagram depicting the operation of various machine learning methods where there is no strict distinction between training and inference methods;

[0028] Figure 4E It is a schematic diagram illustrating the relationship between the architecture training method and the inference method;

[0029] Figure 4F It is a schematic block diagram depicting an example relationship between training and inference methods for machine learning models, where the training methods include various data subset operations;

[0030] Figure 4G It is a schematic block diagram depicting an example of decomposing training data into training subsets, validation subsets, and test subsets;

[0031] Figure 4H It is a schematic block diagram depicting the various operations in the training method incorporated into transfer learning;

[0032] Figure 4I It is a schematic block diagram depicting the various operations in training methods incorporated into online learning;

[0033] Figure 4J It is a schematic block diagram depicting the various components in an example generative adversarial network method;

[0034] Figure 5AThese are schematic illustrations of surgical data that may be received at a processing system in some embodiments;

[0035] Figure 5B This is an example task table that can be used in conjunction with various disclosed embodiments;

[0036] Figure 6 The illustration shows what can happen in some embodiments. Figure 1A and Figure 1B A schematic diagram illustrating an example stage of visual tool operation in a surgical operating room;

[0037] Figure 7A This is a flowchart illustrating various operations in a surgical video processing method that may be implemented in some embodiments;

[0038] Figure 7B It is an illustrative time series of data labeled with visual tool state transition events that may occur in some embodiments;

[0039] Figure 8 The illustration shows a method that can be implemented in some embodiments. Figure 7A A schematic block diagram of the components of a video filtering system for surgical procedures operated by a specific method;

[0040] Figure 9A It is an illustrative integrated machine learning model topology diagram that depicts, as may be implemented in some embodiments, a model topology for determining whether a visualization tool video frame depicts a view inside or outside the patient's body.

[0041] Figure 9B This indicates that in some embodiments it may be possible to... Figure 9A An illustrative topology diagram showing the model architecture and features selected as examples of those used in the illustrative integration of machine learning model topologies;

[0042] Figure 10A This is a schematic diagram illustrating example feature generation and application to various intermediate machine learning models, as can be implemented in various embodiments;

[0043] Figure 10B This is a schematic diagram illustrating example feature generation and application to various intermediate machine learning models, as can be implemented in various embodiments;

[0044] Figure 10C This is a schematic diagram illustrating example feature generation and application to various intermediate machine learning models, as can be implemented in various embodiments;

[0045] Figure 10D This is a schematic diagram illustrating example feature generation and application to various intermediate machine learning models, as can be implemented in various embodiments;

[0046] Figure 10E This is a schematic diagram illustrating example feature generation and application to various intermediate machine learning models, as can be implemented in various embodiments;

[0047] Figure 11A It is a description used for training Figure 9B A schematic diagram of one or more intermediate classifiers in the topology of a machine learning model, representing components of a machine learning model.

[0048] Figure 11B It is a description used for training Figure 9B A schematic diagram of the components of the topological fusion model;

[0049] Figure 12A The illustration shows that, in some embodiments, it can be implemented to train one or more intermediate machine learning models (e.g., as shown in the figure). Figure 11A Flowcharts of various operations that appear in the process;

[0050] Figure 12B This is an illustration of the intermediate machine learning model training process (e.g., as can be seen in some embodiments). Figure 12A The flowchart of various operations applied at box 1205f;

[0051] Figure 12C This can be seen in some embodiments when according to Figure 12B The example training dataset decomposition applied during the training process;

[0052] Figure 13A This is a schematic diagram depicting the application of a smoothing window to a set of data classification values, as may occur in some embodiments;

[0053] Figure 13B It is described via, as can be in some embodiments, in Figure 13A A schematic diagram illustrating the classification and reordering process using the median filter in the window;

[0054] Figure 13C In some embodiments, this can be achieved by Figure 13D A schematic depiction of the in / out classification signal waveforms used to perform the process.

[0055] Figure 13D This is a flowchart illustrating various operations in a segmented surgical procedure that may be implemented in some embodiments;

[0056] Figure 14A Is it possible? Figure 14B and Figure 14C A table of abstract example classification results considered in the uncertainty calculation;

[0057] Figure 14B This is a flowchart illustrating various operations in a process of calculating uncertainty using class counting, as may be implemented in some embodiments.

[0058] Figure 14C This is a flowchart illustrating various operations in a process of calculating uncertainty using entropy, as may be implemented in some embodiments.

[0059] Figure 14D This is an illustrative depiction of uncertain results using a generative machine learning model, as may be employed in some embodiments;

[0060] Figure 15A This is a schematic diagram illustrating an example component deployment topology that may be implemented in some embodiments;

[0061] Figure 15B This is a schematic diagram illustrating an example component deployment topology that may be implemented in some embodiments;

[0062] Figure 15C This is a schematic diagram illustrating an example component deployment topology that may be implemented in some embodiments;

[0063] Figure 16A It is a table that describes the properties of two example datasets used in various example implementations of various embodiments for evaluating various examples;

[0064] Figure 16B It is a graph comparing the F1 scores of results using different intermediate models based on specific features generated from the DaVinci16 dataset;

[0065] Figure 16C It is a graph comparing the F1 scores of results using different intermediate models based on specific features generated from the Cholec80 dataset;

[0066] Figure 17 It is an illustrative set of time series data obtained by exclusively considering various feature types in the example implementation;

[0067] Figure 18A This is a table depicting the classification performance results of an example implementation trained on robotic / non-robotic data and applied to inference on non-robotic / robotic data;

[0068] Figure 18B This is a table depicting the cross-surgical classification performance of an example implementation that uses data acquired from a robotic surgical operating room and is trained on and applied to inferences from data acquired from a non-robotic surgical operating room (and vice versa);

[0069] Figure 18CIt is a graph comparing the classification performance F1 scores of implementations with and without the smoothing method described herein; and

[0070] Figure 19 This is a block diagram of an example computer system that can be used in conjunction with some embodiments.

[0071] The specific examples depicted in the accompanying drawings have been chosen for ease of understanding. Therefore, the disclosed embodiments should not be limited to the specific details in the drawings or the corresponding disclosure. For example, the drawings may not be drawn to scale, the dimensions of certain elements may have been adjusted for ease of understanding, and the operations of embodiments associated with flowcharts may include additional, alternative, or fewer operations compared to those depicted herein. Consequently, some components and / or operations may be separated into different blocks or combined into a single block in a manner different from that depicted. The embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosed examples, and not to limit the embodiments to the specific examples described or depicted. Detailed Implementation

[0072] Example of an operating room overview

[0073] Figure 1A This is a schematic view of various elements appearing in the surgical room 100a during a surgical procedure, as can be seen in some embodiments. In particular, Figure 1A A non-robotic surgical operating room 100a is depicted, in which a patient-side surgeon 105a performs procedures on a patient 120 with the assistance of one or more assistants 105b, who may be surgeons, physician assistants, nurses, technicians, etc. The surgeon 105a may use a variety of tools to perform the procedures, such as visualization tools 110b (e.g., laparoscopic ultrasound or endoscope) and mechanical end effectors 110a (e.g., scissors, retractors, dissecters), etc.

[0074] The visualization tool 110b provides the surgeon 105a with an internal view of the patient 120, for example, by displaying visualization output from a camera mechanically and electrically coupled to the visualization tool 110b. The surgeon can view the visualization output, for example, through an eyepiece coupled to the visualization tool 110b or on a display 125 configured to receive the visualization output. For example, in the case where the visualization tool 110b is an endoscope, the visualization output can be a color or grayscale image. The display 125 can allow the assistant 105b to monitor the progress of the surgeon 105a during surgery. The visualization output from the visualization tool 110b can be recorded and stored for future review, for example, by capturing the visualization output in parallel with it being provided to the display 125 using hardware or software on the visualization tool 110b itself, or by capturing the output from the display 125 as soon as it appears on the screen, etc. While this document can broadly discuss two-dimensional video capture using visualization tool 110b, when visualization tool 110b is an endoscope, it will be understood that in some embodiments, visualization tool 110b can capture depth data in lieu of or append to two-dimensional image data (e.g., using a laser rangefinder, stereo vision, etc.). Therefore, it will be understood that when such three-dimensional depth data is available, it is possible to apply the two-dimensional operations discussed herein (with necessary modifications) to such three-dimensional depth data. For example, the input to a machine learning model can be extended or modified to accept features derived from such depth data.

[0075] A single surgical procedure may include the execution of several sets of actions, each set of actions forming a discrete unit referred to herein as a task. For example, locating a tumor may constitute a first task, resecting the tumor may constitute a second task, and closing the surgical site may constitute a third task. Each task may include multiple actions; for example, a tumor resection task may require several cutting actions and several cauterizing actions. While some surgical procedures require tasks to assume a specific order (e.g., resection occurs before closure), in some surgical procedures the order and presence of some tasks may allow for variation (e.g., eliminating prophylactic tasks or reordering resection tasks where the order is invalid). Transitions between tasks may require surgeon 105a to remove tools from the patient, replace tools with different tools, or introduce new tools. Some tasks may require removing visualization tools 110b and repositioning visualization tools 110b relative to their position in a previous task. While some assistants 105b may assist in performing surgically related tasks, such as administering anesthesia 115 to patient 120, assistants 105b may also assist in these task transitions, for example, anticipating the need for new tools 110c.

[0076] Technological advancements have enabled, for example Figure 1AThe described program can also be executed by a robotic system, as well as programs that cannot be executed in a non-robotic operating room 100a. Specifically, Figure 1B This is as can be seen with respect to some embodiments occurring in the use of surgical robots (such as da Vinci). TM A schematic view of various components present in the operating room 100b during a surgical procedure using a surgical system. Here, a patient-side trolley 130, having tools 140a, 140b, 140c, and 140d respectively attached to each of a plurality of arms 135a, 135b, 135c, and 135d, can occupy the position of the patient-side surgeon 105a. As previously described, tools 140a, 140b, 140c, and 140d may include visualization tools 140d, such as endoscopes, laparoscopic ultrasound, etc. The operator 105c, who may be a surgeon, can view the output of the visualization tools 140d via a display 160a on a surgeon's console 155. By manipulating a handheld input mechanism 160b and a pedal 160c, the operator 105c can remotely communicate with the tools 140a-140d on the patient-side trolley 130 to perform surgical procedures on the patient 120. In fact, in some embodiments, because communication between the surgeon's console 155 and the patient-side cart 130 can occur across a remote communication network, the operator 105c may or may not be in the same physical location as the patient-side cart 130 and the patient 120. The electronics / console 145 may also include a display 150 that depicts the patient's vital signs and / or the output of the visualization tool 140d.

[0077] Similar to the task shifts in non-robotic surgical operating room 100a, surgical procedures in operating room 100b may require the removal or replacement of tools 140a-140d (including visualization tool 140d) for various tasks, as well as the introduction of new tools, such as new tool 165. As previously stated, as the surgical procedure progresses, one or more assistants 105d can now anticipate such changes and make any necessary adjustments together with the operator 105c.

[0078] Similar to the non-robotic surgical operating room 100a, outputs from visualization tools 140d can be recorded here, such as from the patient-side trolley 130, the surgeon's console 155, and the monitor 150. While some tools 110a, 110b, and 110c in the non-robotic surgical operating room 100a can record additional data, such as temperature, motion, conductivity, and energy levels, the presence of the surgeon's console 155 and the patient-side trolley 130 in operating room 100b facilitates the recording of far more data than just the outputs from visualization tools 140d. For example, operator manipulation of the handheld input mechanism 160b by the operator 105c, activation of the pedal 160c, and eye movements within the monitor 160a can all be recorded. Similarly, the patient-side trolley 130 can record tool activations (e.g., application of radiant energy, closure of scissors, etc.) and end effector movements throughout the surgical procedure.

[0079] Machine Learning Fundamentals - Overview

[0080] This section provides a basic description of the machine learning model architectures and methods that may be associated with the various embodiments disclosed. Machine learning encompasses a broad, heterogeneous field and has undergone many abrupt and overlapping developments. Given this complexity, practitioners do not always use terminology consistently or with strict clarity. Therefore, this section aims to provide common ground to better ensure that the reader understands the substance of the disclosed embodiments. It will be understood that it is not feasible to exhaustively address all known possible variations of all known machine learning models and their architectures, tasks, methods, and methodologies. Rather, it will be understood that the examples discussed herein are merely representative and that the various embodiments disclosed may employ many other architectures and methods different from those explicitly discussed.

[0081] In order to orient readers relative to existing literature, Figure 2A The commonly accepted groupings of machine learning models and methodologies (also known as techniques) are depicted in the form of a schematic Euler diagram. In contrast to... Figure 2F Before providing a more comprehensive description of the field of machine learning, we will refer to... Figures 2B to 2E Describe them in their conventional way Figure 2A Grouping is used to orient readers.

[0082] Figure 2A The common grouping of machine learning models and their methodologies is usually based on the nature of the inputs the model is expected to receive or the methodology on which the model is expected to operate. Unsupervised learning methodologies derive inferences from input datasets lacking output metadata (also known as "unlabeled data"), or by ignoring such metadata if it exists. For example, as in Figure 2BAs shown, the unsupervised K-Nearest Neighbors (KNN) model architecture can receive multiple unlabeled inputs (represented by circles in feature space 205a). The feature space is the mathematical space of the inputs, on which a given model architecture is configured to operate. For example, if a 128x128 grayscale pixel image is provided as input to a KNN, it can be viewed as a linear array of 16,384 "features" (i.e., the original pixel values). Then the feature space would be a 16,384-dimensional space (…). Figure 2B (Only a two-dimensional space is shown for ease of understanding). Conversely, if a Fourier transform is applied to pixel data, for example, the resulting frequency amplitude and phase can be input as "features" into the model architecture. Although input values ​​in feature space are sometimes referred to as feature "vectors," it will be understood that not all model architectures are expected to receive feature inputs in a linear form (e.g., some deep learning networks expect input features to be matrices or tensors). Therefore, unless otherwise indicated by the context, references to feature vectors, feature matrices, etc., should be considered as examples of possible forms that can be input into the model architecture. Similarly, references to "input" will be understood to include any possible type or form of features acceptable to the architecture. Continuing Figure 2B For example, a KNN classifier can output the association between an input vector and various groups determined by the KNN classifier, as indicated by the squares, triangles, and hexagons in the figure. Therefore, unsupervised methodologies can include, for example, determining clusters in the data, reducing or changing the feature dimensions used to represent the data input, as in this example.

[0083] Supervised learning models receive an input dataset along with output metadata (called "labeled data") and modify the parameters of the model architecture (such as the biases and weights of a neural network, or the support vectors of an SVM) based on this input data and metadata to better map subsequent inputs to the desired output. For example, an SVM supervised classifier can be like... Figure 2C The operation is as shown, receiving multiple input feature vectors, represented by circles, as training inputs in feature space 210a, where each feature vector is accompanied by an output label A, B, or C, for example, as provided by the practitioner. According to supervised learning methodologies, the SVM uses these labeled inputs to modify its parameters such that when the SVM receives a new, previously unseen input 210c in the form of feature vectors from feature space 210a, the SVM can output a determined desired classification "C" in its output. Therefore, supervised learning methodologies can include, for example, performing classification, performing regression, etc., as in this example.

[0084] Semi-supervised learning methodologies rely on both labeled and unlabeled data to inform the parameter adjustments of their model architecture. For example, supervised neural network classifiers can... Figure 2DThe operation is illustrated by receiving some training input feature vectors labeled with class A, B, or C in feature space 215a, and some training input feature vectors without such labels (e.g., depicted by circles lacking letters). Without considering the unlabeled inputs, a naive supervised classifier can distinguish inputs in class B and class C based on a simple planar separation 215d in the feature space between the available labeled inputs. However, by considering both unlabeled and labeled input feature vectors, a semi-supervised classifier can employ a more subtle separation 215e. Unlike the simple separation 215d, the subtle separation 215e can correctly classify a new input 215c as class C. Therefore, semi-supervised learning methods and architectures can include applications of both supervised and unsupervised learning, where at least some of the available data is labeled.

[0085] at last, Figure 2A The conventional grouping of reinforcement learning methodologies distinguishes them into those in which an agent (e.g., a robot or digital assistant) takes actions (e.g., moving a manipulator, making suggestions to the user) that influence the agent's environmental context (e.g., the object's position in the environment, the user's preferences, etc.) to precipitate a new environmental state and some associated environment-based rewards (e.g., a positive reward if the environmental object is now closer to the target state, a negative reward if the user is unhappy, etc.). Therefore, reinforcement learning can include, for example, updating digital assistants (based on user behavior and expressed preferences), autonomous robots maneuvering in factories, and computers playing chess.

[0086] As mentioned earlier, although many practitioners will recognize Figure 2A The conventional classification method, but Figure 2A The grouping of these groups obscures the rich diversity of machine learning and may not adequately represent machine learning architectures and techniques that fall into multiple groups or fall entirely outside of them (e.g., random forests and neural networks can be used for both supervised and unsupervised learning tasks; similarly, some generative adversarial networks, while employing supervised classifiers, do not easily fall into any of these groups). Figure 2A (within any group). Therefore, although this article can refer to Figure 2A Various terms are used to facilitate the reader's understanding, but this description should not be limited to... Figure 2A A convention that demands uniformity. For example, Figure 2F A more flexible machine learning classification method is proposed.

[0087] In particular, Figure 2FMachine learning can be approximated as including model 220a, model architecture 220b, methodology 220e, method 220d, and implementation 220c. At a high level, model architecture 220b can be considered as species of their corresponding genus model 220a (model A has possible architectures A1, A2, etc.; model B has possible architectures B1, B2, etc.). Model 220a refers to a description of a mathematical structure suitable for implementation as a machine learning architecture. For example, KNN, neural networks, SVM, Bayesian classifiers, principal component analysis (PCA), etc., represented by boxes “A”, “B”, “C”, etc., are examples of models (ellipses in the figure indicate additional items). While models can specify general computational relationships, such as SVM including hyperplanes, neural networks having layers or neurons, etc., models can also omit specific architecture structures, such as the architecture's choice of hyperparameters and data flows to perform a specific task, such as SVM using radial basis function (RBF) kernels, neural networks being configured to receive inputs of dimension 256x256x3, etc. These structural features can be selected by practitioners or derived from the training or configuration process, for example. Note that the domain of model 220a also includes combinations of its members, for example, when creating an ensemble model (see below). Figure 3G (Discussion) or when using a model pipeline (below about) Figure 3H discuss).

[0088] For clarity, it will be understood that many architectures include both parameters and hyperparameters. Architectural parameters refer to the configuration values ​​of the architecture, which can be adjusted directly based on the received input data (such as adjusting the weights and biases of a neural network during training). Different architectures can have different parameter choices and relationships between them, but changes to parameter values ​​(e.g., during training) are not considered changes to the architecture. In contrast, architectural hyperparameters refer to the configuration values ​​of the architecture that are not directly adjusted based on the received input data (e.g., the K neighbors in a KNN implementation, the learning rate in a neural network training implementation, the kernel type of an SVM, etc.). Therefore, changing hyperparameters generally changes the architecture. It will be understood that some methodological operations (such as validation discussed below) can adjust hyperparameters during training, and thus adjust the architecture type. Therefore, some implementations can consider multiple architectures, although only some of these can be configured to be used or used at a given time.

[0089] Similar to models and architectures, at a higher level, methods 220d can be considered species belonging to the genus Methodology 220e (Methodology I has methods I.1, I.2, etc.; Methodology II has methods II.1, II.2, etc.). Methodology 220e refers to algorithms adapted as methods for performing tasks using one or more specific machine learning architectures, such as training architectures, testing architectures, validating architectures, performing architecture inference, using multiple architectures in generative adversarial networks (GANs), etc. For example, gradient descent is a methodology describing methods for training neural networks, ensemble learning is a methodology describing methods for training groups of architectures, etc. While a methodology can specify general algorithmic operations, such as gradient descent taking traversal steps along a cost or error surface, ensemble learning considering intermediate results of its architecture, etc., a method specifies how a particular architecture should execute the algorithm of the methodology; for example, gradient descent takes traversal backpropagation on a neural network and performs stochastic optimization via Adam with specific hyperparameters, ensemble systems include random forest ensembles applying AdaBoost with specific configuration values, training data being organized into a specific number of folds, etc. When people use additional or modified features to enhance existing architectures or methods (e.g., GAN architectures and GAN training methods might be considered as encompassing deep learning architectures and deep learning training methods), they will understand that architectures and methods themselves can have sub-architectures and sub-methods. They will also realize that not all possible methodologies will apply to all possible models (e.g., suggesting gradient descent on a PCA architecture without further explanation seems absurd). They will understand that methods can include some actions of the practitioner or can be fully automated.

[0090] As the examples above demonstrate, when one moves from models to architectures and from methodologies to methods, aspects of architecture can appear in methods, and aspects of methods can appear in architectures, because some methods may only be applicable to certain architectures, and some architectures may only be suitable for certain methods. Recognizing this interaction, implementation 220c is a combination of one or more architectures with one or more methods to form a machine learning system configured to perform one or more specific tasks, such as training, inference, generating new data using GANs, etc. For clarity, the architecture of this implementation does not need to actively execute its methods, but can simply be configured to execute methods (e.g., as when accompanying training control software is configured to pass input through the architecture). Applying the method will result in the execution of tasks, such as training or inference. Therefore, Figure 2FThe hypothetical implementation A (indicated by "Imp.A") described herein comprises a single architecture and a single method. This could correspond, for example, to an SVM architecture configured to identify objects in a 128x128 grayscale image using a hyperplane support vector separation method employing an RBF kernel in a 16,384-dimensional space. The use of the RBF kernel and the choice of feature vector input structure reflect both the choice of architecture and the choice of training and inference methods. Thus, it will be understood that some description of the architecture structure can imply aspects of the corresponding method, and vice versa. Hypothetical implementation B (indicated by "Imp.B") could correspond, for example, to training method II.1, which can switch between architectures B1 and C1 based on validation results before applying inference method III.3.

[0091] The close relationship between the architecture and methods within the implementation method facilitates Figure 2A Many ambiguities arise because these groups do not easily capture the close relationship between methods and architectures in a given implementation. For example, very minor changes in a method or architecture can lead to ambiguity. Figure 2A The inter-group movement model implementation involves, for example, a practitioner training a random forest using a first method that incorporates labeled (supervised) data, and then applying a second method using the trained architecture to detect clusters in unlabeled (unsupervised) data, rather than performing inference on the data. Similarly, Figure 2A Grouping can make it difficult to categorize aggregation methods and architectures, for example, as shown below regarding... Figure 3F and Figure 3G The aggregation methods and architectures discussed can be applied to Figure 2A Some, none, or all of the techniques were found in the groups. Therefore, the following sections refer to... Figures 3A to 3G and Figures 4A to 4J The relationships between various example model architectures and example methods are discussed to promote clarity and reader identification of the relationships between architectures, methods, and implementations. It will be understood that the tasks discussed are exemplary, and references to, for example, classification operations are made to facilitate understanding; this should not be construed as suggesting that the implementation must be specifically designed for this purpose.

[0092] For clarity, it will be understood that, without explicit linguistic indication, the above is relative to... Figure 2FThe explanations provided are merely intended to aid the reader's understanding and therefore should not be interpreted in a restrictive manner. For example, one will naturally understand that "method" 220d is a computer-implemented method, but not all computer-implemented methods are methods in the sense of "method" 220d. A computer-implemented method can be logic without any machine learning functionality. Similarly, the term "methodology" is not always used in the sense of "methodology" 220e, but can refer to a method without machine learning functionality. Likewise, while the terms "model," "architecture," and "implementation method" are used in 220a, 220b, and 220c above, in the absence of such an effect in the language, these terms are not limited to their meanings in the sense of "methodology" 220d. Figure 2F The difference is that it can be used to refer to the topology of machine learning components.

[0093] Machine Learning Fundamentals - Example Implementation

[0094] Figure 3A This is a schematic depiction of the operation of an example SVM machine learning model architecture. At a high level, given data from two classes (e.g., images of dogs and images of cats) as input features, in... Figure 3A The diagram, represented by circles and triangles, shows that the SVM seeks to determine a hyperplane separator 305a that maximizes the minimum distance from each class member to the separator 305a. Therefore, the training feature vector 305f has a minimum distance 305e to the separator 305a across all its peers. Conversely, the training feature vector 305g has a minimum distance 305h to the separator 305a across all its peers. Thus, the margin 305d formed between these two training feature vectors is a combination of distances 305h and 305e (reference lines 305b and 305c are provided for clarity), and as a maximal-minimum separator, it identifies the training feature vectors 305f and 305g as support vectors. While this example describes linear hyperplane separation, different SVM architectures accommodate different kernels (e.g., RBF kernels), which can facilitate non-linear hyperplane separation. The separator can be found during training, and subsequent inference can be achieved by considering the position of new inputs in the feature space relative to the separator. Similarly, while this example depicts two-dimensional feature vectors (in a two-dimensional plane of paper) for clarity, one will understand that the architecture may accept features with more dimensions (e.g., a 128x128 pixel image might have 16,384 dimensions as input). Although the hyperplane in this example separates only two classes, multi-class separation can be achieved in various ways, such as using an integrated architecture with SVM hyperplane separation in one-to-one, one-to-all, and other configurations. Practitioners often use LIBSVM when implementing SVM. TM and scikit-learn TMThe library. People will understand many different machine learning models (such as logistic regression classifiers) that seek to identify the separating hyperplane.

[0095] In the example SVM implementation above, practitioners determine the feature format as part of the implementation's architecture and methodology. For some tasks, it may be desirable to process the input to determine the new or different feature formats themselves. In fact, some random forest implementations can adjust the feature space representation in this way. For example, Figure 3B An example random forest model architecture comprising multiple decision trees 310b is depicted at a high level, each decision tree receiving all or part of the input feature vector 310a at its root node. Although three trees are shown in this example architecture, with a maximum depth of three levels, it will be understood that forest architectures with fewer or more trees and different levels (even between trees within the same forest) are possible. When each tree considers its input portion, it refers all or part of the input to subsequent nodes, such as path 310f, based on whether the input portion satisfies or does not satisfy conditions associated with various nodes. For example, when considering an image, a single node in a tree can query whether the pixel value at a certain point in the feature vector is above or below a certain threshold. In addition to the threshold parameter, some trees may also contain additional parameters, and their leaves may contain the probability of correct classification. Each leaf of a tree may be associated with a provisional output value 310c for consideration by a voting mechanism 310d to produce a final output 310e, for example, by majority voting in the trees or by a probability-weighted average of the predictions from each tree. This architecture can be applied to various training methods, such as training different subsets of data on different trees.

[0096] The tree depth and variety of trees in a random forest can facilitate the model's consideration of feature relationships rather than directly comparing features in the initial input. For example, if the initial features are pixel values, the tree can identify relationships between task-related groups of pixel values, such as the relationship between "nose" and "ear" pixels in cat / dog classification. However, binary decision tree relationships may impose limitations on the ability to discern these "higher-order" features.

[0097] like Figure 3C The neural networks in the example architecture can also infer higher-order features and relationships between the initial input vectors. However, each node in the network can be associated with various parameters and connections to other nodes, facilitating more complex decision-making and intermediate feature generation than the binary relationships of a regular random forest tree. Figure 3CAs shown, a neural network architecture may include an input layer, at least one hidden layer, and an output layer. Each layer comprises a set of neurons that can receive multiple inputs and provide output values, also known as activation values. The final output value 315b of the output layer is used as the final result of the network. Similarly, the input 315a of the input layer can be received from input data instead of the previous neuron layers.

[0098] Figure 3D Depicting Figure 3C The input and output relationship at node 315c. Specifically, the output n of node 315c... out It can be related to its three (zero-based indexed) inputs as follows:

[0099]

[0100] Where w i These are the weight parameters for the output of the i-th node in the input layer, n. i is the output value of the activation function from the i-th node in the input layer, b is the bias value associated with node 315c, and A is the activation function associated with node 315c. Note that in this example, the sum is for each of the three input layer node output and weight pairs, and only a single bias value b is added. The activation function A can determine the node's output based on the weight value, bias value, and the value of the previous layer node. During training, each of the weight and bias parameters can be adjusted depending on the training method used. For example, many neural networks employ a methodology called backpropagation, where, in some forms, the weights and bias parameters are randomly initialized, the training input vector is passed through the network, and the difference between the network's output value and the expected output value of the vector's metadata is determined. This difference can then be used as a metric to adjust the network parameters, propagating the error as a correction throughout the network so that the network is more likely to produce the correct output when it encounters an input vector in the future. Although for clarity... Figure 3CThe example implementation shows three nodes in the input layer, but one will understand that more or fewer nodes may exist in different architectures (e.g., there might be 16,384 such nodes to receive pixel values ​​in the 128x128 grayscale image example above). Similarly, while each layer in this example architecture is shown as fully connected to the next layer, one will understand that other architectures may not connect every node between layers in this way. Not all neural network architectures process data entirely from left to right, nor do they consider only a single feature vector at a time. For example, recurrent neural networks (RNNs) encompass a class of neural network methods and architectures that consider previous input instances when considering the current instance. Architectures can be further differentiated based on the activation functions used at various nodes, such as: logistic functions, corrected linear unit functions (ReLU), softplus functions, etc. Therefore, there are considerable differences between architectures.

[0101] People will recognize that many of the example machine learning implementations discussed so far in this overview are “discriminative” machine learning models and methodologies (SVM, logistic regression classifiers, etc.). Figure 3D (e.g., neural networks with nodes in the network). Typically, a discriminative approach assumes a form that seeks the probability of finding the following equation 2:

[0102] P(output|input) (2)

[0103] In other words, these models and methodologies seek structural classification (e.g., SVM hyperplanes) and estimate the parameters associated with that structure based on training data (e.g., support vectors determine the separating hyperplane). However, it will be understood that not all models and methodologies discussed in this paper can assume this discriminative form; instead, one of several “generative” machine learning models and their corresponding methodologies (e.g., Naive Bayes classifiers, Hidden Markov Models, Bayesian Networks, etc.) can be assumed. These generative models instead assume a form that seeks the probability of finding the following Equation 3:

[0104] P(output), P(input|output) (3)

[0105] In other words, these models and methodologies seek to reflect the characteristic relationship between inputs and outputs (e.g., Bayesian neural networks, their initial parameters, and priors), estimate these parameters from the training data, and then compute the value of Equation 2 using Bayesian rules. It will be understood that directly performing these computations is not always feasible, and therefore numerical approximation methods can be employed in some of these generative models and methodologies.

[0106] Readers will understand that such generative methods can be used in this paper (with necessary modifications) to achieve the results exhibited by discriminative implementation methods, and vice versa. For example, Figure 3E The illustration shows an example node 315d that can appear in a Bayesian neural network. Unlike node 315c, which simply receives numerical values, it will be understood that nodes in a Bayesian neural network (such as node 315d) can receive weighted probability distributions 315f, 315g, 315h (e.g., parameters of such distributions) and can output distribution 315e themselves. Therefore, it will be recognized that while classification uncertainty in a discriminative model can be determined, for example, via various post-processing techniques (e.g., comparing the output of a discarded ergonomic application with that of a discriminative neural network), a similar measurement of uncertainty can be achieved by taking the output probability distribution of a generative model, for example, by considering the variance of distribution 315e. Therefore, just as references to a particular machine learning implementation herein are not intended to exclude any alternatives to similar functional implementations, references to discriminative implementations herein should not be construed as excluding alternatives to generative counterparts, where applicable.

[0107] Returning to the general discussion of machine learning methods, although Figure 3C An example neural network architecture with a single hidden layer is depicted, but many neural network architectures can have more than one hidden layer. Some networks with many hidden layers have produced surprisingly efficient results, and the term "deep learning" has been applied to these models to reflect the large number of hidden layers. Here, deep learning refers to the architecture and approach that employs at least one neural network architecture with more than one hidden layer.

[0108] Figure 3F This is a schematic depiction of the operation of an example deep learning model architecture. In this example, the architecture is configured to receive a two-dimensional input 320a, such as a grayscale image of a cat. When used for classification, as in this example, the architecture can typically be divided into two parts: a feature extraction part, which includes a series of layer operations; and a classification part, which determines the output value based on the relationships between the extracted features.

[0109] Many different feature extraction layers are possible, such as convolutional layers, max-pooling layers, dropout layers, cropping layers, etc., and many of these layers are themselves susceptible to variations (e.g., 2D convolutional layers, 3D convolutional layers, convolutional layers with different activation functions, etc.) and different methods and methodologies used for network training, inference, etc. As shown in the figure, these layers can produce multiple intermediate values ​​320b-320j of different dimensions, and these intermediate values ​​can be processed along multiple paths. For example, initially, the grayscale image 320a can be represented as a feature input tensor of dimension 128x128x1 (e.g., a grayscale image 128 pixels wide and 128 pixels high) or a feature input tensor of dimension 128x128x3 (e.g., an RGB image 128 pixels wide and 128 pixels high). Multiple convolutions with different kernel functions in the first layer can contribute multiple intermediate values ​​320b from this input. These intermediate values ​​320b can themselves be considered by two different layers to form two new intermediate values ​​320c and 320d along separate paths (although two paths are shown in this example, one will understand that more paths or a single path is possible in different architectures). Additionally, when an image has red, green, and blue values ​​for each pixel, for example, the "x3" dimension in a 128x128x3 feature tensor (for clarity, this input has three "tensor" dimensions, but 49,152 individual "feature" dimensions), data can be provided in multiple "channels." Various architectures can operate on channels individually or collectively in various layers. The ellipses in the figure indicate the presence of additional layers (e.g., some networks have hundreds of layers). As shown, the size and dimension of intermediate values ​​can change, for example, after pooling, as in value 320e. In some networks, intermediate values ​​can be considered at layers between paths, as shown between intermediate values ​​320e, 320f, 320g, and 320h. Ultimately, the final feature set appears at intermediate sets 320i and 320j and is fed into a set of one or more classification layers 320k and 320l, for example, via flattening layers, SoftMax layers, fully connected layers, etc., to produce output values ​​320m at the output nodes of layer 320l. For example, if N classes are to be identified, there can be N output nodes reflecting the probability that each class is the correct class (e.g., here the network is identifying one of three classes and indicating the class "cat" as the most likely for a given input), although some architectures may have fewer or more outputs. Similarly, some architectures may accept additional inputs (e.g., some floodfill architectures utilize evolved mask structures that can be received as inputs other than the input feature data and produced as outputs in modified form in addition to the classification output values; similarly, some recurrent neural networks can store values ​​from one traversal so that they can be fed into subsequent traversals along with other inputs), and may include feedback loops, etc.

[0110] TensorFlow TM Caffe TM and Torch TM These are examples of common software library frameworks used to implement deep neural networks, although many architectures can be created "from scratch," simply representing layers as operations on value matrices or tensors, and representing data as values ​​in these matrices or tensors. Examples of deep learning network architectures include VGG-19, ResNet, Inception, DenseNet, and others.

[0111] Although the example machine learning architecture is already about Figures 3A to 3F Discussions were held, but many machine learning models and corresponding architectures are formed by combining, modifying operations and structures, or adding them to other architectures and technologies. For example, Figure 3G This is a schematic depiction of an ensemble machine learning architecture. Ensemble models encompass a wide variety of architectures, including, for example, "meta-algorithm" models that use multiple weak learning models to work together to form a stronger model, such as AdaBoost. Figure 3A Random forests can be seen as another example of such ensemble models, although random forests themselves can be intermediate classifiers in ensemble models. One or more intermediate classifiers include a random forest classifier configured to receive a set of linked features, which includes a first part associated with Hu moment features and a second part associated with Haralick image texture features.

[0112] exist Figure 3G In the example, the initial input feature vector 325a can be input, in whole or in part, to various model implementations 325b, which can be from the same or different models (e.g., SVM, neural networks, random forests, etc.). The outputs from these models 325c can then be received by a "fusion" model architecture 325d to generate the final output 325e. The fusion model implementation 325d itself can be the same or a different model type than one of implementations 325b. For example, in some systems, the fusion model implementation 325d can be a logistic regression classifier and the model 325b can be a neural network.

[0113] As people will understand, integrated model architecture can facilitate... Figures 3A to 3F The paradigm architecture offers greater flexibility, and one should realize that modifications to the architecture or its methodology (sometimes relatively minor) can foster novel behaviors that are not easily applied to other applications. Figure 2ARegular grouping. For example, PCA is often described as an unsupervised learning method and its corresponding architecture because it discriminates feature representations with reduced dimensionality from input data lacking labels. However, PCA is frequently used with labeled inputs to facilitate supervised classification, as described by M. Turk and A. Pentland in their application of EigenFaces, “Eigenfaces for Recognition”, J. Cognitive Neuroscience, Vol. 3, No. 1, 1991. Figure 3H An example of a machine learning pipeline topology for such modifications is depicted. Similar to EigenFaces, one can use unsupervised methods to determine feature representations in box 330a (e.g., using PCA to determine the principal components of each group of facial images associated with one of several people). As an unsupervised method, Figure 2A The standard grouping of data does not typically allow this PCA operation to be interpreted as "training". However, by transforming the input data (e.g., facial images) into a new representation (principal component feature space) in box 330b, one can create a data structure suitable for applications of subsequent inference methods.

[0114] For example, in box 330c, the new incoming feature vector (new face image) can be transformed into an unsupervised form (e.g., principal component feature space), and then a metric (e.g., the distance between the principal components of each person's face image group and the principal component representation of the new vector) or other subsequent classifier (e.g., SVM, etc.) can be applied in box 330d to classify the new input. Thus, model architectures (e.g., PCA) that are unsuitable for certain methodologies (e.g., metric-based training and inference) can be made so suitable through methodological or architectural modifications (such as pipelines). Similarly, one will understand that this pipeline is merely an example—by assigning the new inference input to the class of the group that has the first moment in the feature space closest to the inference input, Figure 2B The KNN unsupervised architecture and methods can be similarly used for supervised classification. Therefore, these pipeline approaches can be considered machine learning models in this paper, although they are not usually referred to as such.

[0115] Some architectures can be used with training methods, and then some of these trained architectures can be used with inference methods. However, it will be understood that not all inference methods perform classification, and not all trained models can be used for inference. Similarly, it will be understood that not all inference methods require the training method to be pre-applied to the architecture to process new inputs for a given task (e.g., when KNN generates classes by directly considering the input data). Regarding training methods, Figure 4AThis is a schematic flowchart depicting common operations in various training methods. Specifically, in box 405a, either directly by the practitioner or the architecture, training data can be assembled into one or more training input feature vectors. For example, a user could collect images of dogs and cats with metadata tags for a supervised learning method, or collect unlabeled stock prices over time for unsupervised clustering. As discussed, the raw data can be preprocessed to transform into feature vectors, or it can be used directly as features in their raw form.

[0116] In box 405b, training methods can adjust the parameters of the architecture based on training data. For example, the weights and biases of a neural network can be updated via backpropagation, and SVMs can select support vectors based on hyperplane computation, etc. One will understand, as regarding... Figure 3G The pipeline architecture discussed here is not the case; however, not all model architectures can update parameters within the architecture itself during "training." For example, in Eigenfaces, the determination of principal components of facial identity groups can be interpreted as the creation of new parameters (principal component feature space) rather than the adjustment of existing parameters (e.g., adjusting neural network architecture weights and biases). Therefore, in this paper, the determination of Eigenfaces principal components from training images will still be interpreted as a training method.

[0117] Figure 4B This is a schematic flowchart depicting the various operations common to inference methods in various machine learning models. As mentioned earlier, not all architectures or methods can include inference functionality. Where the inference method is applicable, in box 410a, the practitioner or architecture can assemble the raw inference data (e.g., a new image to be classified) into inference input feature vectors, tensors, etc. (e.g., using the same feature input form as training data). In box 410b, the system can apply the trained architecture to the input inference feature vectors to determine the output, such as classification, regression results, etc.

[0118] During "training," some methods and architectures can consider the input training feature data holistically, in a single pass, or ergonomically. For example, in some implementations, decomposition via PCA can be implemented as a non-ergonomic matrix operation. SVM, depending on its implementation, can be trained by a single pass through the input. Finally, some neural network implementations can be trained by performing multiple passes through the input vector during gradient descent.

[0119] Regarding the traversal training method Figure 4CThis is a schematic flowchart depicting the traversal training operations, as may occur in some architectures and methods, such as in box 405b. A single traversal can apply the methods in the flowchart once, while implementations performing multiple traversals can apply the methods multiple times. In box 415a, the parameters of the architecture can be initialized to default values. For example, in some neural networks, weights and biases can be initialized to random values. In some SVM architectures, for example, conversely, the operations in box 415a may not be applied. Since each training input feature vector is considered in box 415b, the system can update the model parameters in 415c. For example, an SVM training method may or may not select a new hyperplane because the new input feature vectors are considered and it is determined whether they affect the support vector selection. Similarly, a neural network method can update its weights and biases, for example, based on backpropagation and gradient descent. When all input feature vectors are considered, the model can be considered "trained" if the training method is called to perform only one traversal. Methods called for multiple traversals can be applied again. Figure 4C The operation (naturally, again avoiding initialization in box 415a to agree with the parameter values ​​determined in the previous traversal) and training is completed when conditions are met, for example, the error rate between predicted tags and metadata tags is reduced below a threshold.

[0120] As mentioned earlier, a wide variety of machine learning architectures and methods exist, including those with explicit training and inference steps, such as... Figure 4E As shown, and those architectures and methods that do not have explicit training and inference steps, such as Figure 4D As summarized in [the text]. For example... Figure 4E The method described is used to train a 425a neural network architecture to recognize newly received images at inference 425b, while Figure 4D Implementations such as reducing data dimensionality via PCA or performing KNN clustering are described, wherein implementation 420b receives input 420a and produces output 420c. For clarity, it will be understood that while some implementations may receive data input and produce output (e.g., an SVM architecture with an inference method), some implementations may only receive data input (e.g., an SVM architecture with a training method), and some implementations may only produce output without receiving data input (e.g., a trained GAN architecture with a random generator method for generating new data instances).

[0121] Figure 4D and Figure 4E The operations can be further extended in some methods. For example, some methods extend the training, such as... Figure 4F The diagram illustrates this, where the training method further includes various data subset operations. For example... Figure 4GAs shown, some training methods can divide the training data into a training data subset 435a, a validation data subset 435b, and a test data subset 435c. For example... Figure 4F As shown, when training the network in box 430a, the training method can first iteratively adjust the network parameters using, for example, backpropagation, based on all or a portion of the training data subset 435a. However, in box 430b, a subset of the data reserved for validation in 435b can be used to evaluate the effectiveness of the training. Not all training methods and architectures can guarantee finding the optimal architecture parameters or configuration for a given task; for example, they may get stuck in local minima, or they may employ inefficient learning step size hyperparameters, etc. The method can validate the current hyperparameter configuration in box 430b using training data 435b, which differs from the training data subset 435a that anticipates such defects, and adjust the architecture hyperparameters or parameters accordingly. In some approaches, the method can iterate between training and validation, as shown by arrow 430f, using validation feedback to continue training on the remainder of the training data subset 435a, restarting training on all or part of the training data subset 435a, adjusting the hyperparameters or topology of the architecture (e.g., when additional hidden layers can be added to a neural network in meta-learning), etc. Once the architecture has been trained, the method can evaluate its effectiveness by applying it to all or part of the test data subset 435c. Using different data subsets for validation and testing can also help avoid overfitting, where the training method tailors the architecture's parameters too closely to the training data, reducing better generalizations once the architecture encounters new inference inputs. If the test results are unsatisfactory, the method can restart training with different parameter configurations (architectures with different hyperparameter configurations, etc., as indicated by arrow 430e). The testing in box 430c can be used to confirm the effectiveness of the trained architecture. Once the model is trained, inference 430d can be performed on newly received inference inputs. People will understand that this verification method can vary, for example, when the method performs a grid search over the possible hyperparameter space to determine the architecture best suited for the task.

[0122] Many architectures and methods can be modified to integrate with other architectures and methods. For example, some architectures successfully trained for one task can be trained more effectively for similar tasks, rather than starting with, for example, randomly initialized parameters. Methods and architectures that take parameters from a first architecture and apply them to a second architecture (in some cases, the architecture can be the same) are called “transfer learning” methods and architectures. Given a pre-trained architecture 440a (e.g., a deep learning architecture trained to identify birds in images), a transfer learning method can perform additional training using data from a new task domain (e.g., providing labeled data of images of cars to identify cars in images), making inference 440e possible in this new task domain. The transfer learning training method may or may not distinguish between the training 440b, validation 440c, and testing 440d sub-methods and data subsets as described above, as well as the traversal operations 440f and 440g. It will be understood that the pre-trained model 440a can be received as the entire trained architecture, or, for example, as a list of trained parameter values ​​to be applied to parallel instances of the same or similar architectures. In some transfer learning applications, some parameters of a pre-trained architecture can be "frozen" to prevent them from being adjusted during training, while other parameters are allowed to change as the architecture is trained using data from a new domain. This approach preserves the general advantages of the architecture's initial training while customizing the architecture for the new domain.

[0123] The combination of architecture and method can also be extended over time. For example, the "online learning" method anticipates applying an initial training method 445a to the architecture, subsequently applying that trained architecture to the inference method 445b, and periodically updating 445c by applying another training method 445d, which can be the same as method 445a but typically used for new training data input. The online learning method can be useful, for example, in the case where the robot is deployed to a remote environment after the initial training method 445a, where it can encounter additional data that can improve the application of the inference method at 445b. For example, in the case of deploying multiple robots in this way, when one robot encounters a "true positive" identification (e.g., a new core sample with a classification verified by a geologist; a new patient characteristic verified by an operating surgeon during surgery), the robot can transmit that data and result as new training data input to its peer robot for use with method 445d. The neural network can perform backpropagation adjustments at training method 445d using the true positive data. Similarly, SVM can consider at training method 445d whether new data affects its support vector selection and contributes to the adjustment of its hyperplane. While online learning is often part of reinforcement learning, it can also appear in other methods such as classification, regression, and clustering. The initial training method may or may not include the training 445e, validation 445f, and testing 445g sub-methods, as well as the traversal adjustments 445k and 445l at training method 445a. Similarly, online training may or may not include the training 445h, validation 445i, and testing sub-methods 445j, as well as the traversal adjustments 445m and 445n, and if included, they can differ from the sub-methods 445e, 445f, 445g, and traversal adjustments 445k and 445l. In fact, the subsets and ratios of training data assigned for validation and testing can differ at each training method 445a and 445d.

[0124] As discussed above, many machine learning architectures and methods do not need to be exclusively used for any one task, such as training, clustering, inference, etc. Figure 4JAn example GAN architecture and method are described. In this GAN architecture, the generator sub-architecture 450b can competitively interact with the discriminator sub-architecture 450e. For example, the generator sub-architecture 450b can be trained to generate synthetic “fake” challenges 450c, such as synthetic portraits of non-existent individuals, while the discriminator sub-architecture 450e is trained to distinguish the “fake” challenges from real, true positive data 450d, such as genuine portraits of real people. Such methods can be used to generate, for example, synthetic assets that look like real-world data, to be used as, for example, additional training data. Initially, the generator sub-architecture 450b can be initialized with random data 450a and parameter values, resulting in a highly unbelievable challenge 450c. The discriminator sub-architecture 450e can initially be trained with true positive data 450d, and therefore can initially easily distinguish the fake challenge 450c. However, for each training epoch, the generator's loss 450g can be used to improve the training of the generator sub-architecture 450b, and the discriminator's loss 450f can be used to improve the training of the discriminator sub-architecture 450e. This competitive training can ultimately produce synthetic challenges 450c that are difficult to distinguish from true positive data 450d. For clarity, it will be understood that "adversarial network" in the GAN context refers to the competition between the generator and discriminator described above, and "adversarial input" refers to an input specifically designed to influence a particular output (possibly an output unintentionally designed by the implementation designer) in the implementation.

[0125] Data Overview

[0126] Figure 5A This is a schematic illustration of surgical data that may be received at a processing system in some embodiments. Specifically, the processing system may receive raw data 510, such as video from visualization tools 110b or 140d, which includes consecutive individual frames over time 505. In some embodiments, raw data 510 may include video and system data from multiple surgical procedures 510a, 510b, 510c or only a single surgical procedure.

[0127] As mentioned, each surgical procedure may include a group of actions, each group forming a discrete unit referred to herein as a task. For example, surgical procedure 510b may include tasks 515a, 515b, 515c, and 515e (ellipse 515d indicates that there may be more intermediate tasks). Note that some tasks may be repeated during the procedure, or their order may be changed. For example, task 515a may involve locating a fascial segment, task 515b may involve dissecting the first part of the fascia, task 515c may involve dissecting the second part of the fascia, and task 515e may involve cleaning and cauterizing the fascial area before closure.

[0128] Each task 515 can be associated with a corresponding set of frames 520a, 520b, 520c, and 520d, and a device dataset including operator kinematic data 525a, 525b, 525c, and 525d, patient-side device data 530a, 530b, 530c, and 530d, and system event data 535a, 535b, 535c, and 535d. For example, for video acquired from a visualization tool 140d in the surgical operating room 100b, operator-side kinematic data 525 may include translation and rotation values ​​from one or more handheld input mechanisms 160b at the surgeon's console 155. Similarly, patient-side kinematic data 530 may include data from a patient-side cart 130, data from sensors located on one or more tools 140a-140d, 110a, rotation and translation data from arms 135a, 135b, 135c, and 135d, etc. System event data 535 may include data on parameters that take discrete values, such as activation of one or more pedals 160c, tool activation, system alarm activation, energy application, button press, camera movement, etc. In some cases, task data may include one or more of frame set 520, operator-side kinematics 525, patient-side kinematics 530, and system event 535, instead of all four.

[0129] It will be understood that, although kinematic data are presented as waveforms and system data as continuous state vectors in this document for clarity and ease of understanding, some kinematic data may be assumed to be discrete over time (e.g., an encoder measuring continuous part points may be sampled at fixed intervals), and conversely, some system values ​​may be assumed to be continuous over time (e.g., values ​​may be interpolated, as when a parametric function can be adapted to individual sampled values ​​from a temperature sensor).

[0130] Furthermore, although surgical procedures 510a, 510b, 510c and tasks 515a, 515b, 515c are shown here as adjacent for ease of understanding, it will be understood that gaps may exist between surgical procedures and tasks in real-world surgical video. Therefore, some video and data may be task-independent. In some embodiments, these non-task regions may themselves be represented as tasks, e.g., "gap" tasks where no "real" task occurs.

[0131] The discrete set of frames associated with a task can be determined by the task's start and end points. Each start and end point can be determined by tool actions or changes in the body's state influenced by the tool. Therefore, data acquired between these two events can be associated with the task. For example, the start and end actions of task 515b can occur at timestamps associated with positions 550a and 550b, respectively.

[0132] Figure 5B This is a table depicting example tasks that can be used in conjunction with various disclosed embodiments, along with their corresponding start and end points. Specifically, the data associated with the task "Colon Mobilization" is data acquired between the time when the tool first interacts with the colon or surrounding tissue and the time when the tool last interacts with the colon or surrounding tissue. Thus, any one of the frame set 520, operator-side kinematics 525, patient-side kinematics 530, and systemic events 535 with timestamps between this start and end point is data associated with the task "Colon Mobilization." Similarly, the data associated with the task "Pelvic Fascia Anatomy" is data acquired between the time when the tool first interacts with the pelvic fascia (EPF) and the time when it last interacts with the EPF after prostate degreasing and dissection. The data associated with the task "Tip Anatomy" corresponds to data acquired between the time when the tool first interacts with tissue at the prostate and the time when the prostate has been freed from all appendages of the patient's body. It will be understood that the task start and end times can be selected to allow time overlap between tasks, or can be selected to avoid such time overlap. For example, in some embodiments, a task can be “paused,” such as when a surgeon involved in a first task switches to a second task before completing the first, completes the second task, and then returns to complete the first task. Therefore, while start and end points can define task boundaries, it will be understood that data can be annotated to reflect timestamps related to more than one task.

[0133] Additional task examples include “2-handed suturing,” which involves performing four horizontal interrupted sutures using a two-handed technique (i.e., in the case of only two hands, e.g., no single-handed suturing action occurs in between, the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle withdraws from the tissue). The “Uterine Horn” task involves dissecting the broad ligaments from the left and right uterine horns, as well as cutting the uterine body (it will be understood that some tasks have more than one condition or event determining their start or end time, as here, the task begins when the dissecting instruments contact the uterine horn or the uterine body, and ends when both the uterine horn and the uterine body are disconnected from the patient). The “1-handed suturing” task involves performing four vertical interrupted sutures using a single-handed technique (i.e., in the case of only one hand, e.g., no two-handed suturing action occurs in between, the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle withdraws from the tissue). The task “Suspensory Ligaments” involves dissecting the lateral lobes of each suspensory ligament to expose the ureter (i.e., the start time is when the dissection of the first lobe begins, and the stop time is when the dissection of the last lobe is completed). The task “Continuous Suturing” involves performing continuous suturing with four occlusions (i.e., the start time is when the suture needle first penetrates the tissue, and the stop time is when the needle is withdrawn from the tissue after all four occlusions have been completed). As a final example, the task “Rectal Artery / Venus” involves dissecting and ligating the superior rectal artery and vein (i.e., the start time is when the dissection of the artery or vein begins, and the stop time is when the surgeon stops contacting the ligation after ligation).

[0134] Example field of view in surgical data

[0135] When one or more of the data in frames 520, 525, 530, and 535 are available, they can be used for data analysis, such as monitoring surgeon performance and training machine learning classifiers for various applications. Unfortunately, in its initial acquired form, the visualization tool output in frame 520 may include not only the field of view within the patient but also an external field of view capturing sensitive or personal information present in the operating room, such as the faces of team members, the patient's identifying features, etc. This change in the field of view can also indicate that data in frames 525, 530, and 535 may no longer be relevant to the task in the surgical procedure. This capture of the external field can be undesirable because it may raise privacy or security concerns, thereby, for example, restricting access to the output used for data analysis purposes. Similarly, the data may be irrelevant to downstream processing, and therefore its existence represents an undesirable imposition on memory storage and downstream filtering.

[0136] Figure 6 The illustration shows what may occur in some embodiments from Figure 1A and Figure 1BThis is a schematic diagram illustrating an example phase of the output of a visualization tool (e.g., visualization tool 110b or visualization tool 140d, such as laparoscopic ultrasound or endoscopy) in a surgical operating room. As the operating room begins operation, team members can begin recording using visualization tool 110b or visualization tool 140d, resulting in an output visualization captured in video, regardless of whether the tool is still inside the patient. Therefore, the tool can be in a “pre-activated” state 605a before the operating room begins its first operation, such as when the tool is switched off or switched on but covered by a protective cover. In this state, the tool’s field of view 605b, and any resulting video output, is unlikely to include any sensitive information. Nevertheless, it may be desirable to remove these video frames because they do not depict the field of view relevant to the surgical procedure. Similarly, it may be desirable to remove data 525, 530, and 535 corresponding to these frames, as they are not yet relevant to the surgical procedure.

[0137] During the subsequent "pre-operating room" state 610a, team members (e.g., team member 105b or 105d) may activate the instrument before it is inserted into the patient, for example, by removing the instrument's cap or powering it. Similarly, the instrument may have been previously activated, but video recording only begins at this point. During this period, the field of view 610b may include sensitive information such as the faces 630d or other identifying features of team members, patient identification charts 630c, patient faces 630b or other identifying features, a surgical room whiteboard 630a with patient information and surgical schedules, etc. Distributing this information outside the institution performing the surgery may violate contractual or regulatory requirements, such as various HIPAA protections. Similar to state 605a, it may also be desirable to ignore data 525, 530, and 535 captured during this period.

[0138] Once the surgical procedure begins, the tool can enter an "in vivo operation" state 615a, where its field of view 615b depicts the gross, non-identifiable features of the patient's internal anatomy. As mentioned, various tasks throughout the surgical procedure may require the removal and / or repositioning of the visualization tool 110b or 140d. Such adjustments can transition the tool to an "occasional removal" state 615d, where its field of view 615c can again include sensitive information before the tool returns to the "in vivo operation" state 615a. There are various reasons for removing the tool, such as changing the arm in which the visualization tool is located to obtain different views of the anatomy, cleaning the tip of the endoscope to remove contaminants / fluids during surgery, etc. Therefore, sometimes tool removal and reinsertion can be expected as part of a task operation or transition, while at other times they may be temporary, unexpected events.

[0139] When the surgical procedure is finally terminated, the visualization tool can be removed in the "Operating Room Post-Operating" state 620a, and it can again present sensitive information within its field of view 620b. If multiple surgeries are performed during the entire video recording process (e.g., in the case of simply recording during the day's surgical procedures), the tool can be deactivated and, once the surgical procedures in the operating room have been completed (e.g., when the operating room terminates its surgical procedures for the day) and enters the "Post-Activation" state 625a at 640, it can repeatedly traverse states 610a, 615a, 615d, and 620a, after which the "Post-Activation" state 625a can again depict a blank field of view 625b. Although the video recorded from the output of visualization tool 110b or visualization tool 140d can generally follow Figure 6 This is a phase, but people will understand that some videos may deviate from this example pattern, for example, in cases where the video is prematurely terminated due to lack of storage during “in-body operation” status 615a, or in cases where the competent authority edits the video before it is released, etc.

[0140] Of course, if we consider identification Figure 6 The data 525, 530, and 535 clearly show the insertion and removal of visualization tools, making corresponding excisions in the data relatively straightforward. Motion of visualization tools captured in kinematic data or salinity levels captured in event or sensor data can be used to identify which data are associated with the "in vivo operation" state 615a. However, many operating rooms are in the form of operating room 100a rather than 100b, and while both operating rooms can capture video data, capturing data 525, 530, and 535 in operating room 100a may be less common. Therefore, ideally, video-only data from both operating rooms 100a and 100b could be processed to remove non-surgical frames, allowing more data to be used for downstream processing (e.g., some deep learning algorithms benefit from having access to more data). Furthermore, by reviewing data 525, 530, and 535 based solely on video, one can verify data 525, 530, and 535 when they are available.

[0141] Example video content processing methodology

[0142] Figure 7A This is a flowchart illustrating example operations in a surgical video processing method 700, which may be implemented in some embodiments. It will be understood that not all blocks of method 700 may be performed by the same processing system or in the same location, although this may be the case in some embodiments. In block 720a, the processing system may acquire surgical data for one or more surgical procedures from one or more visualization tools. For example, refer to... Figure 7BThe illustrative time series data, as time progresses during the surgical procedure (750), shows corresponding data values ​​that can be recorded in dataset 755 (e.g., video frames). (It will be understood that, as described herein, in some cases, the video may be compressed and the frames will need to be extracted or manipulated after necessary modifications). Therefore, each data point (e.g., a video frame) can be associated with a unique timestamp.

[0143] In box 720b, the system can distinguish between surgical and non-surgical data, such as video frames acquired when a visualization tool uses the system and methods described herein to potentially capture sensitive or irrelevant information in one of states 605a, 610a, 615d, 620a, or 625a. For clarity, such identification can specify frames before time 760a, between times 760c and 760d, between times 760f and 760g, etc., for removal from the dataset.

[0144] In box 720c, the system can smooth the classification of frames acquired in box 720b. For example, if a machine learning classifier is used in box 720b to distinguish between surgical and non-surgical data, false positive and false negative classifications may exist in the output. Smoothing can help adjust the classification to compensate for such false positives and false negatives, for example, in cases where such classifications result in statistically impossible lengths for either surgical or non-surgical data.

[0145] In box 720d, the processing system can then use frame classification and corresponding timestamps to infer when the surgical procedure began and ended (e.g., times 760b and 760e, 760h, etc.). It will be understood that this may not be necessary in cases where only a single surgical procedure is provided in box 720a, or where recognition is performed in real-time as the surgery progresses. It will also be understood that in some embodiments, the end of the video can be considered the end of the surgery if the video terminates at a time with a surgical start time but lacking a corresponding surgical stop time. Knowing when the surgeries within dataset 755 begin and end can facilitate dividing a single data capture into multiple component segments, each depicting a separate surgical operation. These segments can then be output for subsequent processing, such as human-based annotation or machine learning analysis of individual surgical segments.

[0146] Example video content processing system component topology

[0147] To facilitate a more detailed understanding Figure 7A The method Figure 8 The illustration shows a method that can be implemented in some embodiments. Figure 7AThis is a schematic diagram of the components of a surgical procedure video filtering system. It will be understood that a surgical procedure video filtering system may include multiple component processing systems, which may themselves be different software, hardware, or firmware implementations. As will be discussed, these component systems may not be co-located and may be designed for different functions. Therefore, two components may be different software lines running on the same or different computer systems, different hardware, firmware, etc.

[0148] The processing system can receive or manipulate a surgical dataset 810 in real time, which includes at least video frames 805a and 805b having portions 820a, 820b, 820c, 820d, and 820e depicting a field of view within the patient and portions 815a, 815b, 815c, 815d, and 815e depicting a field of view outside the patient. Initially, a preprocessor component 825a can receive the dataset 810 for processing. The preprocessor component 825a can downsample the dataset to facilitate downstream analysis, convert the data to a more suitable format (such as when video is captured in compressed form and will be converted to discrete frames by the preprocessor), decrypt the data (if it is encrypted), etc. Downsampling saves computational resources because visualization tools or recording devices can capture video at a much higher frame rate than required to distinguish portions of the dataset associated with the field of view within or outside the patient. This saving can be particularly useful when subsequent components operate in environments with limited memory or processing power, or when identification is performed in real time during surgery and transmitted over a network to a component in the cloud with significant delay.

[0149] In some embodiments, if dataset 810 includes systemic events or kinematic data that clearly indicate surgical or non-surgical portions, preprocessor component 825a can classify frames before submitting them to more complex video-based algorithms to save computational resources. Similarly, portions of a dataset can be quickly categorized where the state is easily perceived from the video (e.g., in cases where a video frame depicts a “pre-activated” state 605a and is completely black, or where a user interface in a GUI displayed in the frame indicates the location of visualization tools inside or outside the patient) or from metadata accompanying the video (e.g., manual annotations by surgical team members indicating when the procedure was terminated).

[0150] However, dataset 810 will typically only include video data. Preprocessor component 825a can provide downsampled data to feature extractor component 825b, which can generate features 830a from the frames. Although features 830a are represented here as a linear vector of values, it will be understood that features can take a non-linear form. In fact, the same data can be reformatted into linear or non-linear forms, such as when an RGB 63x64 pixel image is represented as a linear vector of 12,288 values ​​or a tensor of dimension 3x64x64. Thus, for example, in the case where the data consists only of video frames, each feature vector can be a pixel value associated with a single frame (although in some embodiments, the frame may be color-shifted, compressed, reduced, etc. via PCA). In some embodiments, where dataset 810 includes event or kinematic data (although, again, this is generally not the case), feature extractor component 825b can append discrete values ​​derived from such data to a linear representation (or other suitable representation) of the video frame pixel vector to assist downstream machine learning models in recognition (where such models are configured to receive such data).

[0151] The feature extractor component 825b can pass the generated feature vector 830a to the machine learning model 825c for initial classification. For example, the model can receive one or more video frames and output classification values ​​(e.g., "within the patient" or "outside the patient / irrelevant data"). Such output can be in the form of an array 830b, where each value represents the classification result for the corresponding frame. It will be understood that in some embodiments, such an array can be formed by applying consecutive frames to the model in chronological order and appending the resulting output.

[0152] Such binary results can also be represented in this paper by waveform 830f (naturally, although a continuous waveform is shown here to facilitate understanding, one will understand that the classification can be applied in practice to discrete video frames), which illustrates how the classification value of the corresponding data can take values ​​within or outside of time as time progresses from left to right. As indicated in example waveform 830f, the model may not provide a perfect classification, resulting in false positives or false negatives. For example, an interval 830c within the patient may be too short to be reasonably interpreted as a true data point within the patient (in fact, some such intervals caused by model misclassification may be too short to be physically possible). Similarly, data point 830e may be too short to be reasonably interpreted as reflecting the removal of the instrument from the patient. Conversely, some intervals (such as interval 830d) may be long enough that they reasonably reflect the actual removal and reinsertion of the instrument during surgery, or generally, if longer, the intervals that occur between surgical procedures.

[0153] Model component 825c can pass the result represented in waveform 830f to smoothing component system 825d. In some embodiments, model component 825c and smoothing component system 825d can be on the same system. However, as mentioned, it will be understood that the smoothing component system may not be co-located with model component 825c. Therefore, when moving the result at each of arrows 835a, 835b, 835c, 835d, 835e, 835f, and 835g, encryption / decryption can be performed because the data may still contain sensitive information (in some embodiments, although a provisional classification exists, the actual cut-off of the data based on the classification may not occur until later).

[0154] The smoothing component 825d can "smooth" categorical values, removing false positives and false negatives reflected by regions 830c and 830e discussed above. For example, the smoothing component 825d can apply time windows (e.g., kernel filters) to the dataset, as discussed elsewhere in this paper, removing regions that are too small to align with the expected inner / outer duration. This can result in a "clean" dataset, as represented by waveform 830g. Note that the truly removed regions 830d can be retained after smoothing.

[0155] The smoothing component 825d can provide these clean classification results to the start / end time detection component 825e. In some embodiments, frames classified as outside the patient may be whitewashed (data replaced with zero values) or cropped to comply with regulatory requirements, for example. After such an operation, it is feasible to move the data to another location (e.g., where the start / end detector component 825e may be located). However, the smoothing component 825d and the start / end detector component 825e can often be juxtaposed and may actually be different software blocks running on the same computer system (e.g., the same computer system containing all the previous components discussed in other corresponding software code blocks).

[0156] The start / end time detector component 825e can evaluate the intervals involved, as discussed elsewhere in this document, to identify where a surgical procedure begins and ends. For example, the start / end time detector component 825e can generate data 830h annotated with start / stop times by determining time 830i as the start of a new surgical procedure and time 830j as the end of a surgical procedure based on the length of previous and subsequent “outer” intervals. Note that the actual “outer” region 830d will still be retained as an “outer” interval within the surgical procedure (and therefore can be removed or whitewashed). Again, it will be understood that the start / end time detector component 825e may be absent or disabled when filtering is applied to real-time data during surgery.

[0157] In some embodiments, the start / end time detector component 825e can provide the results to the post-processing component 825f. For example, the post-processing component 825f can whitewash, blur, or cut off the data, or divide the data into different segments (e.g., for both the surgical area and the actual “outer” area 830d of the data) to suit downstream processing. Such a downstream system can access the data output, which is stored in a storage processing component 825g (such as a cloud-based web server).

[0158] One will understand that while each of arrows 835a, 835b, 835c, 835d, 835e, 835f, and 835g reflects the movement of results or data from one component to another, since the components may not be juxtaposed, each arrow can also reflect the encryption and decryption of data or results. However, once the "non-surgical" data has been cleared or removed, encryption may no longer be necessary. Conversely, in some embodiments, all components will reside on a single computer system (e.g., each component is a separate block of code running on the same computer system). In this case, encryption may also be unnecessary to comply with regulatory requirements. In fact, in the case of successful removal, encryption may not be necessary because any data leaving the controlled environment has already had its non-surgical data removed.

[0159] Example machine learning recognition system

[0160] Figure 9A This is an example of an illustrative integrated machine learning model topology 900a, which can be implemented in some embodiments to determine whether a video frame of a visualization tool depicts a view inside or outside a patient's body. Given a video frame 905, the processing system can generate "direct features" 910 by applying logical operations (e.g., computer code) to the frame itself, and can generate "network features" 920 by feeding the video frame to one or more deep learning networks 915. Both the direct features 910 and the network features 920 can then be fed to one or more machine learning classifiers 925 to produce one or more predictive classifications 930 of whether the frame depicts a field of view inside or outside the patient. It will be understood that the deep learning network 915 can be used in its pre-trained form (even on datasets unrelated to surgery, such as ImageNet or public objects in context datasets). However, in some embodiments, transfer learning can be applied, such as when layers other than the final head layer of the network can be frozen (i.e., frozen). Figure 3F The "feature extraction" layer) and the model is trained (i.e., the head layer). Figure 3F The weights in the layers of the “classification” can be varied to identify inner and outer frames before the network is incorporated into the model.

[0161] These predicted classifications 930 can then be considered separately from the fusion predictor 935 (e.g., a fusion machine learning model or fusion logic, such as majority voting logic) and the uncertainty calculator 940 to produce a fused field-of-view prediction and a measurement of uncertainty 950 for the prediction of frame 945, respectively. As discussed elsewhere in this document, one will understand that instead of, for example, the uncertainty calculator 940 applied to the discriminative classifier model 925, the generative model 920 can be replaced and the uncertainty determined, for example, based on the variance of the probability distribution output by the generative classifier. In some embodiments, separate classifiers can be used, such as a discriminative classifier for predicting the fused value 945 and a separate generative classifier for determining the uncertainty 950.

[0162] To facilitate reader comprehension, it can be based on Figure 9A Implementation topology Figure 9B In some embodiments, this can be achieved in Figure 9A The illustration illustrates an example selection of models used in an integrated machine learning model topology. Specifically, this instantiation can directly generate color histogram 910a features, color speckle 910b features, image texture 910c features, and image moment features 910d from image 905. Experiments have shown that, in some embodiments, selecting one or more of these features can facilitate the generalization of the classifier across a wide range of surgical procedures.

[0163] Color histogram feature 910a can represent the colors appearing in an image in a way that is invariant to small changes in camera viewpoint. For example, a system can extract an HSV color space histogram with an 8x8 bin size for each HSV color channel. It will be understood that many libraries (such as OpenCV) TM This provides such functionality; for example, lines of code C1 and C2 demonstrate an example histogram calculation:

[0164] image=cv2.cvtColor(img,cv2.COLOR_RGB2HSV) (C1)

[0165]

[0166] In line C1, the image is converted to the appropriate form, "img" is the image 905, "[0,1,2]" is the channel to be considered, "bins" is the range of values ​​to be grouped together in the histogram, and "[0,256,,0,256,0,256]" is the range to be used.

[0167] Color blob feature 910b can reflect brightness or color compared to surrounding pixels. For example, color blob feature 910b can be a Hessian determinant (DoH) method for detecting blobs, where the center point and volume of the detected blobs are used as measures. Such dark / bright regions in an image can also be determined by using Laplacian of Gaussian, difference of Gaussians, Hessian difference methods, etc., on each color channel of the image. The output of each channel can be a list of x,y positions of blobs of a specific size. This output can be truncated to include only a threshold number (e.g., 100) of the largest blobs in each color channel. For example, data from scikit-image can be used. TM The library's skimage.feature.blob_doh function is shown in line C3 of the code:

[0168] blobs_doh=blob_doh(img,max_sigma=30,threshold=.01) (C3)

[0169] Image texture features 910c can reflect spatial variations in pixel intensity within an image, indicating, for example, the consistency of repeating patterns on the surface of an object. In some embodiments, the texture descriptor can be one or more of the 13 characteristic parameters calculated based on the gray-level co-occurrence matrix of the image, as proposed by Haralick and Shanmugan. Tissue and instrument texture features can differ significantly from the features generated based on the “texture” appearing within the field of view of the visualization tool when the instrument is withdrawn from the patient. For this purpose, scikit-image, for example, can be used. TM The greycomatrix and greycoprops functions, for example, provide "patchwork" for a given image, as shown in listing C4:

[0170]

[0171] Image moment features 910d can be a weighted average of image pixel intensities, thus capturing information about the shape of blobs in the image. Some embodiments employ Hu moment invariants for this purpose, computing the first seven most important moments. Because Hu moments are invariant to image transformations of translation, scaling, and rotation, they can provide useful general object representations regardless of orientation. It will be understood that many libraries facilitate the computation of such moments, such as OpenCV, as shown in listing C5. TM The library's HuMoments function, for example, when applied to an image "img":

[0172] imu_hu_moment=cv2.HuMoments(cv2.moments(img)) (C5)

[0173] People will understand that they can easily create “fusion” combinations of features 910a, 910b, 910c, and 910d by linking (or otherwise combining) the various outputs of features 910a, 910b, 910c, and 910d.

[0174] Image 905 can also be fed as input to various pre-trained implementations of neural networks, such as implementations of the VGG16 network 915a (features are the network's output, 512x1 feature vector), implementations of the ResNet50 network 915b (features are the network's output, 2048x1 feature vector), and implementations of the MobileNet network 915c (features are the network's final output, 1280x1 feature vector). It will be understood that examples of such pre-trained networks are available in many libraries, such as Keras. TM Libraries (such as tf.keras.applications.VGG16, tf.keras.application.MobileNet, tf.keras.application.ResNet50, etc.) can each load networks pre-trained on a database (such as the ImageNet database).

[0175] The input shape for each network parameter can be a resized 128x73x3 version of image 905, where the image is RGB color. Regarding VGG16, in some embodiments, features from the output of the final convolutional block can be fed into global average pooling to reduce spatial dimensionality, resulting in a vector of 512 features per video frame. Similarly, some embodiments can apply global average pooling to the outputs of ResNet50 and MobileNet networks, yielding 2048 and 1280 features, respectively.

[0176] The resulting features can then be concatenated and applied to intermediate classifiers 925a, 925b, 925c, and 925d. Specifically, in this example implementation, SVM 925a, logistic regression classifier 925b, random forest 925c, and Adaboost ensemble 925d are used. One will understand the various methods used to implement these classifiers, for example, through specific libraries (such as LIBSVM). TM ) or more general libraries (such as scikit-learn) TM (Tool kit).

[0177] For example, listing C6 provides an example implementation of SVM 925a (one will understand that sckikit-learn...). TMThe SGDCIassifier in the code uses an SVM as the default model. Listing C7 provides an example implementation of the logistic regression classifier 925b, Listing C8 provides an example implementation of the random forest classifier 925c, and Listing C9 provides an example implementation of the Adaboost ensemble 925d (using a decision tree as the base classifier). Some implementations may additionally or alternatively employ a gradient boosting classifier as an intermediate model 925, as shown in Listing C10 for scikit-learn. TM The library call is shown in the call.

[0178] In addition, some implementations may additionally or alternatively employ unsupervised models, such as KMeans and GaussianMixture.

[0179] The results from each of these models (i.e., the predicted frame field of view classifications 930a, 930b, 930c, and 930d, respectively) can then be provided to a fusion predictor 935a (which itself can be a machine learning model, such as a logistic regression classifier, SVM, etc., or logic, such as voting logic) and to an uncertainty calculator 940a. The fusion predictor and the uncertainty calculator can each produce a final prediction 945a and an uncertainty 950a, respectively.

[0180] As will be discussed in more detail in this article, the uncertainty calculator 940a can, for example, use tools from SciPy. TM The library's `scipy.stats` entropy function determines the entropy of the classification probabilities from categories 930a, 930b, 930c, and 930d. In some embodiments, uncertainty can be used to make decisions about which frames should be blacked out and which should be retained (e.g., removing frames predicted as "inside" but with high uncertainty). Uncertainty can also be used to detect model drift due to changes in the data source. That is, if the image 905 fed to the model changes drastically (e.g., if the data applied at inference time is incorrectly obtained from a source other than the endoscope, even if it is unrelated to the surgical procedure), the model still makes an inside / outside prediction 945a, but the uncertainty value 950a may fall far outside the normal distribution, such that the system or human operator will recognize something is wrong. The method further includes applying a windowed filter to produce multiple smooth, fused predictions.

[0181] Example features and intermediate model relationships

[0182] Again, although Figure 9BAn example selection of features and intermediate classifiers for one embodiment of a class is described, but other embodiments may use different feature and classifier selections. For example, in Figure 10A Instead of concatenating all direct feature outputs and providing the same result to the intermediate model, in some embodiments, the system instead determines only the image histogram 1005a and color blob 1005b features from the incoming image frame 1000. The image histogram 1005a may take the form of a 3x8 data structure (e.g., cell counts in each of the eight possible histogram orientations for each of the three separate channels), and the color blob 1005b may take the form of a 100x2x3 data structure (e.g., as described above, the 100 largest blobs and their two-dimensional positions for each of the three color channels). Each of these features can be reordered 1005c, 1005d as linear vectors and separately fed to the logistic regression classifier 1005e and the random forest classifier 1005f. The results from each of these classifiers can then be considered by the fusion predictor 935 and the uncertainty calculator 940, as discussed elsewhere in this document.

[0183] As with Figure 9B A different example class implementation, in which... Figure 10B In one embodiment, Hu moments 1010a and Haralick image texture features 1010b can only be derived from the incoming image frame 1000. Haralick image texture features 1010b can take the form of a 3x13 data structure (e.g., one or more of the 14 statistics identified by Haralick for each of the three channels). Among the 14 Haralick statistics are angular second moments, contrast, correlation, variance, inverse moments, mean sum, sum variance, sum entropy, entropy, variance difference, entropy, correlation measures 1 and 2, and the maximum correlation coefficient; the last one is omitted in this example due to its computational complexity. Hu moments 1010a can take the form of a 3x7 data structure, since the seven most important moments are selected for each channel. Each of these features can be reordered 1010c, 1010d as linear vectors and separately fed to the Adaboost classifier 1010e (e.g., as described above) and the SVM 1010f (e.g., also as described above). The results from each of these classifiers can then be considered by the fusion predictor 935 and the uncertainty calculator 940, as discussed elsewhere in this paper.

[0184] Such as about Figure 9B As mentioned, some embodiments may link features together before providing them to an intermediate model. Figure 10C Another such example is provided here. For example, Figure 10BAs shown, Hu moment features 1015a and Haralick features 1015b can be derived from the incoming image frame 1000. Similarly, the features can be reordered 1015c and 1015d into linear 1x21 and 1x39 feature vectors, respectively. However, these can then be concatenated 1015e and provided as a single vector of length 60 (21+39=60) to one or more intermediate classifiers, in this case, a single random forest intermediate model 1015f (again, in some embodiments, in the case of only one intermediate model, this model can also act as a fusion predictor 935a).

[0185] Similar joining and separation of feature vectors between direct features can also be performed using neural network features. For example, in Figure 10D In this model, the VGG19 network 1020a, ResNet network 1020b, and MobileNet network 1020c can generate 1x512, 1x2048, and 1x1024 dimensional outputs from the incoming image frame 1000, respectively (based on their corresponding global pooling operations). These outputs can be separately and directly fed to each of the Adaboost intermediate model 1020d, SVM intermediate model 1020e, and Random Forest intermediate model 1020f.

[0186] Furthermore, as yet another example embodiment, it will be understood that, Figure 9B As shown, feature vectors can be merged between direct features 910 and neural network features 915. Furthermore, one will understand the changes in the intermediate classifier. Therefore, in another example embodiment, Figure 10E The diagram illustrates a combination of features and models, where a network model (such as MobileNet network 1025d) can produce a 1x512 output, and a RestNet50 network 1025c can produce a 1x2048 output. Direct features can be generated, here histogram 1025a and Haralick image features 1025b. Each of these feature vectors can be fed into a corresponding classifier. Specifically, histogram output 1025a can be fed into a logistic regression intermediate classifier 1025e, Haralick image texture features 1025b can be fed into an SVM 1025f, ResNet50 output 1025c can be fed into a separate SVM 1025g, distinct from SVM 1025f, and MobileNet output 1025d can be fed into a random forest intermediate classifier 1025h. In a variant of the embodiment, Haralick image texture output 1025b and ResNet50 output 1025c can be concatenated and fed into a single SVM classifier.

[0187] Although no other options and combinations are presented above, one will readily recognize the additional variations in the features used, when and how features are linked, and the choice of intermediate classifiers applied to the features. Furthermore, one will understand that each of the disclosed combinations can represent a feature selection that is more suitable for certain operational situations than other combinations. For example, although... Figure 10A A pair of features can impose minimal computational overhead and is therefore suitable for real-time applications with limited resources; however, for certain surgical procedures, they may still be less effective than other disclosed combinations. Specifically, if the surgical procedure under consideration involves fluorescence imaging, some hues in the fluorescence-associated color spot 1005b may appear similar to hues appearing in some external endoscopic images (e.g., depending on the lighting in the operating room). In this case, alternatively, for example... Figure 10B or Figure 10C The combination of features is possible because the Haralick image texture 1010b and image shape information captured in Hu moment 1010a can be used to distinguish frames, regardless of the similarity of fluorescence hues. Therefore, while fewer features generally result in lower computational overhead and simplified processing, one may also wish to consider the nature of the environment when selecting the various proposed combinations.

[0188] Example Training Methodology

[0189] In some embodiments, the training of model 900b can be performed in stages. For example, Figure 11A It is a description used for training Figure 9B A schematic diagram of one or more intermediate classifier components in the topology. In the case where the fusion predictor 935 is a component performing logical operations rather than a machine learning model, according to... Figure 11A Performing training is sufficient to train architecture 900b (although, as mentioned, in the case of network 915 employing transfer learning, their head layers can be trained separately). However, in the case of the fusion predictor 935 having parameters to be calibrated or being itself a machine learning model (e.g., SVM, logistic regression classifier, etc.), it can be as follows: Figure 11B The second phase of training is shown in the diagram.

[0190] Specifically, while architecture 900b is configured to receive individual frames 905, training can be performed by providing multiple frames 1105a, annotated with their states as IN frames or OUT frames (i.e., depicting the field of view inside or outside the patient). Each of these frames can be used to generate direct features 1105b (e.g., using feature extractor component 825b and example library calls presented herein) and organized into a set 1105c suitable for training intermediate machine learning classifiers 925. It will be understood that while set 1105c is shown here as a linear array with corresponding annotations (e.g., corresponding to IN or OUT frames), it will be understood that in some embodiments, the features may not be linear, and individual “feature sets” may include feature vectors or tensors of different formats for each of their respective intermediate machine learning classifiers 925. The training system 1105d (e.g., a computer system that performs the training operations discussed herein) can use set 1105c to train each intermediate machine learning model to produce trained classifiers 1105e, 1105f, 1105g, 1105h.

[0191] The trained classifiers 1105e, 1105f, 1105g, and 1105h can then be used to train the fusion classifier model 1120c (or, in the case that the fusion classifier is a logic rather than a machine learning model, to configure parameters). Specifically, the annotated frame data 1110a can be converted back into features 1110b and organized into set 1110c (although in some embodiments data 1110a may differ from data 1105a, it will be understood that, where they are identical, set 1105c can be reused to generate training set 1120a, as discussed below). When applied to the now-trained classifiers 1105e, 1105f, 1105g, and 1105h, the corresponding classification outputs can now form training set 1120a, which can be used by training system 1120b to create the trained fusion classifier model 1120c.

[0192] Again, as discussed elsewhere in this document, the fusion predictor 1120c can be logic (e.g., voting logic) rather than a trained classifier that combines the predictions of each classifier to generate a final prediction. For example, the fusion predictor could be code that determines the argmax of each classifier and then performs majority voting on the result. In some embodiments, voting could instead involve averaging the probabilities from each predictor for each class to produce a new class probability vector, then normalizing that vector to confirm that the average sum of probabilities across classes is still 1, and then taking the argmax of that average probability vector. Here, in Figure 11BHowever, the fusion model 1120c can be, for example, a logistic regression model that learns how to weight the class probabilities of each classifier to overcome the bias of each method.

[0193] It will also be understood that, in some embodiments, the annotated feature vector training sets 1105c and 1110c can be the same set or different sets (i.e., sets generated from different annotated input images). Reusing set 1105c as set 1110c is desirable when acquiring new data or regenerating features is not expected or feasible. In some embodiments, available data can be... Figure 11A and Figure 11B The two training operations are divided to better ensure a robust machine learning model.

[0194] Figure 12A This is a flowchart illustrating various operations in the process of training one or more intermediate machine learning models 925, as may be implemented in some embodiments. Generally, practitioners may want to determine which model selection is most suitable for the data type under consideration, and which feature types or combinations are most suitable for each model. While human reviewers can manually examine the training results to select feature vectors and models, it will be understood that the training process can be automated or used in conjunction with meta-learning machine learning methods.

[0195] Specifically, at block 1205a, the training system may receive annotated video of a surgical procedure (e.g., inside or outside the patient). In some embodiments, the video may have been annotated by a human reviewer. However, in some embodiments, the video may be “annotated” based on system or kinematic data accompanying or within the video. For example, as discussed, kinematic data 530 (e.g., indicating the position and orientation of an instrument over time) or event data 535 captured simultaneously with the video may have already indicated when the field of view of the video is inside or outside the patient. Similarly, some systems may include a user interface in the video capture indicating whether the field of view is inside or outside the patient (e.g., template matching for identifying icons indicating inside / outside positioning in the interface may be used for “annotation”).

[0196] In box 1205b, the system can preprocess the video. Such preprocessing may involve downsampling, resizing video frames to a uniform form, converting compressed video into individual frames, etc. In box 1205c, the training system can convert annotated video frames into annotated feature vectors based on the selection or omission of direct features and neural network features described herein. For example, the practitioner or training system may decide to select some of “direct features” 910 and “network features” 920, all possible features, various connections of two or more selections, etc.

[0197] Then, in boxes 1205d and 1205e, the system can iterate through the intermediate model types 925 selected for consideration. Similar to feature selection, the practitioner or meta-learning training system can select only one model (e.g., random forest) or multiple model types for consideration (e.g., logistic regression classifier model, SVM, and random forest). For each model type, in box 1205f, the practitioner or training system can train the model on various selections of available data for each selected feature type, e.g., as referenced... Figure 12B To describe in more detail, such training can generate multiple “candidate” trained models, which can be stored in box 1205g for consideration in box 1205h. For example, in box 1205h, each of the stored trained models and its validation results can be considered to select the most suitable trained model for use with the fusion classifier (e.g., selecting the models with the top 5 validation results, expecting the fusion classifier to perform a majority vote). Once a selection is made, the feature generation process can be simplified for inference (e.g., generating only the feature types used by the selected model for the input video data).

[0198] Figure 12B This is a flowchart illustrating various operations during the intermediate machine learning model training process. For example, in some embodiments, it can be... Figure 12A The application is in box 1205f. In boxes 1210a and 1210b, the system can iterate through the expected feature types, such as those selected in box 1205c.

[0199] In some embodiments, at block 1210b, features from frames of all available training videos can be collected, and the frames are split into training and test sets (training here includes validation, and the test set is used to evaluate the final performance). In some embodiments, the system can instead first split the videos into training and test video groups, and then collect frames from each video group for training and testing (again, training here includes validation, and the test set is used to evaluate the final performance). The latter approach can promote model generality in terms of test videos, i.e., whether the model can perform well on new videos, while the former approach can emphasize robustness in terms of frame-level modeling.

[0200] In box 1210c, the system can select the folding distribution of training data in the chosen feature format. It will be understood that "K-folds" training cross-validation is a technique to avoid overfitting the data. For example, refer to... Figure 12CExample training data 1215a is provided, which may be in the format of selected features and annotated as discussed elsewhere herein. In box 1210c, this data can be divided into a training portion 1215b for training during validation and a test portion 1215c for evaluating the final result (in some embodiments, the test portion 1215c may be omitted and all available training data may be used as the training portion 1215b). The training portion 1215b itself can be used to determine the model's hyperparameters, while the test portion 1215c may be withheld to provide a final evaluation of the generated model. For this purpose, the training portion 1215b itself can be divided into "folds" of approximately equal data groups (three such folds are shown here). In each training iteration, the version of the model's hyperparameters can be determined by training the model using all folds (e.g., in the first model, folds 2 and 3 may be used for training, and fold 1 for validation; in the second model, folds 1 and 3 may be used for training, and fold 2 for validation, etc.).

[0201] As discussed herein, many datasets will have significantly more "inner" frames than "outer" frames, and this asymmetric representation can also exist in individual folds of data 1215a. Therefore, in some embodiments, at box 1210l, the system may consider whether the training data within the currently considered fold is balanced. If not, at box 1210e, elements of the underrepresented data (typically "outer" data) can be upsampled, for example, using the Synthetic Minority Oversampling Technique (SMOTE) algorithm. In this way, upsampling at each fold traversal helps ensure consistency between training and validation folds.

[0202] In block 1210f, the system can determine the hyperparameters of the current fold selection by training the model according to the model's training methodology. Using the remaining folds of the data, the system can then perform validation, evaluating the trained model in block 1210g and saving the model and results in blocks 1210h and 1210i, respectively. In some embodiments, the evaluation in block 1210g can involve validating individual frames or validating the entire video of the data. The former can provide robustness between video captures, while the latter can provide robustness within a single video. In the latter, in some embodiments, the same or similar operations performed by the smoothing component 825d can be used to temporally smooth the predictions, where such smoothing will also be applied during future inference. In some embodiments, each of these variations can be pursued in each traversal of the K-fold to provide a wider range of possible model choices for use.

[0203] After the training fold is performed in box 1210d, the system can review the saved results in box 1210i and select the best-performing model in box 1210k. However, in some embodiments, this evaluation can be postponed until all models of all model types have been trained, and then compared in box 1205h (e.g., some meta-learning algorithms may wish to make all possible combinations of trained models available for competitive comparison).

[0204] Example Smoothing Methodology

[0205] After applying the classification operations described herein, post-processing operations such as smoothing (e.g., via the smoothing component 825d) can be applied to help reduce false negative and false positive classifications. Figure 13A This is a schematic depiction of applying a smoothing window 1340 to a set of data classification values, as may occur in some embodiments. Generally, the window 1340 can slide in the temporal direction (e.g., from left to right as depicted herein, where frames are arranged in chronological order from left to right) taking into account the classification of frames before and after the frame being considered. For example, some embodiments employ a 3-second window 1340, considering frames 1.5 seconds past and 1.5 seconds future relative to the currently considered frame. It will be understood that buffered values ​​can be prepended and appended to the frame sequence (e.g., the classification value of the first frame can be prepended within the window for initial consideration, and the classification of the final value is appended within the window for final frame consideration), although some embodiments may instead simply omit smoothing for frames at each end. Similarly, the window may not be constant during the sliding operation, expanding and contracting respectively over regions where more or less noise values ​​are expected. In this way, false positive / negative regions 1345a, 1345c, and 1345d can be removed from the initial classification results 1335a per frame, per second, etc., to produce a smooth output 1335b with smooth regions 1345e, 1345f, and 1345g.

[0206] The filtering operation within window 1340 can vary between embodiments. In some embodiments, a valid window takes the majority vote among the predictions in the window. It will be understood that such voting can be implemented in various ways, for example, via majority voting logic or a median filter. For clarity, Figure 13BThe illustration shows an example of how this median filter operates for a five-frame window. The median filter arranges the classifications 1350a of each frame appearing in the initial time frame sequence into a new "ascending" order 1350b, and then uses the value at the median position 1355 as the value of the center frame under consideration (since there are only two classification values, it will be understood that selecting the median / middle in this way for odd-numbered frames is equivalent to a majority vote). Alternatively, one could apply a Gaussian filter, such as one used in SciPy. TM The library provides "gaussian_filter1d(input_seq, sigma = 0.7)" for window smoothing.

[0207] Smoothing can also be achieved through Hidden Markov Model (HMM) filtering. In these embodiments, the HMM model parameters can be estimated from data labeled with benchmark truth values ​​using the expectation-maximization of the Baum-Welch algorithm. One will understand various methods for implementing such functions, for example, using the popular hmmlearn. TM Python facilitates unsupervised learning. After examining the annotations of the benchmark truth frames, parameters can be "guessed" by estimating reasonable initial probabilities and transformation matrices. This is based on the example hmmlearn. TM The implementation of Python libraries can be shown in code listings C11-C14:

[0208] hmm_decoder=MultinomialHMM(n_components=2) (C11)

[0209]

[0210] `smoothed = hmm_decoder.decode(sequence.reshape(-1,1))` (C14) where row C12 refers to the transition matrix (e.g., the probability that the next frame is an OUT or IN frame given an OUT state in the first frame—as indicated, in this example, there is a 95% probability that an OUT frame will follow another OUT frame and an IN frame will follow another IN frame). Similarly, row C13 indicates that, given a frame classified as IN by the model, there is a 98% probability that it is actually IN and a 2% probability that it is actually OUT. Similarly, C13 indicates that, given a frame classified as OUT by the model, there is an 88% probability that it is actually OUT and a 12% probability that it is actually IN. One will understand that the probabilities in rows C12 and C13 can be determined in various ways, including visual inspection of the training data. For example, the probabilities in row C13 can be determined by computing a confusion matrix from the model's classification and then normalizing the rows of the confusion matrix. Similarly, the probability in line C12 can be determined by counting all true positive transitions in the dataset (i.e., from IN to OUT in consecutive video frames or from OUT to IN in consecutive frames) and dividing by the total number of such transitions.

[0211] Using Hidden Markov Models (HMMs), the system can traverse along the initial predictions and adjust the output to the most probable classification based on the HMM if the HMM differs from the initial predictions. For example, one would understand that HMMs can be used in conjunction with forward-backward algorithms to smooth the initial predictions.

[0212] Of course, one will also understand that in some embodiments, any revised frame classifications after smoothing may not be applied until after smoothing is complete (i.e., smoothing results may be stored in a temporary buffer because they are only generated and output when the smoothing process is complete). Otherwise, as Figure 13B In the example, once the window encounters four categories with the same value, that value will dominate and be assigned to all subsequent frames.

[0213] Example surgical procedure start / end time detector

[0214] After classification, and in some embodiments after smoothing, the classification results can be provided to the start / end time detector 825e, which can perform... Figure 13D The various operations shown in process 1325. Appendix Figure 13C Depicting in some embodiments, such as that which can be generated by Figure 13DExample internal / external classification signal waveforms (e.g., after smoothing) of the process operation. In example waveform 1305, regions 1310a, 1310c, and 1310e of frames classified as external may reflect actual interoperative time intervals, while regions 1320a, 1320b, and 1320c of frames classified as external may alternatively reflect brief removals and reinsertions of visualization tools. Therefore, the start / end time detector 825e seeks to identify each of intervals 1310b and 1310d as an interval encompassing different surgical cases.

[0215] In box 1330a, the start / end time detector 825e can receive frame classifications and corresponding timestamps, for example, as shown in waveform 1305. In boxes 1330b and 1330c, the detector can then determine whether all frame intervals classified as outside the patient, i.e., intervals 1340a, 1340b, 1340c, 1340d, and 1340e, have been considered. The detector can identify an “interval” as any consecutive classification group. If the system has not yet noticed the starting frame of a surgical case, or if the system has just noticed the last frame of a surgical case and has not yet started a new entry, as demonstrated in box 1330d, then in box 1330e the system can start a new surgical case entry, marking the start of the current “intra” interval as the start time of the surgical case.

[0216] In box 1330f, the system may consider whether the duration of the next “outer” interval exceeds a threshold T1. In some embodiments, T1 may be approximately 29 minutes, as few visualization tools remove amounts exceeding this amount of time from the baseline truth video of the considered surgery, and the inter-operative period typically exceeds this. However, it will be understood that the interval can be adjusted according to context (e.g., a military emergency surgical ward in a war zone may experience much faster turnaround times between trauma surgeries compared to a domestic surgical ward performing elective surgeries). If there is no next “outer” interval or the duration of the next “outer” interval exceeds the T1 threshold, the process may transition back to box 1330b. Conversely, if the next “outer” interval exceeds the threshold T1, then in box 1330g, the end time of the currently considered “internal” interval can be recorded as the end time of the current surgery.

[0217] It will be understood that if the final interval under consideration is an "inner" interval or if the duration of the last considered "outer" interval is less than the T1 threshold, then the currently considered surgery may have no end time in the video (as may happen, for example, when the recorded surgery runs out of a recording medium and continues on a separate recording medium). If this is the case in box 1330h, then the end of the video in box 1330i (i.e., the timestamp of the last frame in the entire video) can be marked as the end of the currently considered surgical case.

[0218] At this point, some embodiments may terminate the start / end time detection process. However, in some embodiments, the system may verify the plausibility of the identified surgical start and stop times (e.g., to remove spurious surgical cases). In these embodiments, the process may continue to block 1330j to consider the start / stop time of each surgical case recorded via previous traversal. For each surgical case considered in block 1330k, the system may verify in block 13301 that the duration of the surgical case is less than a second threshold T2, and if so, remove the surgical case from the created entry group in block 1330m. In some embodiments, T2 may be approximately four minutes, as checks on results from the benchmark ground truth video indicate that human actions between surgeries (e.g., visual tool cleaning) may result in the creation of spurious surgical cases between real surgical cases. It will be understood that the choice of threshold T2 may be influenced by the choice of the window used for smoothing and the surgical context (e.g., some training "surgeries" in a virtual environment may only consist of a few minutes). After considering all surgical cases in block 1330j, post-processing may terminate and the results may be provided to processing unit 825f (e.g., for non-surgical frames to be removed, data to be encrypted, etc.) or stored in storage processing unit 825g. Although the identified cases (e.g., in the case represented as an array of timestamps) may be self-evident, in some embodiments, post-processing may also include explicit identification of the start time 1315a, 1315c and end time 1315b, 1315d of the corresponding surgical procedures.

[0219] Example of an uncertain algorithm

[0220] People will understand the various processes used to determine uncertainty at calculator 940. For example, Figure 14B and Figure 14C Each of them depicts a reference. Figure 14A An example process for measuring uncertainty using the set of hypothetical results in a table. Figure 14BIn example procedure 1400a, the computer system can initialize the retainer variable "max" at block 1405a with the maximum count across all categorized classes (i.e., "IN" or "OUT" categories). Then, as indicated in block 1405b, the system can iterate through each class. While considering each class at block 1405c, the maximum count "max_cnt" for the class can be determined at block 1405d and compared to the current value of the retainer "max" at block 1405e. If max_cnt is larger, then at block 1405f, max can be reassigned to the value of max_cnt.

[0221] For example, refer to Figure 14A The table shows the hypothetical values ​​for classes "within" and "outside", and given the results of five intermediate classifiers, as indicated, the calculator 940 can produce predictions as indicated in the table. Specifically, intermediate model 1 produces a 50% probability that a frame belongs to class "within" and a 50% probability that a frame belongs to class "outside". During the first traversal of box 1405c, the system can consider the class "within" value for each frame. Here, class "within" is the class most predicted by classifiers 1, 2, 3, and 5 (ties are considered as the most predicted result). Since it is the most predicted class in these four sets, "max_cnt" is 4 for the "within" class. Since 4 is greater than 0, the system will assign "max" to 4 in box 1405f. A similar procedure for subsequent traversals determines that the max_cnt value is 2 for class "outside". Since the subsequent "max_cnt" is determined to be less than 4, "max" will remain 4 when processing transitions to box 1405g after considering each class. In this box, the uncertainty can be output, as shown in Equation 4:

[0222]

[0223] Where "model_cnt" represents the number of intermediate classifiers. Continuing on... Figure 14A The example table has five classifier results, and therefore the uncertainty is 1-4 / 5, or 0.2.

[0224] Figure 14C Another example procedure 1400b for calculating uncertainty is depicted. Here, in block 1410a, the system can set the "entropy" holder variable to 0. In blocks 1410b and 1410c, the system can reconsider each class, in block 1410d, determine the mean of that class, and in block 1410e, append the logarithm of the mean to the entropy variable, where the logarithm is base-number of classes. For example, see reference... Figure 14A From the table, people will understand that the average value within a class is

[0225]

[0226] This shows the corresponding mean calculation for classes "outside" the given classes. Once all classes have been considered, in box 1410f, the final uncertainty can be output as the entropy value divided by the negative of the number of classes. Therefore, for Figure 14A The example mean in the table can lead to a final uncertainty value of approximately 0.12.

[0227] People will realize Figure 14C The process involves calculating the Shannon entropy. Specifically, where y c,n This represents the SoftMax prediction output of the c-th class in the n-th frame set.

[0228] As indicated above, this can then be incorporated into the calculation of Shannon's entropy H.

[0229]

[0230] Where Class_Cnt is the total number of classes (e.g., in...). Figure 14A In the table, Class_Cnt is 2). It will be understood that, by convention, "0logciass_cnt 0" is 0 in these calculations.

[0231] People will understand. Figure 14B and Figure 14C The approaches are complementary in some respects, and therefore in some embodiments both can be used and their results averaged.

[0232] For completeness, as discussed, when the fusion model is a generative model, uncertainty can be measured based on the predictions rather than by considering the outputs of multiple models as described above. For example, in Figure 14D In this model, the fusion predictor 935a is a generative model 1425b, configured to receive the connected feature results 1425a and output predictions 1425c and 1425d (i.e., "inside" or "outside" classification). For example, a Bayesian neural network could output a distribution, selecting the distribution with the highest probability as the prediction (here, prediction 1425d). The uncertainty logic here can evaluate the uncertainty based on the variance of the predicted distribution 1425d.

[0233] People will understand additional methods for assessing uncertainty. For example, in the case that the fusion predictor 935a is a neural network, the traversal application during inference with various nodes dropped in the neural network can also produce a distribution similar to the prediction distribution 1425d, and the uncertainty can be calculated based on the variance of that distribution through uncertainty logic.

[0234] Example deployment topology

[0235] As discussed above, it will be understood that components 825a, 825b, 825c, 825d, 825e, 825f, and 825g can reside in many different locations, including all appearing in the same location. For example, Figure 15A This is a schematic diagram illustrating an example component deployment topology 1500a that may be implemented in some embodiments. Here, components 825a, 825b, 825c, and 825d are typically integrated into a single "internal / external processing system" 1505c, while components 825e, 825f, and 825g are integrated into a surgical segmentation system 1505d. In this topology, each of systems 1505c and 1505d resides in the same robotic system or surgical instrument (e.g., on-device computer system, such as those provided by Advantech) present in operating room 1505a. TM Vega-6301 produced TM The 4K HEVC encoder instrument is used in conjunction with the system 1505b. For example, the system may be software code running on the processor of the patient-side cart 130 or the electronics / console 145, or firmware software on the tool 110b. Positioning systems 1505c and 1505d within the operating room 1505a in this manner allows for secure data processing, facilitating the transfer of processed data 1505e to another local computer system 1505h or the transmission of processed data 1505f outside the operating room to a remote system 1505g, free from the regulatory or other obstacles discussed herein. That is, because sensitive portions of the video and the corresponding kinematic and system data have been removed, data 1505e and 1505f may be subject to less stringent regulatory requirements (e.g., encryption) than without such review.

[0236] Therefore, the local computer system 1505h could be, for example, a hospital network server providing access to external service providers or other internal data processing teams. Similarly, the off-site computer system 1505g could be a cloud storage system, a storage device belonging to a third-party service provider or regulatory agency, etc. It will be understood that in some topologies, the segmented system 1505g can be relocated to either the local system 1505h or the off-site system 1505g.

[0237] However, some embodiments contemplate, for example Figure 15BThe topology 1500b is a topology in which one or both of the processing systems 1510d and 1500e (as shown here) are relocated to the local system 1510e, which is not juxtaposed with the robotic system or surgical instrument 1510b, but remains within the control of the operating room or operating mechanism 1510a. This topology can be useful when the anticipated processing is resource-intensive and a dedicated processing system (such as the local system 1510e) can be specifically tailored to perform such processing efficiently (compared to the potentially more limited resources of the robotic system or surgical instrument 1510b). The robotic system or surgical instrument 1510b can now provide the local system 1510e with the initial raw data 1510c (possibly encrypted) for processing.

[0238] Once processed and sensitive data removed, the system can provide data 1510g (which may now be unencrypted) to off-site system 1510h, such as a cloud server, again because the regulatory burden or risk exposure can now be significantly reduced after the review process. Similarly, it will be understood that systems 1510d and 1510f do not necessarily need to coexist as shown in the diagram. For example, the segmented system 1510f could reside on off-site system 1510h. This arrangement might be appropriate when it is certain that system 1510d will remove any sensitive information and off-site system 1510h has greater computational or network resources for additional processing (such as using system 1510f to identify different surgical procedures).

[0239] In some embodiments, processing can be performed entirely on an off-site system 1515e (such as a cloud server system) with considerable and flexible data processing capabilities. Figure 15C As shown, moving processing systems 1515f and 1515g to a cloud system could result in raw data 1515c (including sensitive information) from the robotic system or surgical instrument 1515b leaving the control of the operating room 1515a. This may be suitable, for example, for situations where data is encrypted and / or travels (or is manually delivered) via a controlled network channel to an off-site system 1515e. Figure 15C The 1500c topology is suitable for situations where the processed data will be received by various downstream systems that are also located in the cloud or off-site networks.

[0240] Example of results

[0241] The exemplary implementations of various embodiments are simplified for practice to evaluate the effectiveness of the methods disclosed herein. Figure 16AThis is a table depicting the attributes of two example datasets used in evaluating various example implementations of various embodiments: namely, the DaVinci16 dataset, which includes robot-assisted procedures such as in operating room 100b, and the Cholec80 dataset, which includes endoscopic video recordings from operating rooms such as operating room 100a. Both datasets contain in vivo and out vivo frames. The DaVinci16 dataset consists of 16 video sessions, each video containing daVinci X... TM or Xi TM Surgical procedures performed on a surgical system. Each video was sampled at 30fps and its duration ranged from approximately 20 to 150 minutes. The Cholec80 dataset is a public dataset containing laparoscopic cholecystectomy surgeries. It consists of a total of 80 video sessions sampled at 25fps, with video durations ranging from approximately 7 to 100 minutes. Each video session in the Cholec80 dataset contains a single surgical procedure, while a given video session in the DaVinci16 dataset can contain one or more surgical procedures. All videos from both datasets were downsampled to 1 FPS, and the size of each video frame was resized to a resolution of 128x72 pixels to minimize processing time. Since there is no prior benchmark ground truth knowledge in the Cholec80 dataset, the labels for all endoscopic frames were generated through observation and manual annotation. Four annotators familiar with the procedures depicted in the videos were recruited to watch the videos and label each video frame with a binary label indicating "inside the body" or "outside the body". In total, the DaVinci16 dataset contains 84,622 frames, of which 16,271 are external endoscopic frames, and the Cholec80 dataset contains 176,192 frames, of which 3,444 are external endoscopic frames.

[0242] Figure 16B and Figure 16C Illustrative graphs show example F1 score results using different features and classifiers on two datasets. That is, each graph illustrates the results using data from robot-assisted surgery (…). Figure 16B (DaVinci16 dataset) and laparoscopic surgical data ( Figure 16C The different features derived from the Cholec80 dataset are used as quantitative results for frame recognition of the input. For each dataset, different features (blobs, colors, etc.) are generated, a portion of which is used to train individual intermediate models (logistic regression, random forest, etc., discussed in code listings C6-C10, for example) and the remainder is used to validate their performance and generate F1 scores, as depicted in these figures.

[0243] As indicated, color histograms provided best classification performance on both datasets, with peak recalls of 98% and 96% at the frame and video session levels, respectively. These results also show that network features can achieve similar accuracy for endoscopic frame recognition. Network features achieved average recalls of 78%–89% at the frame level and 77%–91% at the session level. Specifically, VGG features consistently outperformed ResNet features on both datasets, followed by MobileNet features. This suggests that deep features, despite initially being learned from unrelated image data, can still perform well in classifying extracorporeal frames in endoscopic videos. While these results from this example implementation indicate that some models and features perform better than others, it will be understood that such results are unique to the context and implementation chosen for the experiment, and that in other cases, alternative choices of features and models may be more appropriate and produce different results.

[0244] Figure 17 The illustration shows a schematic diagram of example segmented bars used for endoscopic extracorporeal frame recognition in a robot-assisted surgical session on the DaVinci16 and Cholec80 datasets targeting combinations of specific features. Due to... Figure 16B and Figure 16C The random forest model, which performs well in isolation, is used here (both as intermediate model 925 and fusion model 935), but evaluated using different input features. Each feature block contains three rows: depicting the data baseline ground truth (top row), the prediction using the feature type (middle row), and the corresponding difference compared to the baseline ground truth frame label (bottom row). Similarly, it will be understood that these results are unique to the context of the example implementation, and implementations of the same or different embodiments may not produce exactly the same results.

[0245] Two validation sets were run during training. First, performance was evaluated using three-fold cross-validation across the entire DaVinci16 dataset. This setup randomly split all sessions into two sets, where in each fold, 80% of sessions were used for training / validation, and the remaining 20% ​​were reserved for testing. To handle imbalanced samples and achieve robust modeling, minority class observations from the training data in each fold were oversampled. The SMOTE oversampling method (SMOTE(random_state=12, ratio=1.0)) was used, where the "SMOTE" function is learned via imblearn. TMThe library function `imblearn.over_sampling.SMOTE` generates new samples in the underrepresented "outer" class (i.e., extra-endoscopic frames). It will be understood that this oversampling may be suitable for many of these embodiments with such dataset asymmetry as described in this paper. This approach can improve modeling performance when training on imbalanced data and may not leak any test data information into the training. In these examples, all evaluation metrics are computed on the test set in each fold and averaged across the folds to reduce any evaluation bias. The same validation strategy is continued on the Cholec80 dataset, where the training data is oversampled in each fold.

[0246] like Figure 16A As shown, only a relatively small fraction of observations in the dataset are labeled as frames outside the endoscope. Therefore, the aforementioned metrics are computed relative to each class, and the average across all classes is then considered an unbiased rating of the imbalanced data. In addition to frame-level ratings that do not incorporate temporal continuity, session-level reports that assess higher-level performance across video sessions are generated. This is done by first computed the performance metrics in each video and then averaging the metrics across all test video sessions. Session-level ratings can be interpreted as describing how the model generalizes beyond the sample data, i.e., unseen video sessions.

[0247] Similarly, due to Figure 16B and Figure 16C The isolated, well-performing random forest model was used in an additional cross-surgical experiment, where the random forest model (again used as both intermediate model 925 and fusion model 935) was trained on robot-assisted surgical data (DaVinci16) and applied to laparoscopic surgical data (Cholec80) without any model adaptation, and vice versa. This experiment aimed to validate the model's generalizability across both types of surgery. The results are presented in… Figure 18A and Figure 18B In the table.

[0248] Specifically, Figure 18A This is a table comparing the average cross-validation results of different features for frame-level and session-level recognition on DaVinci16 (“Robot” row) and Cholc80 video (“Laparo.” row). Figure 18ATables of cross-surgical outcomes (i.e., models trained on DaVinci16 and applied to Cholec80, or vice versa) suggest that knowledge of endoscopic image features (e.g., color) can also be robust enough to generalize to different surgical types. Therefore, given its scalability and performance, models of various embodiments can be adapted to a wide range of surgical environments. Consequently, once trained, the clinical and technical communities can often use the models to erase endoscopic videos from minimally invasive surgeries. Figure 18A As shown, the example implementation using only the random forest model was able to detect and remove extra-endoscopic frames, with a highest recall of 96.27% in robot-assisted surgery and a highest recall of 92.71% in laparoscopic surgery.

[0249] To further demonstrate cross-domain compatibility, Figure 18B This is a table depicting the performance of cross-surgical classification using the example implementation. To evaluate the performance of the model identifying each endoscopic frame, the average precision, average recall, and average F1 score for each endoscopic frame class were considered. Here, precision is calculated as the ratio between true positives and the total number of positive predictions, recall is the ratio between true positives and the actual number of positives, and the F1 score is calculated as a weighted average of precision and recall as an overall measure of accuracy.

[0250] As indicated, in this specific context, VGG16 features achieved the best performance when transferring a model trained on robot-assisted surgery to laparoscopic surgery, with an average F1 score of 0.75 and an average recall of 75.53%. Conversely, color features provided the highest F1 score of 0.93 and a recall of 96.01% when transferring knowledge from laparoscopic surgery to robot-assisted surgery. This suggests that VGG16 and color vision features can be useful for knowledge transfer across different types of surgery. De-tuning the deep learning model against the clinical data under consideration before feature extraction can improve the performance of the deep learning model. Similarly, more data collection can potentially lead to a more robust model.

[0251] Based on these results, one can decide on the best implementation of the chosen features and intermediate models. For example, given data similar to that considered here, a system could be implemented with only a random forest intermediate model and an SVM, each model receiving a concatenated feature vector of color and VGG feature vectors. It will be understood that, given other datasets, different features and models can be chosen. For example, the model could only receive feature vectors that perform well and only generate feature vectors that produce the desired classification rate when consumed by the appropriate corresponding model.

[0252] Figure 18CThe diagram illustrates the use of F1 scores to measure performance with and without smoothing. Figure 18A and Figure 18B A graph showing a quantitative comparison of classification performance in the case of random forests. For simplicity, the results presented are averages across all features in each surgical dataset. Post-processing resulted in better performance compared to predictions without post-processing, with an average improvement of 2.66% in F1 score on the DaVinci16 dataset and an average improvement of 5.07% on the Cholec80 dataset.

[0253] Computer System

[0254] Figure 19 This is a block diagram of an example computer system that can be used in conjunction with some embodiments. The computing system 1900 may include interconnects 1905 that connect various components, such as one or more processors 1910, one or more memory units 1915, one or more input / output systems 1920, one or more storage systems 1925, one or more network adapters 1930, etc. The interconnects 1905 may be, for example, one or more bridges, traces, buses (e.g., ISA, SCSI, PCI, I2C, Firewire buses, etc.), wires, adapters, or controllers.

[0255] One or more processors 1910 may include, for example, Intel TM Processor chips, math coprocessors, graphics processors, etc. One or more memory components 1915 may include, for example, volatile memory (RAM, SRAM, DRAM, etc.), non-volatile memory (EPROM, ROM, flash memory, etc.), or similar devices. One or more input / output devices 1920 may include, for example, display devices, keyboards, pointing devices, touchscreen devices, etc. One or more storage devices 1925 may include, for example, cloud-based storage, removable USB storage, disk drives, etc. In some systems, memory components 1915 and storage devices 1925 may be the same component. Network adapter 1930 may include, for example, a wired network interface, a wireless interface, Bluetooth, etc. TM Adapters, line-of-sight interfaces, etc.

[0256] People will recognize that in some embodiments, only some, alternative, or all of the components may be present. Figure 19Additional components beyond those depicted. Similarly, in some systems, components may be combined or used for dual purposes. These components can be implemented using dedicated hardwired circuit systems, such as one or more ASICs, PLDs, FPGAs, etc. Therefore, some embodiments may be implemented in, for example, programmable circuit systems (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in dedicated hardwired (non-programmable) circuit systems, or in a combination of these forms.

[0257] In some embodiments, data structures and message structures may be stored or transmitted via network adapter 1930 via a data transmission medium (e.g., signals on a communication link). Transmission can occur across various media, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections. Therefore, "computer-readable medium" can include computer-readable storage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0258] One or more memory components 1915 and one or more storage devices 1925 may be computer-readable storage media. In some embodiments, one or more memory components 1915 or one or more storage devices 1925 may store instructions that can perform or cause to perform the various operations discussed herein. In some embodiments, the instructions stored in memory 1915 may be implemented as software and / or firmware. These instructions may be used to perform operations on one or more processors 1910 to perform the processes described herein. In some embodiments, such instructions may be provided to one or more processors 1910 by, for example, downloading instructions from another system via network adapter 1930.

[0259] Comment

[0260] The accompanying drawings and descriptions are illustrative. Therefore, neither the specification nor the drawings should be construed as limiting this disclosure. For example, titles or subtitles are provided merely for convenience and to facilitate understanding. Therefore, titles or subtitles should not be construed as limiting the scope of this disclosure, for example, by grouping or simply combining features presented in a particular order to facilitate understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the event of conflict, this document (including any definitions provided herein) shall prevail. The use of one or more synonyms herein does not preclude the use of other synonyms. The use of examples anywhere in this specification (including examples of any terms discussed herein) is merely illustrative and not intended to further limit the scope and meaning of this disclosure or any illustrative terms.

[0261] Similarly, although specific representations are made in the accompanying figures, those skilled in the art will understand that the actual data structures used to store information can differ from those shown. For example, data structures may be organized in different ways, may contain more or less information than shown, may be compressed and / or encrypted, etc. Common or well-known details may be omitted from the figures and disclosure to avoid confusion. Similarly, the figures may depict specific sets of operations to facilitate understanding, which are merely examples of a broader set of such operations. Thus, it will be readily apparent that the same purpose or effect depicted in some flowcharts can often be achieved using additional, alternative, or fewer operations. For example, data may be encrypted, although not shown in the figures, items may be considered different loop patterns (“for” loops, “while” loops, etc.), or ordered in different ways to achieve the same or similar effects, etc.

[0262] The references to “embodiment” or “an embodiment” herein mean that at least one embodiment of this disclosure includes the specific features, structures, or characteristics described in connection with that embodiment. Therefore, the phrase “in one embodiment” in various places herein does not necessarily refer to the same embodiment in every one of those places. Individual or alternative embodiments may not be mutually exclusive with other embodiments. It will be appreciated that various modifications can be made without departing from the scope of the embodiments.

Claims

1. A computer-implemented method for classifying video frames into regions depicted inside or outside a patient's body, the method comprising: Acquire multiple features associated with the video frame, which was captured by the surgical system during a surgical procedure; Multiple classification predictions are generated by submitting at least some of the multiple features to one or more intermediate classifiers. as well as The video frames are classified into regions depicted inside or outside the patient's body based on the multiple classification predictions, wherein the classification is used to either white out or cut off video frames depicting regions outside the patient's body.

2. The computer-implemented method of claim 1, wherein the one or more intermediate classifiers include one or more of support vector machines, logistic regression classifiers, random forests, neural networks, and ensemble classifiers.

3. The computer-implemented method according to claim 1, wherein the plurality of features includes network features.

4. The computer-implemented method of claim 3, wherein the plurality of features further includes direct features.

5. The computer-implemented method according to claim 4, wherein, The direct features include one or more of the following: color histogram features, speckle features, image texture features, and image moment features.

6. The computer-implemented method according to claim 4 or claim 5, wherein, The network features include one or more of the features generated from the VGG16 neural network, the features generated from the ResNet50 neural network, and the features generated from the MobileNet neural network.

7. The computer-implemented method of claim 1, wherein classifying the video frame based on the plurality of classification predictions includes classifying the video frame as the most common prediction among the plurality of classification predictions.

8. The computer-implemented method according to claim 1 or claim 7, the method further comprising: The video frame classification is reassigned based on the application of a median filter to the time window for frame classification, wherein the window for frame classification includes the video frame classification.

9. The computer-implemented method according to claim 1, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; and A random forest classifier is configured to receive color blob features.

10. The computer-implemented method according to claim 1, wherein, The one or more intermediate classifiers include: An AdaBoost classifier, configured to receive Hu moment features; and Support vector machines are configured to receive Haralick image texture features.

11. The computer-implemented method according to claim 1, wherein, The one or more intermediate classifiers include: A random forest classifier is configured to receive a set of features of a connection, the set of features of the connection including a first part associated with Hu moment features and a second part associated with Haralick image texture features.

12. The computer-implemented method according to claim 1, wherein, The one or more intermediate classifiers include: The AdaBoost classifier is configured to receive features generated from the VGG19 neural network. A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

13. The computer-implemented method according to claim 1, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; A support vector machine classifier, configured to receive Haralick image texture features; A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

14. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, the method comprising: Multiple features associated with video frames captured by the surgical system during a surgical procedure are acquired; Multiple classification predictions are generated by submitting at least some of the multiple features to one or more intermediate classifiers. as well as The video frames are classified into regions depicted inside or outside the patient's body based on the multiple classification predictions, wherein the classification is used to either white out or cut off video frames depicting regions outside the patient's body.

15. The non-transitory computer-readable medium of claim 14, wherein the one or more intermediate classifiers comprise one or more of support vector machines, logistic regression classifiers, random forests, neural networks, and ensemble classifiers.

16. The non-transitory computer-readable medium of claim 14, wherein the plurality of features includes network features.

17. The non-transitory computer-readable medium of claim 16, wherein the plurality of features further includes direct features.

18. The non-transitory computer-readable medium according to claim 17, wherein, The direct features include one or more of the following: color histogram features, speckle features, image texture features, and image moment features.

19. The non-transitory computer-readable medium according to claim 17 or claim 18, wherein, The network features include one or more of the features generated from the VGG16 neural network, the features generated from the ResNet50 neural network, and the features generated from the MobileNet neural network.

20. The non-transitory computer-readable medium of claim 14, wherein classifying the video frame based on the plurality of classification predictions includes classifying the video frame as the most common prediction among the plurality of classification predictions.

21. The non-transitory computer-readable medium according to claim 14 or claim 20, wherein the method further comprises: The video frame classification is reassigned based on the application of a median filter to the time window for frame classification, wherein the window for frame classification includes the video frame classification.

22. The non-transitory computer-readable medium according to claim 14, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; and A random forest classifier is configured to receive color blob features.

23. The non-transitory computer-readable medium according to claim 14, wherein, The one or more intermediate classifiers include: An AdaBoost classifier, configured to receive Hu moment features; and Support vector machines are configured to receive Haralick image texture features.

24. The non-transitory computer-readable medium according to claim 14, wherein, The one or more intermediate classifiers include: A random forest classifier is configured to receive a set of features of a connection, the set of features of the connection including a first part associated with Hu moment features and a second part associated with Haralick image texture features.

25. The non-transitory computer-readable medium according to claim 14, wherein, The one or more intermediate classifiers include: The AdaBoost classifier is configured to receive features generated from the VGG19 neural network. A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

26. The non-transitory computer-readable medium according to claim 14, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; A support vector machine classifier, configured to receive Haralick image texture features; A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

27. A computer system comprising: At least one processor; At least one memory, the at least one memory including instructions configured to cause a computer system to perform a method, the method comprising: Multiple features associated with video frames captured by the surgical system during a surgical procedure are acquired; Multiple classification predictions are generated by submitting at least some of the multiple features to one or more intermediate classifiers; and The video frames are classified into regions depicted inside or outside the patient's body based on the multiple classification predictions, wherein the classification is used to either white out or cut off video frames depicting regions outside the patient's body.

28. The computer system of claim 27, wherein the one or more intermediate classifiers include one or more of support vector machines, logistic regression classifiers, random forests, neural networks, and ensemble classifiers.

29. The computer system of claim 27, wherein the plurality of features includes network features.

30. The computer system of claim 29, wherein the plurality of features further includes direct features.

31. The computer system according to claim 30, wherein, The direct features include one or more of the following: color histogram features, speckle features, image texture features, and image moment features.

32. The computer system according to claim 30 or claim 31, wherein, The network features include one or more of the features generated from the VGG16 neural network, the features generated from the ResNet50 neural network, and the features generated from the MobileNet neural network.

33. The computer system of claim 27, wherein classifying the video frame based on the plurality of classification predictions includes classifying the video frame as the most common prediction among the plurality of classification predictions.

34. The computer system according to claim 27 or claim 33, wherein the method further comprises: The video frame classification is reassigned based on the application of a median filter to the time window for frame classification, wherein the window for frame classification includes the video frame classification.

35. The computer system according to claim 27, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; and A random forest classifier is configured to receive color blob features.

36. The computer system according to claim 27, wherein, The one or more intermediate classifiers include: An AdaBoost classifier, configured to receive Hu moment features; and Support vector machines are configured to receive Haralick image texture features.

37. The computer system according to claim 27, wherein, The one or more intermediate classifiers include: A random forest classifier is configured to receive a set of features of a connection, the set of features of the connection including a first part associated with Hu moment features and a second part associated with Haralick image texture features.

38. The computer system according to claim 27, wherein, The one or more intermediate classifiers include: The AdaBoost classifier is configured to receive features generated from the VGG19 neural network. A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

39. The computer system according to claim 27, wherein, The one or more intermediate classifiers include: A logistic regression classifier, configured to receive image histogram features; A support vector machine classifier, configured to receive Haralick image texture features; A support vector machine classifier is configured to receive features generated from a ResNet50 neural network; and A random forest classifier is configured to receive features generated from a MobileNet neural network.

40. A method for classifying each visualization tool video frame in a set as computer-implemented, whether inside or outside a patient's body, the method comprising: Receive a plurality of video frames, at least in part, captured using a visualization tool, said video frames being captured by a surgical system during a surgical procedure; Generate multiple direct features from the multiple video frames; Generate multiple neural network features from the multiple video frames; Multiple outputs are generated by applying the direct features and the neural network features to multiple intermediate classifiers; The fusion predictor is used to generate multiple fusion prediction results from the multiple outputs; as well as Each visualization tool video frame in the set of visualization tool video frames is classified based on the multiple fusion prediction results, wherein the classification is used to whiten or excise one or more video frames depicting areas outside the patient's body.

41. The computer-implemented method according to claim 40, the method further comprising: Multiple uncertainty predictions are generated, at least in part, by determining multiple entropies from the multiple outputs.

42. The computer-implemented method according to claim 40 or claim 41, the method further comprising: A windowed filter is applied to produce multiple smooth fused prediction results.

43. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, the method comprising: Receive a plurality of video frames, at least in part, captured using a visualization tool, said video frames being captured by a surgical system during a surgical procedure; Multiple direct features are generated from the multiple video frames, and the direct features include one or more of color histogram features, speckle features, image texture features, and image moment features; Generate multiple neural network features from the multiple video frames; Multiple outputs are generated by applying the direct features and the neural network features to multiple intermediate classifiers; The fusion predictor is used to generate multiple fusion prediction results from the multiple outputs; as well as Each visualization tool video frame in the set of visualization tool video frames is classified based on the multiple fusion prediction results, wherein the classification is used to whiten or excise one or more video frames depicting areas outside the patient's body.

44. The non-transitory computer-readable medium of claim 43, wherein the method further comprises: Multiple uncertainty predictions are generated, at least in part, by determining multiple entropies from the multiple outputs.

45. The non-transitory computer-readable medium according to claim 43 or claim 44, wherein the method further comprises: A windowed filter is applied to produce multiple smooth fused prediction results.

46. ​​A computer system comprising: At least one processor; At least one memory, the at least one memory including instructions configured to cause the computer system to perform a method, the method comprising: Receive a plurality of video frames, at least in part, captured using a visualization tool, said video frames being captured by a surgical system during a surgical procedure; Multiple direct features are generated from the multiple video frames, and the direct features include one or more of color histogram features, speckle features, image texture features, and image moment features; Generate multiple neural network features from the multiple video frames; Multiple outputs are generated by applying the direct features and the neural network features to multiple intermediate classifiers; The fusion predictor is used to generate multiple fusion prediction results from the multiple outputs; and Each visualization tool video frame in the set of visualization tool video frames is classified based on the multiple fusion prediction results, wherein the classification is used to whiten or excise one or more video frames depicting areas outside the patient's body.

47. The computer system of claim 46, wherein the method further comprises: Multiple uncertainty predictions are generated, at least in part, by determining multiple entropies from the multiple outputs.

48. The computer system according to claim 46 or claim 47, wherein the method further comprises: A windowed filter is applied to produce multiple smooth fused prediction results.

Images