Systems and methods for surgical procedure data classification

Abstract

Various disclosed embodiments relate to computer-implemented systems and methods for identifying surgical tasks from surgical data. In some embodiments, an integrated model configured to receive video data, kinematic data, and system event data from the operating room can be implemented. This integrated model can implement modular streams for processing the data, facilitating prediction even when fewer than all data types are available. In some embodiments, smoothing operations can help facilitate more accurate predictions. Various embodiments can be employed in real-time during surgery, providing predictions at intervals of one second.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 116,907, filed November 22, 2020, entitled “SYSTEMS AND METHODS FOR SURGICALDATA CLASSIFICATION,” the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] Various disclosed embodiments relate to computer-implemented systems and methods for identifying surgical tasks from surgical data. Background Technology

[0004] The ever-increasing data collection capabilities of many surgical operating rooms (both those with and without robotic systems) are likely to enable a variety of new improvements and applications. For example, data from surgical robotic systems, endoscopes, and laparoscopic sensors can facilitate the detection of surgical inefficiencies, provide more meaningful feedback to surgeons, identify common characteristics within patient populations, optimize instrument use, and more. These applications can range from offline applications performed after surgery (such as evaluating the performance of multiple surgeons within a hospital system) to real-time applications performed during surgery (such as real-time digital surgeon's assistants or surgical tool optimizers).

[0005] Unfortunately, much of the processing pipeline in these applications requires or benefits from identifying surgical tasks from surgical data. For example, a cloud-based digital assistant might be able to provide real-time advice to a surgeon, but only if the assistant can identify the surgeon's progress during the procedure. While surgical experts can excel at manually identifying tasks within surgical data, relying on human experts to provide such annotations risks introducing human error and subjectivity, is not easily scalable, and is impractical in real-time scenarios such as the real-time assistants described above. However, automated solutions also present their own challenges. While such systems may be more scalable, they must address varying sensor availability in different operating rooms, limited computational resources for real-time applications, and high standards for accurate identification, as incorrect identification can lead to improper bias in downstream machine learning models and the risk of negative patient outcomes in the operating room.

[0006] Therefore, there is a need for systems and methods that can provide accurate and consistent identification of surgical procedure types from surgical data, despite challenges in data availability, data consistency, and the requirement to keep incorrect identifications very low. Attached Figure Description

[0007] The various embodiments described herein can be better understood by referring to the following specific embodiments in conjunction with the accompanying drawings, wherein the same reference numerals indicate the same or similarly functional elements:

[0008] Figure 1A These are schematic views of various elements that appear in the operating room during a surgical procedure, as described in some embodiments.

[0009] Figure 1B These are schematic views of various components that appear in the operating room during a surgical procedure employing a surgical robot, as can be seen in some embodiments.

[0010] Figure 2A It is a schematic Euler diagram depicting the general grouping of machine learning models and methodologies;

[0011] Figure 2B It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example unsupervised learning method;

[0012] Figure 2C It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example supervised learning method;

[0013] Figure 2D It is a description based on Figure 2A A schematic diagram illustrating various operations of a conventional grouping example semi-supervised learning method;

[0014] Figure 2E It is a description based on Figure 2A A schematic diagram illustrating various operations of reinforcement learning methods, using examples of conventional partitioning;

[0015] Figure 2F It is a schematic diagram depicting the relationship between machine learning models, machine learning model architectures, machine learning methodologies, machine learning methods, and machine learning implementation methods;

[0016] Figure 3A It is a schematic depiction of the operation of various aspects of an example support vector machine (SVM) machine learning model architecture;

[0017] Figure 3B It is a schematic depiction of various aspects of the operation of an example random forest machine learning model architecture;

[0018] Figure 3C It is a schematic depiction of various aspects of the operation of an example neural network machine learning model architecture;

[0019] Figure 3D yes Figure 3C A schematic depiction of the possible relationships between inputs and outputs in nodes of an example neural network architecture;

[0020] Figure 3E It is a schematic depiction of example input-output relationship changes that can occur in a Bayesian neural network;

[0021] Figure 3F It is a schematic depiction of various aspects of the operation of an example deep learning architecture;

[0022] Figure 3G It is a schematic depiction of various aspects of the operation of the example integrated architecture;

[0023] Figure 3H It is a schematic block diagram depicting the various operations of the example pipeline architecture;

[0024] Figure 4A It is a schematic flowchart depicting the various operations common to different machine learning model training methods;

[0025] Figure 4B It is a schematic flowchart depicting the various operations common to inference methods of various machine learning models;

[0026] Figure 4C It is a schematic flowchart depicting the various iterative training operations that occur in box 405b in some architectures and training methods.

[0027] Figure 4D It is a schematic block diagram depicting the operation of various machine learning methods where there is no strict distinction between training and inference methods;

[0028] Figure 4E It is a schematic diagram illustrating the relationship between the architecture training method and the inference method;

[0029] Figure 4F It is a schematic block diagram depicting an example relationship between training and inference methods for machine learning models, where the training methods include various data subset operations;

[0030] Figure 4G It is a schematic block diagram depicting an example of decomposing training data into training subsets, validation subsets, and test subsets;

[0031] Figure 4HIt is a schematic block diagram depicting the various operations in the training method incorporated into transfer learning;

[0032] Figure 4I It is a schematic block diagram depicting the various operations in training methods incorporated into online learning;

[0033] Figure 4J It is a schematic block diagram depicting the various components in an example generative adversarial network method;

[0034] Figure 5A These are schematic illustrations of surgical data that may be received at a processing system in some embodiments;

[0035] Figure 5B This is an example task table that can be used in conjunction with various publicly disclosed embodiments;

[0036] Figure 6 This is a schematic diagram of an integrated machine learning model topology, illustrating relationships between model flows that may be implemented in some embodiments to determine task classification of a dataset.

[0037] Figure 7A In some embodiments, such as... Figure 6 Visualization of machine learning model topology; illustrative ensemble machine learning model topology diagram of the example machine learning model used in machine learning model 615a;

[0038] Figure 7B This is a schematic diagram illustrating the operation of a one-dimensional convolutional neural network, as can be applied in some embodiments;

[0039] Figure 7C In some embodiments, such as... Figure 7A A schematic diagram of the machine learning model topology, illustrating an example convolutional neural network (CNN) machine learning model used in an ensemble machine learning model.

[0040] Figure 7D The illustration shows a method for training that can be implemented in some embodiments. Figure 7C The flowcharts depicting various operations in the process of model topology are shown in the text.

[0041] Figure 8A In some embodiments, such as... Figure 7A A schematic block diagram of the recurrent neural network (RNN) model used in the ensemble machine learning model;

[0042] Figure 8B yes Figure 8A A schematic diagram of the RNN model unfolding over time;

[0043] Figure 8CThis is a schematic block diagram of a Long Short-Term Memory (LSTM) unit that may be used in some embodiments;

[0044] Figure 8D This is a schematic block diagram illustrating model topology variations that may be used in some embodiments, combining convolutional and LSTM layers;

[0045] Figure 9A This is a flowchart illustrating the various operations used in training a video frame recognition model.

[0046] Figure 9B This is a flowchart illustrating the various operations used in training a video frame recognition model.

[0047] Figure 10A In some embodiments, such as... Figure 6 A schematic diagram of the machine learning model topology used in the operator-side kinematic machine learning model 615b or the patient-side kinematic machine learning model 615c.

[0048] Figure 10B The illustration shows an example of training that can be applied in some embodiments. Figure 10A The flowchart of various operations in the model process;

[0049] Figure 11A In some embodiments, such as... Figure 6 A schematic diagram of the machine learning model topology used in the event classification model 615d.

[0050] Figure 11B The illustration shows an example of training that can be applied in some embodiments. Figure 11A The flowchart of various operations in the model process;

[0051] Figure 12A This is a list of example texts of JSON extracts depicting portions of raw surgical data that may be processed in some embodiments;

[0052] Figure 12B This diagram illustrates how raw data (such as...) can be used to... Figure 12A A flowchart illustrating the various operations involved in converting the data depicted in the diagram into feature vector form;

[0053] Figure 13A This is an illustrative machine learning model topology diagram, which may be implemented in some embodiments as an incorporation and fusion classification model or logic.

[0054] Figure 13B The illustration shows an example of training that can be applied in some embodiments. Figure 13AThe flowchart of various operations in the model process;

[0055] Figure 14A Is it possible? Figure 14B and Figure 14C A table of abstract example classification results considered in the uncertainty calculation;

[0056] Figure 14B This is a flowchart illustrating various operations in a process of calculating uncertainty using class counting, as may be implemented in some embodiments.

[0057] Figure 14C This is a flowchart illustrating various operations in a process of calculating uncertainty using entropy, as may be implemented in some embodiments.

[0058] Figure 14D This is an illustrative depiction of uncertain results using a generative machine learning model, as may be employed in some embodiments;

[0059] Figure 15 This is a flowchart illustrating various operations for incorporating uncertainty into an ongoing task prediction process, as may be implemented in some embodiments.

[0060] Figure 16A This is a flowchart illustrating various operations in a classification process with smoothing, as may be applied in some embodiments;

[0061] Figure 16B This is a schematic diagram illustrating a classification smoothing operation that may be applied in some embodiments;

[0062] Figure 16C It is a state transition diagram illustrating a set of assumptions about task transition operations that may be implemented in some embodiments;

[0063] Figure 16D It is used for Figure 16C Example state transition probability matrix of the transition graph;

[0064] Figure 16E It is used for Figure 16C Example mission state emission probability matrix in the transition graph;

[0065] Figure 16F It is used for Figure 16C Example task state start probability set in the transition graph;

[0066] Figure 17A Is it possible to use in Figure 17B , Figure 17C and Figure 17E A schematic diagram of annotated surgical training data used to determine task probabilities during the process;

[0067] Figure 17B This is a flowchart illustrating various operations for determining the probability of task initiation, as may be implemented in some embodiments;

[0068] Figure 17C This is a flowchart illustrating various operations for determining a task transition probability matrix, as may be implemented in some embodiments;

[0069] Figure 17D The illustration is as follows: Figure 17C A schematic diagram illustrating the changes in tasks over time during the process;

[0070] Figure 17E This is a flowchart illustrating various operations for determining a mission launch probability matrix, as may be implemented in some embodiments.

[0071] Figure 18A It is a table describing the characteristics of an example dataset used for training and validating example implementations;

[0072] Figure 18B This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on a prostatectomy dataset.

[0073] Figure 18C This is a bar chart showing the average recall accuracy results for various tasks on an example implementation model trained on a pig dataset.

[0074] Figure 19A This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on a cholecystectomy dataset.

[0075] Figure 19B This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on a hysterectomy dataset.

[0076] Figure 19C This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on an inguinal hernia dataset; and

[0077] Figure 20 This is a block diagram of an example computer system that can be used in conjunction with some embodiments.

[0078] The specific examples depicted in the accompanying drawings have been chosen for ease of understanding. Therefore, the disclosed embodiments should not be limited to the specific details in the drawings or the corresponding disclosure. For example, the drawings may not be drawn to scale, the dimensions of certain elements may have been adjusted for ease of understanding, and the operations of embodiments associated with flowcharts may include additional, alternative, or fewer operations compared to those depicted herein. Consequently, some components and / or operations may be separated into different blocks or combined into a single block in a manner different from that depicted. The embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosed examples, and not to limit the embodiments to the specific examples described or depicted. Detailed Implementation

[0079] Example of an operating room overview

[0080] Figure 1A This is a schematic view of various elements appearing in the surgical room 100a during a surgical procedure, as can be seen in some embodiments. In particular, Figure 1A A non-robotic surgical operating room 100a is depicted, in which a patient-side surgeon 105a performs procedures on a patient 120 with the assistance of one or more assistants 105b, who may be surgeons, physician assistants, nurses, technicians, etc. The surgeon 105a may use a variety of tools to perform the procedures, such as visualization tools 110b (e.g., laparoscopic ultrasound or endoscope) and mechanical end effectors 110a (e.g., scissors, retractors, dissecters), etc.

[0081] The visualization tool 110b provides the surgeon 105a with an internal view of the patient 120, for example, by displaying visualization output from a camera mechanically and electrically coupled to the visualization tool 110b. The surgeon can view the visualization output, for example, through an eyepiece coupled to the visualization tool 110b or on a display 125 configured to receive the visualization output. For example, in the case where the visualization tool 110b is an endoscope, the visualization output can be a color or grayscale image. The display 125 can allow the assistant 105b to monitor the progress of the surgeon 105a during surgery. The visualization output from the visualization tool 110b can be recorded and stored for future review, for example, by capturing the visualization output in parallel with it being provided to the display 125 using hardware or software on the visualization tool 110b itself, or by capturing the output from the display 125 as soon as it appears on the screen, etc. While this document can broadly discuss two-dimensional video capture using visualization tool 110b, when visualization tool 110b is an endoscope, it will be understood that in some embodiments, visualization tool 110b can capture depth data in lieu of or append to two-dimensional image data (e.g., using a laser rangefinder, stereo vision, etc.). Therefore, it will be understood that when such three-dimensional depth data is available, it is possible to apply the two-dimensional operations discussed herein (with necessary modifications) to such three-dimensional depth data. For example, the input to a machine learning model can be extended or modified to accept features derived from such depth data.

[0082] A single surgical procedure may include the execution of several sets of actions, each set of actions forming a discrete unit referred to herein as a task. For example, locating a tumor may constitute a first task, resecting the tumor may constitute a second task, and closing the surgical site may constitute a third task. Each task may include multiple actions; for example, a tumor resection task may require several cutting actions and several cauterizing actions. While some surgical procedures require tasks to assume a specific order (e.g., resection occurs before closure), in some surgical procedures the order and presence of some tasks may allow for variation (e.g., eliminating prophylactic tasks or reordering resection tasks where the order is invalid). Transitions between tasks may require surgeon 105a to remove tools from the patient, replace tools with different tools, or introduce new tools. Some tasks may require removing visualization tools 110b and repositioning visualization tools 110b relative to their position in a previous task. While some assistants 105b may assist in performing surgically related tasks, such as administering anesthesia 115 to patient 120, assistants 105b may also assist in these task transitions, for example, anticipating the need for new tools 110c.

[0083] Technological advancements have enabled, for example Figure 1AThe described program can also be executed by a robotic system, as well as programs that cannot be executed in a non-robotic operating room 100a. Specifically, Figure 1B This is as can be seen with respect to some embodiments occurring in the use of surgical robots (such as da Vinci). TM A schematic view of various components present in the operating room 100b during a surgical procedure using a surgical system. Here, a patient-side trolley 130, having tools 140a, 140b, 140c, and 140d respectively attached to each of a plurality of arms 135a, 135b, 135c, and 135d, can occupy the position of the patient-side surgeon 105a. As previously described, tools 140a, 140b, 140c, and 140d may include visualization tools 140d, such as endoscopes, laparoscopic ultrasound, etc. The operator 105c, who may be a surgeon, can view the output of the visualization tools 140d via a display 160a on a surgeon's console 155. By manipulating a handheld input mechanism 160b and a pedal 160c, the operator 105c can remotely communicate with the tools 140a-140d on the patient-side trolley 130 to perform surgical procedures on the patient 120. In fact, in some embodiments, because communication between the surgeon's console 155 and the patient-side cart 130 can occur across a remote communication network, the operator 105c may or may not be in the same physical location as the patient-side cart 130 and the patient 120. The electronics / console 145 may also include a display 150 that depicts the patient's vital signs and / or the output of the visualization tool 140d.

[0084] Similar to the task shifts in non-robotic surgical operating room 100a, surgical procedures in operating room 100b may require the removal or replacement of tools 140a-140d (including visualization tool 140d) for various tasks, as well as the introduction of new tools, such as new tool 165. As previously stated, as the surgical procedure progresses, one or more assistants 105d can now anticipate such changes and make any necessary adjustments together with the operator 105c.

[0085] Similar to the non-robotic surgical operating room 100a, outputs from visualization tools 140d can be recorded here, such as from the patient-side trolley 130, the surgeon's console 155, and the monitor 150. While some tools 110a, 110b, and 110c in the non-robotic surgical operating room 100a can record additional data, such as temperature, motion, conductivity, and energy levels, the presence of the surgeon's console 155 and the patient-side trolley 130 in operating room 100b facilitates the recording of far more data than just the outputs from visualization tools 140d. For example, operator manipulation of the handheld input mechanism 160b by the operator 105c, activation of the pedal 160c, and eye movements within the monitor 160a can all be recorded. Similarly, the patient-side trolley 130 can record tool activations (e.g., application of radiant energy, closure of scissors, etc.) and end effector movements throughout the surgical procedure.

[0086] Machine Learning Fundamentals - Overview

[0087] This section provides a basic description of the machine learning model architectures and methods that may be associated with the various embodiments disclosed. Machine learning encompasses a broad, heterogeneous field and has undergone many abrupt and overlapping developments. Given this complexity, practitioners do not always use terminology consistently or with strict clarity. Therefore, this section aims to provide common ground to better ensure that the reader understands the substance of the disclosed embodiments. It will be understood that it is not feasible to exhaustively address all known possible variations of all known machine learning models and their architectures, tasks, methods, and methodologies. Rather, it will be understood that the examples discussed herein are merely representative and that the various embodiments disclosed may employ many other architectures and methods different from those explicitly discussed.

[0088] In order to orient readers relative to existing literature, Figure 2A The commonly accepted groupings of machine learning models and methodologies (also known as techniques) are depicted in the form of schematic Euler diagrams. In contrast to... Figure 2F Before providing a more comprehensive description of the field of machine learning, we will refer to... Figures 2B to 2E Describe them in their conventional way Figure 2A Grouping is used to orient readers.

[0089] Figure 2A The common grouping of machine learning models and their methodologies is usually based on the nature of the inputs the model is expected to receive or the methodology on which the model is expected to operate. Unsupervised learning methodologies derive inferences from input datasets lacking output metadata (also known as "unlabeled data"), or by ignoring such metadata if it exists. For example, as in Figure 2BAs shown, the unsupervised K-Nearest Neighbors (KNN) model architecture can receive multiple unlabeled inputs (represented by circles in feature space 205a). The feature space is the mathematical space of the inputs, on which a given model architecture is configured to operate. For example, if a 128x128 grayscale pixel image is provided as input to a KNN, it can be viewed as a linear array of 16,384 "features" (i.e., the original pixel values). Then the feature space would be a 16,384-dimensional space (…). Figure 2B (Only a two-dimensional space is shown for ease of understanding). Conversely, if a Fourier transform is applied to pixel data, for example, the resulting frequency amplitude and phase can be input as "features" into the model architecture. Although input values ​​in feature space are sometimes referred to as feature "vectors," it will be understood that not all model architectures are expected to receive feature inputs in a linear form (e.g., some deep learning networks expect input features to be matrices or tensors). Therefore, unless otherwise indicated by the context, references to feature vectors, feature matrices, etc., should be considered as examples of possible forms that can be input into the model architecture. Similarly, references to "input" will be understood to include any possible type or form of features acceptable to the architecture. Continuing Figure 2B For example, a KNN classifier can output the association between an input vector and various groups determined by the KNN classifier, as indicated by the squares, triangles, and hexagons in the figure. Therefore, unsupervised methodologies can include, for example, determining clusters in the data, reducing or changing the feature dimensions used to represent the data input, as in this example.

[0090] Supervised learning models receive an input dataset along with output metadata (called "labeled data") and modify the parameters of the model architecture (such as the biases and weights of a neural network, or the support vectors of an SVM) based on this input data and metadata to better map subsequent inputs to the desired output. For example, an SVM supervised classifier can be like... Figure 2C The operation is as shown, receiving multiple input feature vectors, represented by circles, as training inputs in feature space 210a, where each feature vector is accompanied by an output label A, B, or C, for example, as provided by the practitioner. According to supervised learning methodologies, the SVM uses these labeled inputs to modify its parameters such that when the SVM receives a new, previously unseen input 210c in the form of feature vectors from feature space 210a, the SVM can output a determined desired classification "C" in its output. Therefore, supervised learning methodologies can include, for example, performing classification, performing regression, etc., as in this example.

[0091] Semi-supervised learning methodologies rely on both labeled and unlabeled data to inform the parameter adjustments of their model architecture. For example, supervised neural network classifiers can... Figure 2DThe operation is illustrated by receiving some training input feature vectors labeled with class A, B, or C in feature space 215a, and some training input feature vectors without such labels (e.g., depicted by circles lacking letters). Without considering the unlabeled inputs, a naive supervised classifier can distinguish inputs in class B and class C based on a simple planar separation 215d in the feature space between the available labeled inputs. However, by considering both unlabeled and labeled input feature vectors, a semi-supervised classifier can employ a more subtle separation 215e. Unlike the simple separation 215d, the subtle separation 215e can correctly classify a new input 215c as class C. Therefore, semi-supervised learning methods and architectures can include applications of both supervised and unsupervised learning, where at least some of the available data is labeled.

[0092] at last, Figure 2A The conventional grouping of reinforcement learning methodologies distinguishes them into those in which an agent (e.g., a robot or digital assistant) takes actions (e.g., moving a manipulator, making suggestions to the user) that influence the agent's environmental context (e.g., the object's position in the environment, the user's preferences, etc.) to precipitate a new environmental state and some associated environment-based rewards (e.g., a positive reward if the environmental object is now closer to the target state, a negative reward if the user is unhappy, etc.). Therefore, reinforcement learning can include, for example, updating digital assistants (based on user behavior and expressed preferences), autonomous robots maneuvering in factories, and computers playing chess.

[0093] As mentioned earlier, although many practitioners will recognize Figure 2A The conventional classification method, but Figure 2A The grouping of these groups obscures the rich diversity of machine learning and may not adequately represent machine learning architectures and techniques that fall into multiple groups or fall entirely outside of them (e.g., random forests and neural networks can be used for both supervised and unsupervised learning tasks; similarly, some generative adversarial networks, while employing supervised classifiers, do not easily fall into any of these groups). Figure 2A (within any group). Therefore, although this article can refer to Figure 2A Various terms are used to facilitate the reader's understanding, but this description should not be limited to... Figure 2A A convention that demands uniformity. For example, Figure 2F A more flexible machine learning classification method is proposed.

[0094] Specifically, Figure 1F approximates machine learning as including model 220a, model architecture 220b, methodology 220e, method 220d, and implementation 220c. At a high level, model architecture 220b can be considered as species of their corresponding genus model 220a (model A has possible architectures A1, A2, etc.; model B has possible architectures B1, B2, etc.). Model 220a refers to a description of a mathematical structure suitable for implementation as a machine learning architecture. For example, KNN, neural networks, SVM, Bayesian classifiers, principal component analysis (PCA), etc., represented by boxes “A,” “B,” “C,” etc., are examples of models (ellipses in the figure indicate the presence of additional items). While models can specify general computational relationships, such as SVM including hyperplanes, neural networks having layers or neurons, etc., models can also omit specific architecture structures, such as the architecture's choice of hyperparameters and data flows to perform a specific task, such as SVM using radial basis function (RBF) kernels, neural networks being configured to receive inputs of dimension 256x256x3, etc. These structural features can be selected by practitioners or derived from the training or configuration process, for example. Note that the domain of model 220a also includes combinations of its members, for example, when creating an ensemble model (see below). Figure 3G (Discussion) or when using a model pipeline (see below for details) Figure 3H discuss).

[0095] For clarity, it will be understood that many architectures include both parameters and hyperparameters. Architectural parameters refer to the configuration values ​​of the architecture, which can be adjusted directly based on the received input data (such as adjusting the weights and biases of a neural network during training). Different architectures can have different parameter choices and relationships between them, but changes to parameter values ​​(e.g., during training) are not considered changes to the architecture. In contrast, architectural hyperparameters refer to the configuration values ​​of the architecture that are not directly adjusted based on the received input data (e.g., the K neighbors in a KNN implementation, the learning rate in a neural network training implementation, the kernel type of an SVM, etc.). Therefore, changing hyperparameters generally changes the architecture. It will be understood that some methodological operations (such as validation discussed below) can adjust hyperparameters during training, and thus adjust the architecture type. Therefore, some implementations can consider multiple architectures, although only some of these can be configured to be used or used at a given time.

[0096] Similar to models and architectures, at a higher level, methods 220d can be considered species belonging to the genus Methodology 220e (Methodology I has methods I.1, I.2, etc.; Methodology II has methods II.1, II.2, etc.). Methodology 220e refers to algorithms adapted as methods for performing tasks using one or more specific machine learning architectures, such as training architectures, testing architectures, validating architectures, performing architecture inference, using multiple architectures in generative adversarial networks (GANs), etc. For example, gradient descent is a methodology describing methods for training neural networks, ensemble learning is a methodology describing methods for training groups of architectures, etc. While a methodology can specify general algorithmic operations, such as gradient descent taking traversal steps along a cost or error surface, ensemble learning considering intermediate results of its architecture, etc., a method specifies how a particular architecture should execute the algorithm of the methodology; for example, gradient descent takes traversal backpropagation on a neural network and performs stochastic optimization via Adam with specific hyperparameters, ensemble systems include random forest ensembles applying AdaBoost with specific configuration values, training data being organized into a specific number of folds, etc. When people use additional or modified features to enhance existing architectures or methods (e.g., GAN architectures and GAN training methods might be considered as encompassing deep learning architectures and deep learning training methods), they will understand that architectures and methods themselves can have sub-architectures and sub-methods. They will also realize that not all possible methodologies will apply to all possible models (e.g., suggesting gradient descent on a PCA architecture without further explanation seems absurd). They will understand that methods can include some actions of the practitioner or can be fully automated.

[0097] As the examples above demonstrate, when one moves from models to architectures and from methodologies to methods, aspects of architecture can appear in methods, and aspects of methods can appear in architectures, because some methods may only be applicable to certain architectures, and some architectures may only be suitable for certain methods. Recognizing this interaction, implementation 220c is a combination of one or more architectures with one or more methods to form a machine learning system configured to perform one or more specific tasks, such as training, inference, generating new data using GANs, etc. For clarity, the architecture of this implementation does not need to actively execute its methods, but can simply be configured to execute methods (e.g., as when accompanying training control software is configured to pass input through the architecture). Applying the method will result in the execution of tasks, such as training or inference. Therefore, Figure 2FHypothetical implementation A (indicated by "Imp.A") described herein comprises a single architecture and a single method. This could correspond, for example, to an SVM architecture configured to identify objects in a 128x128 grayscale image using a hyperplane support vector separation method employing an RBF kernel in a 16,384-dimensional space. The use of the RBF kernel and the choice of feature vector input structure reflect both the choice of architecture and the choice of training and inference methods. Thus, it will be understood that some description of the architecture structure can imply aspects of the corresponding method, and vice versa. Hypothetical implementation B (indicated by "Imp.B") could correspond, for example, to training method II.1, which could switch between architectures B1 and C1 based on validation results before applying inference method III.3.

[0098] The close relationship between the architecture and methods within the implementation method facilitates Figure 2A Many ambiguities arise because these groups do not easily capture the close relationship between methods and architectures in a given implementation. For example, very minor changes in a method or architecture can lead to ambiguity. Figure 2A The inter-group movement model implementation involves, for example, a practitioner training a random forest using a first method that incorporates labeled (supervised) data, and then applying a second method using the trained architecture to detect clusters in unlabeled (unsupervised) data, rather than performing inference on the data. Similarly, Figure 2A Grouping can make it difficult to categorize aggregation methods and architectures, for example, as shown below regarding... Figure 3F and Figure 3G The aggregation methods and architectures discussed can be applied to Figure 2A Some, none, or all of the techniques were found in the groups. Therefore, the following sections refer to... Figures 3A to 3G and Figures 4A to 4J The relationships between various example model architectures and example methods are discussed to promote clarity and reader identification of the relationships between architectures, methods, and implementations. It will be understood that the tasks discussed are exemplary, and references to, for example, classification operations are made to facilitate understanding; this should not be construed as suggesting that the implementation must be specifically designed for this purpose.

[0099] For clarity, it will be understood that, without explicit linguistic indication, the above is relative to... Figure 2FThe explanations provided are merely intended to aid the reader's understanding and therefore should not be interpreted in a restrictive manner. For example, one will naturally understand that "method" 220d is a computer-implemented method, but not all computer-implemented methods are methods in the sense of "method" 220d. A computer-implemented method can be logic without any machine learning functionality. Similarly, the term "methodology" is not always used in the sense of "methodology" 220e, but can refer to a method without machine learning functionality. Likewise, while the terms "model," "architecture," and "implementation method" are used in 220a, 220b, and 220c above, in the absence of such an effect in the language, these terms are not limited to their meanings in the sense of "methodology" 220d. Figure 2F The difference is that it can be used to refer to the topology of machine learning components in general.

[0100] Machine Learning Fundamentals - Example Implementation

[0101] Figure 3A This is a schematic depiction of the operation of an example SVM machine learning model architecture. At a high level, given data from two classes (e.g., images of dogs and images of cats) as input features, in... Figure 3A The diagram, represented by circles and triangles, shows that the SVM seeks to determine a hyperplane separator 305a that maximizes the minimum distance from each class member to the separator 305a. Therefore, the training feature vector 305f has a minimum distance 305e to the separator 305a across all its peers. Conversely, the training feature vector 305g has a minimum distance 305h to the separator 305a across all its peers. Thus, the margin 305d formed between these two training feature vectors is a combination of distances 305h and 305e (reference lines 305b and 305c are provided for clarity), and as a maximal-minimum separator, it identifies the training feature vectors 305f and 305g as support vectors. While this example describes linear hyperplane separation, different SVM architectures accommodate different kernels (e.g., RBF kernels), which can facilitate non-linear hyperplane separation. The separator can be found during training, and subsequent inference can be achieved by considering the position of new inputs in the feature space relative to the separator. Similarly, while this example depicts two-dimensional feature vectors (in a two-dimensional plane of paper) for clarity, one will understand that the architecture may accept features with more dimensions (e.g., a 128x128 pixel image might have 16,384 dimensions as input). Although the hyperplane in this example separates only two classes, multi-class separation can be achieved in various ways, such as using an integrated architecture with SVM hyperplane separation in one-to-one, one-to-all, and other configurations. Practitioners often use LIBSVM when implementing SVM. TM and scikit-learn TMThe library. People will understand many different machine learning models (such as logistic regression classifiers) that seek to identify the separating hyperplane.

[0102] In the example SVM implementation above, practitioners determine the feature format as part of the implementation's architecture and methodology. For some tasks, it may be desirable to process the input to determine the new or different feature formats themselves. In fact, some random forest implementations can adjust the feature space representation in this way. For example, Figure 3B An example random forest model architecture comprising multiple decision trees 310b is depicted at a high level, each decision tree receiving all or part of the input feature vector 310a at its root node. Although three trees are shown in this example architecture, with a maximum depth of three levels, it will be understood that forest architectures with fewer or more trees and different levels (even between trees within the same forest) are possible. When each tree considers its input portion, it refers all or part of the input to subsequent nodes, such as path 310f, based on whether the input portion satisfies or does not satisfy conditions associated with various nodes. For example, when considering an image, a single node in a tree can query whether the pixel value at a certain point in the feature vector is above or below a certain threshold. In addition to the threshold parameter, some trees may also contain additional parameters, and their leaves may contain the probability of correct classification. Each leaf of a tree may be associated with a provisional output value 310c for consideration by a voting mechanism 310d to produce a final output 310e, for example, by majority voting in the trees or by a probability-weighted average of the predictions from each tree. This architecture can be applied to various training methods, such as training different subsets of data on different trees.

[0103] The tree depth and variety of trees in a random forest can facilitate the model's consideration of feature relationships rather than directly comparing features in the initial input. For example, if the initial features are pixel values, the tree can identify relationships between task-related groups of pixel values, such as the relationship between "nose" and "ear" pixels in cat / dog classification. However, binary decision tree relationships may impose limitations on the ability to discern these "higher-order" features.

[0104] like Figure 3C The neural networks in the example architecture can also infer higher-order features and relationships between the initial input vectors. However, each node in the network can be associated with various parameters and connections to other nodes, facilitating more complex decision-making and intermediate feature generation than the binary relationships of a regular random forest tree. Figure 3CAs shown, a neural network architecture may include an input layer, at least one hidden layer, and an output layer. Each layer comprises a set of neurons that can receive multiple inputs and provide output values, also known as activation values. The final output value 315b of the output layer is used as the final result of the network. Similarly, the input 315a of the input layer can be received from input data instead of the previous neuron layers.

[0105] Figure 3D Depicting Figure 3C The input and output relationship at node 315c. Specifically, the output n of node 315c... out It can be related to its three (zero-based indexed) inputs as follows:

[0106]

[0107] Where w i These are the weight parameters for the output of the i-th node in the input layer, n. i is the output value of the activation function from the i-th node in the input layer, b is the bias value associated with node 315c, and A is the activation function associated with node 315c. Note that in this example, the sum is for each of the three input layer node output and weight pairs, and only a single bias value b is added. The activation function A can determine the node's output based on the weight value, bias value, and the value of the previous layer node. During training, each of the weight and bias parameters can be adjusted depending on the training method used. For example, many neural networks employ a methodology called backpropagation, where, in some forms, the weights and bias parameters are randomly initialized, the training input vector is passed through the network, and the difference between the network's output value and the expected output value of the vector's metadata is determined. This difference can then be used as a metric to adjust the network parameters, propagating the error as a correction throughout the network so that the network is more likely to produce the correct output when it encounters an input vector in the future. Although for clarity... Figure 3CThe example implementation shows three nodes in the input layer, but one will understand that more or fewer nodes may exist in different architectures (e.g., there might be 16,384 such nodes to receive pixel values ​​in the 128x128 grayscale image example above). Similarly, while each layer in this example architecture is shown as fully connected to the next layer, one will understand that other architectures may not connect every node between layers in this way. Not all neural network architectures process data entirely from left to right, nor do they consider only a single feature vector at a time. For example, recurrent neural networks (RNNs) encompass a class of neural network methods and architectures that consider previous input instances when considering the current instance. Architectures can be further differentiated based on the activation functions used at various nodes, such as: logistic functions, corrected linear unit functions (ReLU), softplus functions, etc. Therefore, there are considerable differences between architectures.

[0108] People will recognize that many of the example machine learning implementations discussed so far in this overview are “discriminative” machine learning models and methodologies (SVM, logistic regression classifiers, etc.). Figure 3D (e.g., neural networks with nodes in the network). Typically, a discriminative approach assumes a form that seeks the probability of finding the following equation 2:

[0109] P(output|input)(2) That is, these models and methodologies seek structural classification (e.g., SVM hyperplane) and estimate the parameters associated with that structure based on training data (e.g., support vectors determine the separating hyperplane). However, it will be understood that not all models and methodologies discussed in this paper can assume this discriminative form, but rather one of several “generative” machine learning models and corresponding methodologies (e.g., Naive Bayes classifiers, Hidden Markov Models, Bayesian Networks, etc.). These generative models instead assume a form that seeks the probability of finding the following Equation 3:

[0110] P(output), P(input|output)(3)

[0111] In other words, these models and methodologies seek to reflect the characteristic relationship between inputs and outputs (e.g., Bayesian neural networks, their initial parameters, and priors), estimate these parameters from the training data, and then compute the value of Equation 2 using Bayesian rules. It will be understood that directly performing these computations is not always feasible, and therefore numerical approximation methods can be employed in some of these generative models and methodologies.

[0112] Readers will understand that such generative methods can be used in this paper (with necessary modifications) to achieve the results exhibited by discriminative implementation methods, and vice versa. For example, Figure 3E The illustration shows an example node 315d that can appear in a Bayesian neural network. Unlike node 315c, which simply receives numerical values, it will be understood that nodes in a Bayesian neural network (such as node 315d) can receive weighted probability distributions 315f, 315g, 315h (e.g., parameters of such distributions) and can output distribution 315e themselves. Therefore, it will be recognized that while classification uncertainty in a discriminative model can be determined, for example, via various post-processing techniques (e.g., comparing the output of a discarded ergonomic application with that of a discriminative neural network), a similar measurement of uncertainty can be achieved by taking the output probability distribution of a generative model, for example, by considering the variance of distribution 315e. Therefore, just as references to a particular machine learning implementation herein are not intended to exclude any alternatives to similar functional implementations, references to discriminative implementations herein should not be construed as excluding alternatives to generative counterparts, where applicable.

[0113] Returning to the general discussion of machine learning methods, although Figure 3C An example neural network architecture with a single hidden layer is depicted, but many neural network architectures can have more than one hidden layer. Some networks with many hidden layers have produced surprisingly efficient results, and the term "deep learning" has been applied to these models to reflect the large number of hidden layers. Here, deep learning refers to the architecture and approach that employs at least one neural network architecture with more than one hidden layer.

[0114] Figure 3F This is a schematic depiction of the operation of an example deep learning model architecture. In this example, the architecture is configured to receive a two-dimensional input 320a, such as a grayscale image of a cat. When used for classification, as in this example, the architecture can typically be divided into two parts: a feature extraction part, which includes a series of layer operations; and a classification part, which determines the output value based on the relationships between the extracted features.

[0115] Many different feature extraction layers are possible, such as convolutional layers, max-pooling layers, dropout layers, cropping layers, etc., and many of these layers are themselves susceptible to variations (e.g., 2D convolutional layers, 3D convolutional layers, convolutional layers with different activation functions, etc.) and different methods and methodologies used for network training, inference, etc. As shown in the figure, these layers can produce multiple intermediate values ​​320b-320j of different dimensions, and these intermediate values ​​can be processed along multiple paths. For example, initially, the grayscale image 320a can be represented as a feature input tensor of dimension 128x128x1 (e.g., a grayscale image 128 pixels wide and 128 pixels high) or a feature input tensor of dimension 128x128x3 (e.g., an RGB image 128 pixels wide and 128 pixels high). Multiple convolutions with different kernel functions in the first layer can contribute multiple intermediate values ​​320b from this input. These intermediate values ​​320b can themselves be considered by two different layers to form two new intermediate values ​​320c and 320d along separate paths (although two paths are shown in this example, one will understand that more paths or a single path is possible in different architectures). Additionally, when an image has red, green, and blue values ​​for each pixel, for example, the "x3" dimension in a 128x128x3 feature tensor (for clarity, this input has three "tensor" dimensions, but 49,152 individual "feature" dimensions), data can be provided in multiple "channels." Various architectures can operate on channels individually or collectively in various layers. The ellipses in the figure indicate the presence of additional layers (e.g., some networks have hundreds of layers). As shown, the size and dimension of intermediate values ​​can change, for example, after pooling, as in value 320e. In some networks, intermediate values ​​can be considered at layers between paths, as shown between intermediate values ​​320e, 320f, 320g, and 320h. Ultimately, the final feature set appears at intermediate sets 320i and 320j and is fed into a set of one or more classification layers 320k and 320l, for example, via flattening layers, SoftMax layers, fully connected layers, etc., to produce output values ​​320m at the output nodes of layer 320l. For example, if N classes are to be identified, there can be N output nodes reflecting the probability that each class is the correct class (e.g., here the network is identifying one of three classes and indicating the class "cat" as the most likely for a given input), although some architectures may have fewer or more outputs. Similarly, some architectures may accept additional inputs (e.g., some floodfill architectures utilize evolved mask structures that can be received as inputs other than the input feature data and produced as modified outputs other than the classification output values; similarly, some recurrent neural networks can store values ​​from one traversal so that they can be fed into subsequent traversals along with other inputs), and may include feedback loops, etc.

[0116] TensorFlow TM Caffe TM and Torch TM These are examples of common software library frameworks used to implement deep neural networks, although many architectures can be created "from scratch," simply representing layers as operations on value matrices or tensors, and representing data as values ​​in these matrices or tensors. Examples of deep learning network architectures include VGG-19, ResNet, Inception, DenseNet, and others.

[0117] Although the example machine learning architecture is already about Figures 3A to 3F Discussions were held, but many machine learning models and corresponding architectures are formed by combining, modifying operations and structures, or adding them to other architectures and technologies. For example, Figure 3G This is a schematic depiction of an ensemble machine learning architecture. Ensemble models encompass a wide variety of architectures, including, for example, "meta-algorithm" models that use multiple weak learning models to work together to form a stronger model, such as AdaBoost. Figure 3A Random forests can be seen as another example of this type of ensemble model, although random forests themselves can be intermediate classifiers in ensemble models.

[0118] exist Figure 3G In the example, the initial input feature vector 325a can be input, in whole or in part, to various model implementations 325b, which can be from the same or different models (e.g., SVM, neural networks, random forests, etc.). The outputs from these models 325c can then be received by a "fusion" model architecture 325d to generate the final output 325e. The fusion model implementation 325d itself can be the same or a different model type than one of implementations 325b. For example, in some systems, the fusion model implementation 325d can be a logistic regression classifier and the model 325b can be a neural network.

[0119] As people will understand, integrated model architecture can facilitate... Figures 3A to 3F The paradigm architecture offers greater flexibility, and one should realize that modifications to the architecture or its methodology (sometimes relatively minor) can foster novel behaviors that are not easily applied to other applications. Figure 2ARegular grouping. For example, PCA is often described as an unsupervised learning method and its corresponding architecture because it discriminates feature representations with reduced dimensionality from input data lacking labels. However, PCA is frequently used with labeled inputs to facilitate supervised classification, as described by M. Turk and A. Pentland in their application of EigenFaces, “Eigenfaces for Recognition”, J. Cognitive Neuroscience, Vol. 3, No. 1, 1991. Figure 3H An example of a machine learning pipeline topology for such modifications is depicted. Similar to EigenFaces, one can use unsupervised methods to determine feature representations in box 330a (e.g., using PCA to determine the principal components of each group of facial images associated with one of several people). As an unsupervised method, Figure 2A The standard grouping of data does not typically allow this PCA operation to be interpreted as "training". However, by transforming the input data (e.g., facial images) into a new representation (principal component feature space) in box 330b, one can create a data structure suitable for applications of subsequent inference methods.

[0120] For example, in box 330c, the new incoming feature vector (new face image) can be transformed into an unsupervised form (e.g., principal component feature space), and then a metric (e.g., the distance between the principal components of each person's face image group and the principal component representation of the new vector) or other subsequent classifier (e.g., SVM, etc.) can be applied in box 330d to classify the new input. Thus, model architectures (e.g., PCA) that are unsuitable for certain methodologies (e.g., metric-based training and inference) can be made so suitable through methodological or architectural modifications (such as pipelines). Similarly, one will understand that this pipeline is merely an example—by assigning the new inference input to the class of the group that has the first moment in the feature space closest to the inference input, Figure 2B The KNN unsupervised architecture and methods can be similarly used for supervised classification. Therefore, these pipeline approaches can be considered machine learning models in this paper, although they are not usually referred to as such.

[0121] Some architectures can be used with training methods, and then some of these trained architectures can be used with inference methods. However, it will be understood that not all inference methods perform classification, and not all trained models can be used for inference. Similarly, it will be understood that not all inference methods require the training method to be pre-applied to the architecture to process new inputs for a given task (e.g., when KNN generates classes by directly considering the input data). Regarding training methods, Figure 4AThis is a schematic flowchart depicting common operations in various training methods. Specifically, in box 405a, either directly by the practitioner or the architecture, training data can be assembled into one or more training input feature vectors. For example, a user could collect images of dogs and cats with metadata tags for a supervised learning method, or collect unlabeled stock prices over time for unsupervised clustering. As discussed, the raw data can be preprocessed to transform into feature vectors, or it can be used directly as features in their raw form.

[0122] In box 405b, training methods can adjust the parameters of the architecture based on training data. For example, the weights and biases of a neural network can be updated via backpropagation, and SVMs can select support vectors based on hyperplane computation, etc. One will understand, as regarding... Figure 3G The pipeline architecture discussed here is not the case; however, not all model architectures can update parameters within the architecture itself during "training." For example, in Eigenfaces, the determination of principal components of facial identity groups can be interpreted as the creation of new parameters (principal component feature space) rather than the adjustment of existing parameters (e.g., adjusting neural network architecture weights and biases). Therefore, in this paper, the determination of Eigenfaces principal components from training images will still be interpreted as a training method.

[0123] Figure 4B This is a schematic flowchart depicting the various operations common to inference methods in various machine learning models. As mentioned earlier, not all architectures or methods can include inference functionality. Where the inference method is applicable, in box 410a, the practitioner or architecture can assemble the raw inference data (e.g., a new image to be classified) into inference input feature vectors, tensors, etc. (e.g., using the same feature input form as training data). In box 410b, the system can apply the trained architecture to the input inference feature vectors to determine the output, such as classification, regression results, etc.

[0124] During "training," some methods and architectures can consider the input training feature data holistically, in a single pass, or ergonomically. For example, in some implementations, decomposition via PCA can be implemented as a non-ergonomic matrix operation. SVM, depending on its implementation, can be trained by a single pass through the input. Finally, some neural network implementations can be trained by performing multiple passes through the input vector during gradient descent.

[0125] Regarding the traversal training method Figure 4CThis is a schematic flowchart depicting the traversal training operations, as may occur in some architectures and methods, such as in box 405b. A single traversal can apply the methods in the flowchart once, while implementations performing multiple traversals can apply the methods multiple times. In box 415a, the parameters of the architecture can be initialized to default values. For example, in some neural networks, weights and biases can be initialized to random values. In some SVM architectures, for example, conversely, the operations in box 415a may not be applied. Since each training input feature vector is considered in box 415b, the system can update the model parameters in 415c. For example, an SVM training method may or may not select a new hyperplane because the new input feature vectors are considered and it is determined whether they affect the support vector selection. Similarly, a neural network method can update its weights and biases, for example, based on backpropagation and gradient descent. When all input feature vectors are considered, the model can be considered "trained" if the training method is called to perform only one traversal. Methods called for multiple traversals can be applied again. Figure 4C The operation (naturally, again avoiding initialization in box 415a to agree with the parameter values ​​determined in the previous traversal) and training is completed when conditions are met, for example, the error rate between predicted tags and metadata tags is reduced below a threshold.

[0126] As mentioned earlier, a wide variety of machine learning architectures and methods exist, including those with explicit training and inference steps, such as... Figure 4E As shown, and those architectures and methods that do not have explicit training and inference steps, such as Figure 4D As summarized in [the text]. For example... Figure 4E The method described is used to train a 425a neural network architecture to recognize newly received images at inference 425b, while Figure 4D Implementations such as reducing data dimensionality via PCA or performing KNN clustering are described, wherein implementation 420b receives input 420a and produces output 420c. For clarity, it will be understood that while some implementations may receive data input and produce output (e.g., an SVM architecture with an inference method), some implementations may only receive data input (e.g., an SVM architecture with a training method), and some implementations may only produce output without receiving data input (e.g., a trained GAN architecture with a random generator method for generating new data instances).

[0127] Figure 4D and Figure 4E The operations can be further extended in some methods. For example, some methods extend the training, such as... Figure 4F The schematic diagram depicts this, where the training method further includes various data subset operations. For example... Figure 4GAs shown, some training methods can divide the training data into a training data subset 435a, a validation data subset 435b, and a test data subset 435c. For example... Figure 4F As shown, when training the network in box 430a, the training method can first iteratively adjust the network parameters using, for example, backpropagation, based on all or a portion of the training data subset 435a. However, in box 430b, a subset of the data reserved for validation in 435b can be used to evaluate the effectiveness of the training. Not all training methods and architectures can guarantee finding the optimal architecture parameters or configuration for a given task; for example, they may get stuck in local minima, or they may employ inefficient learning step size hyperparameters, etc. The method can validate the current hyperparameter configuration in box 430b using training data 435b, which differs from the training data subset 435a that anticipates such defects, and adjust the architecture hyperparameters or parameters accordingly. In some approaches, the method can iterate between training and validation, as shown by arrow 430f, using validation feedback to continue training on the remainder of the training data subset 435a, restarting training on all or part of the training data subset 435a, adjusting the hyperparameters or topology of the architecture (e.g., when additional hidden layers can be added to a neural network in meta-learning), etc. Once the architecture has been trained, the method can evaluate its effectiveness by applying it to all or part of the test data subset 435c. Using different data subsets for validation and testing can also help avoid overfitting, where the training method tailors the architecture's parameters too closely to the training data, reducing better generalizations once the architecture encounters new inference inputs. If the test results are unsatisfactory, the method can restart training with different parameter configurations (architectures with different hyperparameter configurations, etc., as indicated by arrow 430e). The testing in box 430c can be used to confirm the effectiveness of the trained architecture. Once the model is trained, inference 430d can be performed on newly received inference inputs. People will understand that this verification method can vary, for example, when the method performs a grid search over the possible hyperparameter space to determine the architecture best suited for the task.

[0128] Many architectures and methods can be modified to integrate with other architectures and methods. For example, some architectures successfully trained for one task can be trained more effectively for similar tasks, rather than starting with, for example, randomly initialized parameters. Methods and architectures that take parameters from a first architecture and apply them to a second architecture (in some cases, the architecture can be the same) are called “transfer learning” methods and architectures. Given a pre-trained architecture 440a (e.g., a deep learning architecture trained to identify birds in images), a transfer learning method can perform additional training using data from a new task domain (e.g., providing labeled data of images of cars to identify cars in images), making inference 440e possible in this new task domain. The transfer learning training method may or may not distinguish between the training 440b, validation 440c, and testing 440d sub-methods and data subsets as described above, as well as the traversal operations 440f and 440g. It will be understood that the pre-trained model 440a can be received as the entire trained architecture, or, for example, as a list of trained parameter values ​​to be applied to parallel instances of the same or similar architectures. In some transfer learning applications, some parameters of a pre-trained architecture can be "frozen" to prevent them from being adjusted during training, while other parameters are allowed to change as the architecture is trained using data from a new domain. This approach preserves the general advantages of the architecture's initial training while customizing the architecture for the new domain.

[0129] The combination of architecture and method can also be extended over time. For example, the "online learning" method anticipates applying an initial training method 445a to the architecture, subsequently applying that trained architecture to the inference method 445b, and periodically updating 445c by applying another training method 445d, which can be the same as method 445a but typically used for new training data input. The online learning method can be useful, for example, in the case where the robot is deployed to a remote environment after the initial training method 445a, where it can encounter additional data that can improve the application of the inference method at 445b. For example, in the case of deploying multiple robots in this way, when one robot encounters a "true positive" identification (e.g., a new core sample with a classification verified by a geologist; a new patient characteristic verified by an operating surgeon during surgery), the robot can transmit that data and result as new training data input to its peer robot for use with method 445d. The neural network can perform backpropagation adjustments at training method 445d using the true positive data. Similarly, SVM can consider at training method 445d whether new data affects its support vector selection and contributes to the adjustment of its hyperplane. While online learning is often part of reinforcement learning, it can also appear in other methods such as classification, regression, and clustering. The initial training method may or may not include the training 445e, validation 445f, and testing 445g sub-methods, as well as the traversal adjustments 445k and 445l at training method 445a. Similarly, online training may or may not include the training 445h, validation 445i, and testing sub-methods 445j, as well as the traversal adjustments 445m and 445n, and if included, they can differ from the sub-methods 445e, 445f, 445g, and traversal adjustments 445k and 445l. In fact, the subsets and ratios of training data assigned for validation and testing can differ at each training method 445a and 445d.

[0130] As discussed above, many machine learning architectures and methods do not need to be used exclusively for any one task, such as training, clustering, inference, etc. Figure 4JAn example GAN architecture and method are described. In this GAN architecture, the generator sub-architecture 450b can competitively interact with the discriminator sub-architecture 450e. For example, the generator sub-architecture 450b can be trained to generate synthetic “fake” challenges 450c, such as synthetic portraits of non-existent individuals, while the discriminator sub-architecture 450e is trained to distinguish the “fake” challenges from real, true positive data 450d, such as genuine portraits of real people. Such methods can be used to generate, for example, synthetic assets that look like real-world data, to be used as, for example, additional training data. Initially, the generator sub-architecture 450b can be initialized with random data 450a and parameter values, resulting in a highly unbelievable challenge 450c. The discriminator sub-architecture 450e can initially be trained with true positive data 450d, and therefore can initially easily distinguish the fake challenge 450c. However, for each training epoch, the generator's loss 450g can be used to improve the training of the generator sub-architecture 450b, and the discriminator's loss 450f can be used to improve the training of the discriminator sub-architecture 450e. This competitive training can ultimately produce synthetic challenges 450c that are difficult to distinguish from true positive data 450d. For clarity, it will be understood that "adversarial network" in the GAN context refers to the competition between the generator and discriminator described above, and "adversarial input" refers to an input specifically designed to influence a particular output (possibly an output unintentionally designed by the implementation designer) in the implementation.

[0131] Data Overview

[0132] Figure 5A This is a schematic illustration of surgical data that may be received at a processing system in some embodiments. Specifically, the processing system may receive raw data 510, such as video from visualization tools 110b or 140d, which includes consecutive individual frames over time 505. In some embodiments, raw data 510 may include video and system data from multiple surgical procedures 510a, 510b, 510c or only a single surgical procedure.

[0133] As mentioned, each surgical procedure may include a group of actions, each group forming a discrete unit referred to herein as a task. For example, surgical procedure 510b may include tasks 515a, 515b, 515c, and 515e (ellipse 515d indicates that there may be more intermediate tasks). Note that some tasks may be repeated during the procedure, or their order may be changed. For example, task 515a may involve locating a fascial segment, task 515b may involve dissecting the first part of the fascia, task 515c may involve dissecting the second part of the fascia, and task 515e may involve cleaning and cauterizing the fascial area before closure.

[0134] Each task 515 can be associated with a corresponding set of frames 520a, 520b, 520c, and 520d, and a device dataset including operator kinematic data 525a, 525b, 525c, and 525d, patient-side device data 530a, 530b, 530c, and 530d, and system event data 535a, 535b, 535c, and 535d. For example, for video acquired from a visualization tool 140d in the surgical operating room 100b, operator-side kinematic data 525 may include translation and rotation values ​​from one or more handheld input mechanisms 160b at the surgeon's console 155. Similarly, patient-side kinematic data 530 may include data from a patient-side cart 130, data from sensors located on one or more tools 140a-140d, 110a, rotation and translation data from arms 135a, 135b, 135c, and 135d, etc. System event data 535 may include data on parameters that take discrete values, such as activation of one or more pedals 160c, tool activation, system alarm activation, energy application, button press, camera movement, etc. In some cases, task data may include one or more of frame set 520, operator-side kinematics 525, patient-side kinematics 530, and system event 535, instead of all four.

[0135] It will be understood that, although kinematic data are presented as waveforms and system data as continuous state vectors in this document for clarity and ease of understanding, some kinematic data may be assumed to be discrete over time (e.g., an encoder measuring continuous part points may be sampled at fixed intervals), and conversely, some system values ​​may be assumed to be continuous over time (e.g., values ​​may be interpolated, as when a parametric function can be adapted to individual sampled values ​​from a temperature sensor).

[0136] Furthermore, although surgical procedures 510a, 510b, 510c and tasks 515a, 515b, 515c are shown here as adjacent for ease of understanding, it will be understood that gaps may exist between surgical procedures and tasks in real-world surgical video. Therefore, some video and data may be task-independent. In some embodiments, these non-task regions may themselves be represented as tasks, e.g., "gap" tasks where no "real" task occurs.

[0137] The discrete set of frames associated with a task can be determined by the task's start and end points. Each start and end point can be determined by tool actions or changes in the body's state influenced by the tool. Therefore, data acquired between these two events can be associated with the task. For example, the start and end actions of task 515b can occur at timestamps associated with positions 550a and 550b, respectively.

[0138] Figure 5B This is a table depicting example tasks that can be used in conjunction with various disclosed embodiments, along with their corresponding start and end points. Specifically, the data associated with the task "Colon Mobilization" is data acquired between the time when the tool first interacts with the colon or surrounding tissue and the time when the tool last interacts with the colon or surrounding tissue. Thus, any one of the frame set 520, operator-side kinematics 525, patient-side kinematics 530, and systemic events 535 with timestamps between this start and end point is data associated with the task "Colon Mobilization." Similarly, the data associated with the task "Pelvic Fascia Anatomy" is data acquired between the time when the tool first interacts with the pelvic fascia (EPF) and the time when it last interacts with the EPF after prostate degreasing and dissection. The data associated with the task "Tip Anatomy" corresponds to data acquired between the time when the tool first interacts with tissue at the prostate and the time when the prostate has been freed from all appendages of the patient's body. It will be understood that the task start and end times can be selected to allow time overlap between tasks, or can be selected to avoid such time overlap. For example, in some embodiments, a task can be “paused,” such as when a surgeon involved in a first task switches to a second task before completing the first, completes the second task, and then returns to complete the first task. Therefore, while start and end points can define task boundaries, it will be understood that data can be annotated to reflect timestamps related to more than one task.

[0139] Additional task examples include “2-handed suturing,” which involves performing four horizontal interrupted sutures using a two-handed technique (i.e., in the case of only two hands, e.g., no single-handed suturing action occurs in between, the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle withdraws from the tissue). The “Uterine Horn” task involves dissecting the broad ligaments from the left and right uterine horns, as well as cutting the uterine body (it will be understood that some tasks have more than one condition or event determining their start or end time, as here, the task begins when the dissecting instruments contact the uterine horn or the uterine body, and ends when both the uterine horn and the uterine body are disconnected from the patient). The “1-handed suturing” task involves performing four vertical interrupted sutures using a single-handed technique (i.e., in the case of only one hand, e.g., no two-handed suturing action occurs in between, the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle withdraws from the tissue). The task “Suspensory Ligaments” involves dissecting the lateral lobes of each suspensory ligament to expose the ureter (i.e., the start time is when the dissection of the first lobe begins, and the stop time is when the dissection of the last lobe is completed). The task “Continuous Suturing” involves performing continuous suturing with four occlusions (i.e., the start time is when the suture needle first penetrates the tissue, and the stop time is when the needle is withdrawn from the tissue after all four occlusions have been completed). As a final example, the task “Rectal Artery / Venus” involves dissecting and ligating the superior rectal artery and vein (i.e., the start time is when the dissection of the artery or vein begins, and the stop time is when the surgeon stops contacting the ligation after ligation).

[0140] Task classification model topology

[0141] Given one or more of the video, kinematic, and systemic data of a surgical procedure, one may wish to identify the depicted task, such as tasks 515a, 515b, 515c, and 515e. Figure 6 An example ensemble machine learning model topology 600 is depicted, which can be used to determine task classification when given a dataset (such as raw data 510b associated with a specific surgical procedure). For example, the system may receive visualization tool data 605a (such as one or more video frames 610a), operator-side kinematic data 605b (such as a subset 610b of kinematic waveforms within the time frame of acquiring video frame 610a), patient-side kinematic data 605c (such as a subset 610c of kinematic waveforms within the time frame of acquiring video frame 610a), and system event data 605d (such as a subset 610d of state vectors that may also be within the time frame of acquiring video frame 610a).

[0142] Data of one or more of these types can be received at the processing system through corresponding machine learning models (specifically, visualization machine learning model 615a, operator-side kinematics machine learning model 615b, patient-side kinematics machine learning model 615c, and system event machine learning model 615d). Each model can produce corresponding task classification outputs 620a, 620b, 620c, and 620d. For example, in the case where the model selects from 50 task classification possibilities (it will be understood that in some embodiments, some values ​​may correspond to "no task," "unknown," or "detection failed"), each output 620a, 620b, 620c, and 620d may include a vector of 50 probability values ​​for each possible task classification (although this is not necessarily the case; for example, the model outputs fewer than 50 values ​​to facilitate compressed representations, as in an autoencoder).

[0143] The processing system can then merge the outputs 620a, 620b, 620c, and 620d to form a merged vector 625. For example, in the case where each output is a vector of 50 probability values, these vectors can be concatenated to form a vector of 200 values ​​(or concatenated along another dimension to form a 4x50 matrix of values). The processing system can then feed this merged vector 625 into a merged classification fusion machine learning model or logic 630 (e.g., a logistic regression classifier, random forest, software that performs majority voting on the predictions of previous models, etc.) to produce a final merged classification output 635 (e.g., which can again be a vector of 50 probability values).

[0144] It will be understood that data 605a, 605b, 605c, and 605d can be downsampled from their initial capture rate. Such downsampling can facilitate the need for realignment, which in some embodiments can be performed on a per-second basis using the timestamp of each stream after downsampling (such realignment can introduce an acceptable error on the order of tens of milliseconds). In some cases, data from different streams may not be in the same time range or at the same sampling frequency. For example, a video-based model may use, for instance, 32 seconds of past data, a kinematic model may use 128 seconds of previous kinematic data, and a system model may use 196 seconds of previous event data. It will be understood that all these inputs can be used to predict the final second under consideration, although their respective ranges differ. Thus, in some embodiments, video may be initially sampled at 60 frames per second, kinematic data at 50 samples per second, and events may be recorded as they occur (i.e., unsampled). In some embodiments, video data may be downsampled to 1 frame per second (and the size is adjusted to a dimension of 224x224x3, i.e., 224 pixels wide and 224 pixels high for red, green and blue pixels) and kinematic data may be downsampled to 4 samples per second.

[0145] As will be discussed in this paper, data may not always be available for all four streams, and models 615a, 615b, 615c and 615d and / or model or logic 630 may be trained to accept “dummy” values ​​instead, so that the processing can remain resilient to such omissions.

[0146] Task classification model topology—Example video data model

[0147] Figure 7A In some implementations, such as... Figure 6Schematic integration machine learning model topology diagram of an example machine learning model used in the visualization machine learning model 615a of the machine learning model topology. Specifically, the model can receive multiple frames 705a, 705b, 705c at multiple multi-layer convolutional neural networks 705e, 705f, 705g (ellipse 705d indicates the possibility of more intermediate networks). For example, the dimension of frame 705a can be 256x×256x×3 (red, green, and blue pixel values of an image with a width and height of 256 pixels each). Network 705e can process frame 705a to produce a linear vector output of 1×K values (in some embodiments, K is the number of tasks to be predicted), which can be appended to each other. Since there are N frames and N corresponding outputs from the multi-layer convolutional neural networks 705e, 705f, 705g, the resulting structure can have a dimension of N×K. Then, this N×K structure can be considered by one or more layers configured to process time input sequences. For example, in some embodiments, the N×K structure can then be directly submitted to one or more layers to consider the results in sequence, referred to herein as one or more sequence layers 705j. For example, the results can be submitted to RNN 705i to produce 1×T task predictions 705k (e.g., probabilities assigned to each possible task classification via, for example, a final dense and / or SoftMax layer). Some embodiments can alternatively send the N×K structure to a one-dimensional convolutional neural network (Conv1D) 705h (which can then again be a final dense and / or SoftMax layer to produce the final prediction probabilities). As shown herein, some embodiments can employ both RNN 705i and Conv1D layer 705h. For example, the one-dimensional convolutional layer 705h can receive a set of N×K values to produce a set of M×K values, where M < N, and then provide these results to RNN 705i (effectively allowing RNN 705i to operate on a downsampled version of the results). In some embodiments, the one-dimensional convolutional layer 705h can include a kernel of size 3 - 8, with 96 - 480 filters, and 1 - 3 consecutive convolutional layers.

[0148] To facilitate clarity when discussing one-dimensional convolution on a two-dimensional structure, Figure 7B A schematic representation of the expected operation is provided. Specifically, the convolutional kernel or window 715a, shown herein as containing three consecutive frames, can slide from left to right starting from the first of the N multi-layer CNN outputs. Thus, in the case where the window is 3, M is N - 3. Each newly created vector can be determined by combining the vectors within the kernel window according to the weights of the learned kernel. For example, the first value 715b in the new vector can be the weighted sum of the first of the K values in the vector at the site illustrated in window 715a, the second value 715c in the new vector can be the weighted sum of the second of the K values in the vector at the site illustrated in window 715a, and so on.

[0149] Return to Figure 7A , it will be understood that in some embodiments, such as when the multi-layer CNNs 705e, 705f, 705g are themselves trained on the same training data to identify tasks, K can be the same as the number of T task classes to be predicted (i.e., K = T). However, in some embodiments, such as when the multi-layer convolutional neural networks 705e, 705f, 705g are trained end-to-end with the entire model, K can be not equal to T. Enabling K>T can provide greater flexibility in intermediate features, while K<T can improve feature selection, similar to the operations of autoencoders, PCA, etc.

[0150] It will be understood that in an alternative embodiment, the processing system can continuously input each of the frames 705a, 705b, 705c through a single multi-layer CNN, rather than feeding the frames simultaneously through a set of parallel CNNs. In fact, although parallel processing of frames through multiple instances can provide time efficiency for real-time recognition applications during inference, training a single multi-layer CNN can be more time- and resource-efficient. Similarly, an intermediate approach can be adopted, which applies a subset of the frames to one of fewer than the total number of CNN instances (i.e., the number of multi-layer CNNs is less than N).

[0151] It will be understood that multiple multi-layer CNN architectures can be suitable for use as the multi-layer CNNs 705e, 705f, 705g, as long as they provide sufficient ability to identify the tasks to be classified. For example, Figure 7C is a schematic machine learning model topology diagram of a multi-layer CNN variant of the VGG19 architecture (similarly, it will be understood that corresponding variants of similar architectures (with necessary modifications), such as ResNet 50, InceptionV3, etc.). It will be understood that pre-trained implementations of these models are readily available (e.g., the Keras TM library provides a version of VGG19 pre-trained on the ImageNet library).

[0152] In some embodiments, through transfer learning from such a pre-trained version of the model, one can create Figure 7A one or more multi-layer CNNs as shown. Specifically, one can retain the pre-trained layers that receive the input image 710a until the final layer before the fully connected output of the model (here, the max pooling layer 710c). Thus, each of the layers 710b can contain pre-existing hyperparameters from the pre-training (one will recognize that these layers correspond to those described above with respect to Figure 3FThe hyperparameters of these layers (the "feature extraction" layers discussed) can be kept fixed or "frozen" and are not allowed to change during future training specifically for the task recognition context. In contrast, the layers after the max-pooling layer 710c (called the "head layers")... Figure 3F The “classification” layers can be retained and their weights can be changed, or they can be replaced with layers that allow for weight changes. For example, some embodiments replace these layers with layer structure 710d, which has a single fully connected layer followed by a SoftMax layer. Other embodiments may include multiple fully connected layers, as shown in layer structure 710e, which can facilitate greater recognition capabilities. Therefore, layer structure 710d may be more suitable than layer structure 710e when distinguishing a small number of very different tasks. Conversely, layer structure 710e may be more suitable than layer structure 710d when distinguishing many tasks with subtle differences. It will be understood that in some embodiments, not each of the multi-layer CNNs 705e, 705f, 705g can have the same head structure selection (variable selection of the head structure can facilitate more robust recognition in some cases).

[0153] Therefore, in some embodiments, one can utilize, such as Figure 7D The process shown is used to train, for example Figure 7C The example shown illustrates a transfer model. Specifically, in box 720a, the training system can acquire a pre-trained model (e.g., the VGG19 model pre-trained on the ImageNet dataset discussed above) and freeze non-head parameters in box 720b, for example, freezing layer 710b. It will be understood that box 720b may not reflect an affirmative step, but rather, for example, simply ignore the training configuration that updates the weights of the frozen layers. In box 720c, pre-existing unfrozen layers can be modified or replaced (e.g., replaced with layer structure 710e or layer structure 710d), although some embodiments omit box 720c to support modifications to existing head layers from the initial model.

[0154] As will be discussed in more detail, one can now train multi-layer CNNs to directly recognize tasks or they can be integrated with the rest of the model (e.g., trained separately or when integrated with one-dimensional convolutional layers 705h and RNN 705i). Here, in box 720d, the multi-layer CNN model is integrated with the rest of the ensemble model, and the ensemble is trained as a whole in box 720e. Similarly, for clarity, one will understand that in some embodiments, one can instead train the multi-layer CNN model on annotated frame training data (i.e., adjust their unfrozen weights) and then use the outputs of one or more trained multi-layer CNNs on the same or different training data.

[0155] Task classification model topology – example video data model – recursive network

[0156] The RNN 705i can take the form suitable for identifying time-varying patterns associated with each task from the refined features of multi-layer CNNs 705e, 705f, and 705g. Typically, such an RNN 705i can be based on... Figure 8A The topology is structured. Here, the neural network 805b can be arranged to receive input 805c and produce output 805a, as per [reference to...]. Figure 3C , Figure 3D and Figure 3F The above is discussed. However, one or more of the outputs from network 805b can be fed back to network 805b as one or more recursively hidden outputs 805d, and saved in a timely manner during the operation of network 805b.

[0157] For example, Figure 8B Showing with Figure 8A The same RNN, but with different inputs at each time step during inference. In the first traversal at time 1, applying network 805b to the first input 810n produces output 810a and a first hidden recursive output 810i. At time 2, the network can receive the first hidden recursive output 810i and a new input 810o, producing a new output 810b. It will be understood that during the first traversal at time 1, an initial default hidden recursive value 810r can be fed into the network.

[0158] In this way, output 810i and the subsequently generated output 810j can depend on the previous inputs, for example:

[0159] h t = f (h) t-1 , x t (4)

[0160] As shown in ellipse 810s, these traversals can continue for multiple time steps until all input data has been considered. For example, a one-dimensional convolutional layer 705h can produce an MxK output, and therefore, in the M traversals, a data vector of size K can be considered in each traversal.

[0161] Since the penultimate input 810p and the final input 810q are submitted to network 805b (along with the previously generated hidden output 810k), the system can produce a corresponding penultimate output 810c, a final output 810d, a penultimate hidden output 810l, and a final (potentially unused) hidden output 810m. Because the previous outputs 810d were generated without considering all the data inputs, in some embodiments they can be discarded, and only the final output 810d is used as the RNN's prediction. However, in other embodiments, such as when the fusion model is trained to identify predictions from the ergodic nature of the outputs, each output can be considered. One will understand the various ways to use such "many-to-one" RNN topologies (receiving many inputs but producing a single predicted output).

[0162] In some embodiments, network 805b may include one or more Long Short-Term Memory (LSTM) units, such as Figure 8C As indicated. In addition to the hidden output H (corresponding to a portion of hidden output 805d), the LSTM unit can also output the unit state C (also corresponding to a portion of hidden output 805d), which is modified by multiplication operation 815a and addition operation 815b. The Sigmoid neural layers 815f, 815g, and 815i, as well as the tanh layers 815e and 815h, can also perform operations on the input 815j and intermediate results, similarly using the multiplication operations 815c and 815d shown in the figure.

[0163] While a single RNN layer (e.g., an LSTM layer) or Conv1D layer may be sufficient in some embodiments, some embodiments consider combining both approaches. For example, Figure 8D The diagram illustrates model topology 820, which combines RNN 705i and one-dimensional convolution 705h.

[0164] Here, the initial one-dimensional convolutional layer 820g can receive the NxK merged output of multiple CNNs 705e, 705f, and 705g (here, the concatenated input 820h). In some embodiments, the convolutional layer 820g can be followed by a max-pooling layer 820f, which computes the maximum value of the intervals of the feature maps, facilitating the selection of the most salient features. Similarly, in some embodiments, the max-pooling layer 820f can be followed by a flattening layer 820e. The result can then be fed as input to an LSTM layer 820d. In some embodiments, the topology can end at the LSTM layer 820d. However, if the LSTM layer 820d is not yet in a many-to-one configuration, subsequent layers (such as subsequent dense layers 820c and consolidation layers 820b) can be used to perform averaging, SoftMax, etc., to produce output 820a. Again, as previously stated, it will be understood that... Figure 8DOne or more of the dashed layers can be removed in various embodiments that implement combined LSTM and Conv1D.

[0165] When using LSTM (whether alone or, for example, ...) Figure 8D As shown), some embodiments employ a single-layer LSTM model, where the number of layers ranges from 64 to 1024. Similarly, dropout layers can be used between the LSTM and the final dense layer, where the dropout ratio ranges from 0 to 0.5 (again, whether alone or, for example, as shown). Figure 8D (as shown in the image).

[0166] Task classification model topology – Example video data model – Training

[0167] The video-based model described above can be trained in several ways. For example, Figure 9A The diagram illustrates various operations during the training process (e.g., performed by the training system, human trainers, meta-learning systems, etc.), which separately consider one or more multi-layer CNNs 705e, 705f, 705g with one-dimensional convolutional layers 705h and RNNs 705i. Specifically, in box 905a, the training system can receive training data, such as video data whose frames have been annotated with the corresponding task. In box 905b, this data can be processed into a form suitable for performing training. For example, underrepresented tasks can be addressed via techniques such as Synthetic Minority Oversampling (SMOTE) (e.g., using imblearn). TM The algorithm of the library function `imblearn.over_sampling.SMOTE` is synthetically upsampled, although this may not be necessary if the initial training data distribution is adequate (rather than applying upsampling to all data, which can be performed only in training folds that discover underrepresented classes). Box 905b may also include operations such as selecting the number of fully connected layers based on the number of task classifications to be distinguished (and the desired configuration of the multi-layer CNN), e.g., group 710d or 710e. This is useful in cases where the model is to be trained via transfer learning (e.g., as discussed above). Figure 7B As described, the preprocessing at box 905b may involve setting the frozen and unfrozen weights of the model.

[0168] and Figure 7DIn contrast to training the model throughout the ensemble in box 720e, in example process 905, at box 905c, the system can train one or more multi-layer CNNs individually (e.g., for a recognition task). Naturally, the pre-trained model can still be used via transfer learning, as described above at box 905c. Once trained, at box 905d, one or more multi-layer CNNs can transform the training data into their corresponding prediction outputs. These predictions can then be used to train one or more sequence layers 705j, such as Conv1d and / or RNN structures (e.g., ...). Figure 8D (The structure). It will be understood that methods such as Temporal Backpropagation (BPTT) allow for the training of temporal RNN structures via normal backpropagation through the training layers using one-dimensional and other backpropagation methods, and stochastic gradient descent. Therefore, in some embodiments, the RNN can be an LSTM layer loaded with random weights, the learning rate of the stochastic gradient descent optimizer can be variable, but is typically 0.0005, and the LSTM can be evaluated after each epoch on the validation portion of the training data. Training can terminate when the validation accuracy no longer improves above a threshold for consecutive iterations.

[0169] contrast Figure 9A In some embodiments, alternative methods can be derived from, for example... Figure 9B The “end-to-end” process shown 910 trains the ensemble model (e.g., with...). Figure 7D (The example is consistent). That is, after box 910a receives the training data and the data is preprocessed before box 910b, Figure 8B The entire ensemble model can be trained as a group at box 910c. While the model length in this manner may be subject to gradient vanishing, this training method can be applied to situations where the output of one or more multi-layer CNNs is greater than or less than the number of task classifications. This can be useful, for example, in cases where the ensemble model behaves similarly to an autoencoder (multi-layer CNNs output less than the number of classes), to identify the most salient set of features from each image for consideration by one-dimensional convolutional layers and RNN models.

[0170] Task classification model topology—example kinematic data model

[0171] Figure 10A In some embodiments, such as... Figure 6This is an illustrative ensemble machine learning model topology diagram of the example machine learning models used in the operator-side kinematic machine learning model 615b or the patient-side kinematic machine learning model 615c. Specifically, the processing system can receive raw kinematic data 1005 and downsample data 1010 to produce compressed kinematic data 1015, which can then be concatenated to produce concatenated kinematic data 1020. Similarly, the raw kinematic data 1005 can be, for example, a time series of multiple system sensor components sampled at 50 samples per second. Therefore, these values ​​can include, for example, robot joint angle points, robot joint relative translations, tool points in three-dimensional space relative to a camera-centric reference frame, etc. Downsampling data 1010 can also include dimensionality reduction by applying PCA, for example, to normalize the data. In some embodiments, the processing system can whiten the data such that the standard deviation of all data is forced to one. In some embodiments, the PCA algorithm can be an incremental PCA algorithm (e.g., using scikit-learn). TM The library `sklearn.decomposition.Incremental PCA` function is used to convert data into a low-dimensional representation, such as 64 or 96 dimensions. However, in some embodiments, downsampling of kinematic data can be achieved by alternatively undersampling existing data points over time (e.g., sampling every other available point). In other embodiments, these two approaches can be combined, such as applying PCA and undersampling the data.

[0172] The processing system can then provide linked kinematic data 1020 (e.g., a set of linked values ​​from each kinematic data source for each time point) to one or more sequence layers 1050, which can be configured in one of the ways described above with respect to sequence layer 705j (including those described above). Figure 8D The description is structured. For example, one or both of Conv1d layer 1025 and RNN layer 1030 (such as LSTM layer) can be used to generate task recognition output 1035. In some embodiments, only one-dimensional CNN model 1025 can be used for kinematics and RNN model 1030 can be removed, as this has been found to sometimes provide sufficient results, especially when the kinematic data is to be considered in conjunction with data from a system or visualization tool.

[0173] Conversely, in those embodiments that retain the RNN model 1030 and include LSTM layers, some embodiments may employ a bidirectional LSTM with 32 to 1024 units. This architecture is suitable for situations where task recognition is performed without system or visualization tool data, or where such data is expected to be frequently missing.

[0174] Task classification model topology — example kinematic data model — training

[0175] Figure 10B The illustration shows an example of training that can be applied in some embodiments. Figure 10A The flowchart illustrates the various operations in the model's process 1040. Specifically, in block 1040a, the system can receive annotated training data (e.g., one or more tool points over time, operator input points over time, etc.). Following the operations 1010, 1015, and 1020 described above, this data can be converted into feature vector form in block 1040b. In block 1040c, one or more sequence layers 1050 can be trained using these annotated feature vectors to produce the trained model in block 1040d.

[0176] Task classification model topology—Example system event data model

[0177] Unlike kinematic data, which can be sampled periodically at frequent intervals over time, many events can occur within a single instance of time or at irregular intervals. Therefore, system event identification can benefit from topologies different from those previously described. For example, Figure 11A In some embodiments, such as... Figure 6 A schematic diagram of the machine learning model topology used in the system event classification model 615d.

[0178] Specifically, in some embodiments, the system event machine learning model 615d may take the form of a stacked ensemble learning model 1100. The processing system may provide system event data 1105 to one or more base models 1110. In some embodiments, the base models may include a logistic regression model 1110a, a random forest model 1110b, and a neural network model 1110c (although more or fewer models may be considered in some embodiments).

[0179] The base model 1110 can generate multiple classification outputs 1115a, 1115b, and 1115c (e.g., a vector of probability values ​​for the task under consideration, as shown in the figure). In some embodiments, the size of each of the outputs 1115a, 1115b, and 1115c is the number of potential task classifications, although this is not necessary in other embodiments, as previously discussed (e.g., those embodiments seeking to perform feature reduction similar to an autoencoder). The processing system can then concatenate the outputs 1115a, 1115b, and 1115c to form a merged vector 1120.

[0180] The processing system can then provide the merged vector 1120 to one or more fusion models 1125. While ensembles of models can be used to fuse models as can be done to base models, in many embodiments, a single fusion model (which is a random forest 1125a or an extreme random tree 1125b (or, in some embodiments, both, as shown here)) can be used in conjunction with an ensemble of base models to produce good results.

[0181] The final task identification 1130 can then be determined using the outputs from one or more fusion models 1125 (which in some embodiments may be used as task classification output 620d). It will be understood that in the presence of more than one fusion model 1125, accompanying logic (e.g., majority voting of the results of each fusion model) can be used to select the final task identification 1130.

[0182] For clarity, it will be understood that fusion models can utilize scikit-learn. TM The library implements this using the function calls shown in listing C1 or C2:

[0183] RandomForestClassifier(n_estimators=200,class_weight='balanced')(C1)

[0184] ExtraTreesClassifier(n_estimators=200,class_weight='balanced') (C2)

[0185] Similarly, the basic model can be implemented using one or more calls shown in listings C3 through C7:

[0186]

[0187] KNeighborsClassifier(n_neighbors=10) (C4)

[0188]

[0189] RandomForestClassifier(n_estimators=200,class_weight='balanced')(C6)

[0190] ExtraTreesClassifier(n_estimators=200,class_weight='balanced') (C7)

[0191] In other words, regarding rows C6 and C7, separate instances of the same model topology used for fusion models can also appear in the base model.

[0192] Task classification model topology — Example system event data model — Training

[0193] Figure 11B The illustration shows a method that can be used for training in some implementations. Figure 11A The flowchart shows the various operations in the model's process 1150. In box 1150a, the training system can receive annotated system event training data and can convert this data into a feature vector in box 1150b. While one can use these features to train the model in an "end-to-end" manner, as in... Figure 9B The visual models discussed in some embodiments are, here, training of the base model and the fusion model, respectively. Specifically, at boxes 1150c and 1150d, the training system can iterate through the base models and train them based on feature vectors at box 1150e. Once the base models are trained, at box 1150f, the training feature vectors can be transformed into their corresponding outputs from the base models. Using the initial training data annotations, this form of data can also be used to train one or more fusion models at boxes 1150g, 1150h, and 1150i. After training one or more fusion models, the system can produce a final stacked learner at box 1150j for future inference.

[0194] Example system and event data generation

[0195] Both kinematic data 1005 and event data 1105 can be reformatted based on the raw sensor output. In some cases, kinematic data can appear in a form easily distinguishable from event data, such as when kinematic data is provided as a time series of encoder sensor values. However, in some cases, it may be necessary to infer event and kinematic behavior from the raw system output.

[0196] For example Figure 12A This is a sample text list of JSON extracts from sensor data on a robotic system. This data can be, for example, based on... Figure 12B The process is converted to NumPy TM Array for use by Figure 10A and Figure 11AThe system processes this data. In this example, the robotic system can output an array of data entries, each with various attributes. The "Recorded Timestamp" attribute, as shown in line 1, can indicate the time the data was acquired relative to the system clock. Some data can be encrypted to meet privacy and regulatory obligations, and the contents of the entries may need to be decrypted at the location indicated by "decoded_msg_dict" in line 3. The header of the decoded data in line 4 can include several parameters that provide metadata about the event. The decoded data can also include tool-specific attributes, as indicated by "Tool Data" in line (10), providing the name (line 12) and device details (e.g., the serial number in line 11). The "event_entry" parameter can then indicate the data that facilitated the creation of the entry. Here, the tool has been removed from the screen, as indicated by the event name in line 13 and the id in line 17. Other parameters (such as message type (line 18)) can help determine the context of the event (e.g., the tool appears off-screen in response to camera movement, rather than being removed from the patient). Some parameters (such as history buffer (line 16)) can indicate the relationship of the entry to other entries.

[0197] People will understand that system data parameters do not have to be binary, as when the position of an instrument over time is represented by an array of floating-point values. This JSON can be parsed to create a binary NumPy array. TM The feature vectors are arrayed for consumption by the corresponding model. For example, Figure 12B This diagram illustrates how raw data (such as...) can be used to... Figure 12A The flowchart of the various operations in the process 1245 of converting the data depicted in the diagram into the form of a feature vector is shown. In box 1245a, the conversion system may consider whether all event data (e.g., JSON entries) has been considered, and if not, the next entry is considered in box 1245b.

[0198] Some entries can be identified as representing part of a larger kinematic operation or system event. For example, JSON may not include an "energy saturation" event, but considering a series of "energy application" events in the JSON allows one to infer when such an event occurred. Therefore, if the event considered in box 1245c is considered such a partial indication of an event, the system can append that entry to the buffer for later consideration in box 1245d. Once the buffer is complete in box 1245e (e.g., enough data has been collected to infer the complete time-propagation event), the system can convert the buffered data into appropriate feature vector terms in box 1245f for consideration by a machine learning model (or for use during training or inference). For example, multiple stored energy application events in the buffer can be reviewed, and if they occur close enough to cause part saturation, the system can generate a saturation feature in box 1245f, which is timestamped when the system determines that saturation has occurred.

[0199] Once all JSON entries have been considered, the system can be placed in box 1245g in the section to be used for the kinematic model (e.g., Figure 10A The model) and system event model (e.g., Figure 11A The feature vectors of a system event (or system event model) can be distinguished from each other. For example, an "arm swap" system event can occur when an operator reassigns handheld input from one robotic arm to another. The raw JSON entries of each arm's movement can be directly converted into kinematic data values. However, an "arm swap" system event may need to be inferred by recognizing that each arm can be associated with the same input, that the static / active kinematic values ​​of one arm complement the static / active kinematic values ​​of the other arm, and when the static / active relationship becomes reversed (i.e., when the swap event occurs). Therefore, this would be the "system event" feature vector used with the system event model. Thus, the JSON entries associated with kinematic or system data and the events derived from them can be mapped to their appropriate feature vector forms and provided to the appropriate corresponding machine learning model.

[0200] Example Fusion Model

[0201] Figure 13A This is an illustrative machine learning model topology diagram of an incorporated fusion classification model or logic that may be implemented in some embodiments. It will be understood that the model or logic (e.g., software that performs majority voting on predictions from previous models) 1305k may be identical to model or logic 630. In some embodiments, model 1305k may output a task classification for each consecutive time point based on data. In some embodiments, model 1305k may only output the start / stop time of the task and the task name in the video recording.

[0202] As discussed, during inference, one or more of the following can be supplied to the video model 1305e: video 1305a, operator-side kinematic data 1305b, surgeon-side kinematic data 1305c, and system event data 1305d (e.g., regarding...). Figure 7A The discussed), operator-side kinematic model 1305f (as discussed) Figure 10A The surgical lateral kinematic data model 1305g (as discussed) (also regarding Figure 10A (as discussed) and system event model 1305h (such as regarding Figure 11A (As discussed). These models can generate predictions 1305i for their respective data corpora. However, in some embodiments, the "prediction" of the video model 1305e may be the output of a connection from multiple layers of CNNs 705e, 705f, 705g instead of the final output 705k; that is, in these embodiments, the sequence layer 705j may be removed or ignored. In some embodiments, the sequence layer 1050 and the fusion model 1125 may also be removed to support the outputs from the kinematics 1020 of the connection and the merged vectors 1120 for prediction 1305i, respectively. However, in most embodiments, as Figure 13A As shown, the final prediction output of each stream can be considered for each prediction 1305i. The merged structure 1305j from these predictions can be fed into the fusion model 1305k to produce prediction 1305l.

[0203] In some embodiments, the fusion model 1305k is a random forest or an extreme random tree, for example, created using lines C1 or C2 of the code listing above. In some embodiments, the fusion model may be a logistic regression model (e.g., according to line C3 of the code listing). Similarly, it will be understood that in embodiments using only a single one of models 1305e, 1305f, 1305g, 1305g, predictions made by the fusion model 1305k may not be applied.

[0204] In some embodiments, there may also be uncertainty logic 1305m, which can determine uncertainty value 1305n based on prediction 1305i. In some embodiments where the fusion model is a generative model (e.g., a Bayesian neural network), uncertainty can be discerned from inherent characteristics of the prediction distribution 13051 (e.g., the variance of the distribution of the most likely prediction).

[0205] Example fusion model - training

[0206] Training the fusion model 1305k can be done in a manner similar to other models described in this paper. Specifically, Figure 13B The illustration shows an example of training that can be applied in some embodiments. Figure 13AThe flowchart illustrates the various operations involved in the modeling process. After receiving annotated training data in box 1310a and converting the data into an appropriate feature vector form in box 1310b, each of models 1305e, 1305f, 1305g, and 1305g can be trained according to the method previously described in box 1310c. Once trained, these models can be used in box 1310d to convert the training feature vectors into predictions 1305i. In box 1310f, these predictions and corresponding annotations from their respective training feature vectors can then be used to train the fusion model 1305k.

[0207] The modular architecture of the four differentiated models described in this paper, followed by a fusion prediction model, can provide more accurate predictions than any single model on a single data stream. By training the differentiated models 1305e, 1305f, 1305g, and 1305g, the problem of synchronizing data over time is also likely to be less, as each data stream can be processed individually to produce task predictions for each second of the surgery (or other desired appropriate intervals). This overcomes the challenge of sub-second data alignment between streams, especially when the streams have different sampling rates.

[0208] Even when faced with missing data, the methods disclosed herein can facilitate robust recognition models. For example, each data type may not always be available, such as when only video data is available in operating room 100a, or when robotic event data from operating room 100b cannot be synchronized with the corresponding video data from the surgery. Therefore, in box 1310e, the training system may include “virtual” feature vectors that will also be submitted during inference when a given data stream is unavailable. For example, for training data from all four streams, combinations of one and three streams can be replaced with virtual feature vectors to simulate the availability of only the remaining streams. In this way, model 1305k can be resilient to data unavailability when deployed for inference.

[0209] Example of an uncertain algorithm

[0210] In those embodiments involving computational uncertainty, various feasible approaches can be used for such computations, depending on whether the model involved is discriminative or generative. For example, Figure 14B and Figure 14C Each of them is depicted for reference. Figure 14A An example procedure for measuring uncertainty in the assumed result set in the table. Figure 14BIn example procedure 1400a, at block 1405a, the computer system can initialize the hold "max" for the maximum count across all categorized classes, whether profession or program. As indicated in block 1405b, the system can iterate through all classes (i.e., all tasks under consideration). Since each task class has been considered in block 1405c, its maximum count "max_cnt" can be determined in block 1405d and compared with the current value of the hold "max" in block 1405e. If max_cnt is larger, the hold "max" can be reassigned to the value of max_cnt in block 1405f.

[0211] For example, refer to Figure 14A The table of assumptions gives four predictions for each stream from models 1305e, 1305f, 1305g, and 1305h, for classes A, B, C, and D, and for each stream from models 1305e, 1305f, 1305g, and 1305h. The fusion model or logic 1305k can have concluded that the prediction should be for class A, since three of the models predict class A as the most likely class or at least as likely as the other class.

[0212] For example, for prediction stream 1, the model produces a frame set with a 30% probability of belonging to task A, a 20% probability of belonging to task B, a 20% probability of belonging to task C, and a 30% probability of the prediction stream belonging to task D. During the first traversal through box 1405c, the system can consider the value of task A for each stream. Here, task A is the class most predicted among prediction streams 1, 2, and 3 (tie-ups are counted as the most predicted result). Since task A is the class most predicted among these three streams, max_cnt is 3 for that class. Since 3 is greater than 0, the system will assign the hold "max" to 3 in box 1405f. A similar procedure for subsequent traversals can determine that max_cnt is 0 for task B, 0 for task C, and 2 for task D. Since each subsequent max_cnt determination is less than 3, the maximum hold will remain at 3 when the process transitions to box 1405g after considering all classes. In this box 1405g, the uncertainty can be output as:

[0213]

[0214] Continuing on Figure 14A The example table has four prediction streams, and therefore the uncertainty is 1-3 / 4, or 0.25.

[0215] Figure 14CAnother example procedure 1400b for calculating uncertainty is depicted. Here, in block 1410a, the system can set the entropy holder variable to 0. In blocks 1410b and 1410c, the system can reconsider each class, determine the class average in block 1410d, and append the logarithm of the average in block 1410e, where the logarithm is taken as base number of classes. For example, refer to Figure 14A From the table, people will understand that the average value for "Task A" is

[0216]

[0217] The corresponding average calculations for other tasks are shown below. Once all classes have been considered, the final uncertainty can be output in box 1410f as the entropy value divided by the negative number of classes. Therefore, for Figure 14A The example table in the table can result in a final uncertain value of approximately 0.227.

[0218] People will understand Figure 14C The process involves calculating the Shannon entropy. Specifically, where y c,n This represents the SoftMax prediction output of the c-th class of the n-th prediction stream.

[0219]

[0220] As indicated above, this can then be incorporated into the calculation of Shannon's entropy H.

[0221]

[0222] Where Class_Cnt is the total number of classes (e.g., in...). Figure 14A In the table, Class_Cnt is 4). It will be understood that, by convention, "0log Class_Cnt "0" is 0 in these calculations.

[0223] People will understand. Figure 14B and Figure 14C The approaches can be complementary. Therefore, in some embodiments, both can be performed and the uncertainty is determined as the average of their results.

[0224] For completeness, as discussed, when model 1305k is a generative model, uncertainty can be measured from the prediction of 1305l rather than by considering the outputs of multiple models as described above. For example, in Figure 14DIn this example, the fusion model is the generative model 1425b, which is configured to receive the connected feature results 1425a and output predictions 1425c, 1425d, and 1425e (in this example, only three tasks are predicted). For example, a Bayesian neural network can output a distribution, selecting the distribution with the highest probability as the prediction (here, prediction 1425d). The uncertainty logic 1305m here can assess the uncertainty based on the variance of the predicted distribution 1425d.

[0225] One will understand additional methods used to assess uncertainty. For example, in the case that the fusion model 1305k is a neural network, the traversal application during inference with various node drops in the neural network can similarly produce a distribution similar to the prediction distribution 1425d, and the uncertainty logic 1305m can calculate the uncertainty based on the variance of the prediction distribution 1425d. With dummy values ​​already inserted into the stream, one will understand that this stream can be omitted from the above uncertainty analysis (e.g., uncertainty logic 1305m can consider only the non-dummy stream).

[0226] Example real-time processing

[0227] As discussed herein, various disclosed embodiments can be applied in real time during surgical procedures, for example, on a patient-side trolley 130 or a surgeon's console 155 or a computer system located in the operating room. Figure 15 This is a flowchart illustrating various operations in example process 1505 for a real-time application of the various systems and methods described herein. Specifically, in block 1505a, the computer system can receive frames from an ongoing surgical procedure. Until a sufficient amount of data is received in block 1505b to perform a prediction (e.g., enough frames to generate a downsampled dataset for at least one stream), the system can delay a timeout interval in block 1505c (e.g., there is enough data to process each corresponding stream for each available model).

[0228] Once a sufficient number of frames have been received at box 1505b, the system can consider at box 1505p whether dummy replacement variables would be appropriate (e.g., if there is enough data to perform a prediction, but for less than all streams, or if the data source is offline or otherwise unavailable). If so, dummy values ​​can be inserted at box 1505q as described herein. Predictions can then be performed at box 1505d on the predictions of the available data streams (replacing the unavailable ones with dummy values). At box 1505e, if the uncertainty corresponding to the prediction outcome is still unacceptable, for example, below a threshold (e.g., entropy greater than half of the maximum possible entropy), Figure 14AIf each average value in the data is less than 0.5 (e.g.), the system can wait for another timeout interval in block 1505g, receive an additional frame of the ongoing surgery in block 1505h, and perform a new prediction using the available frames in block 1505d. In some embodiments, even if the uncertainty is unacceptable, a provisional prediction result can be reported in block 1505f. It will be understood that the uncertainty calculation can be adjusted by inserting dummy values ​​(e.g., ignoring the flow or introducing random numerical values ​​or medians in the uncertainty calculation).

[0229] Once acceptable uncertainty has been achieved, the system can report the prediction results to any consuming downstream applications (e.g., a cloud-based surgical assistant) at box 1505i. Then, at box 1505j, the prediction can be periodically checked to determine if the task has changed (or if the initial prediction was incorrect) until the end of the session. Therefore, at box 1505k, the system can receive additional data from the ongoing surgery, reconsider the suitability of a dummy replacement at box 1505r, and if so, insert such a dummy value at box 1505s before incorporating the new data into the new prediction at box 1505l. At box 1505m, if the new prediction is the same as the previous most certain prediction, or if the uncertainty of the new prediction is sufficiently high, the system can wait for an additional timeout interval at box 1505n. However, if the uncertainty of the prediction at box 15051 is lower than the uncertainty achieved by the previous prediction and the prediction differs, the system can update the results at box 1505o (e.g., based on a change in the surgical procedure to the next task). The output prediction results can facilitate, for example, operations in real-time digital assistants, tool optimization algorithms, and providing alerts to surgical personnel.

[0230] As discussed below, smoothing can be applied in real time with each new forecast or after a period of time (e.g., 2 minutes, depending on the needs of the application under consideration).

[0231] Smoothing - Overview

[0232] While the systems and methods described above may be sufficient in some cases, experiments have shown that smoothing the predicted outputs from one or more fusion models 1305k can yield more feasible results in certain situations. Specifically, Figure 16A This is a flowchart illustrating various operations in a classification process with smoothing processing 1600, which may be applied in some embodiments. As previously described, the system may receive one or more of video, kinematic, or system event data at block 1605 and apply the classifier (and possibly dummy variables) described herein at block 1610 to obtain predictions. However, predictions obtained over time (e.g., such as...) Figure 15(In the real-time example) it can benefit from post-processing, such as the smoothing operation in box 1615.

[0233] like Figure 16B As shown, the initial task predictions 1630a made by the fusion model 1305k over time 1620 may include many false positives. In this example, there are only four tasks, but it will be understood that such false positives can increase as more tasks are considered. Applying a smoothing operation 1625 can help reduce such false positives, providing a more consistent and realistic output 1630b.

[0234] In some embodiments, smoothing can be achieved by moving the window in time over task prediction 1630a and assigning the majority vote within the window as the corresponding value in the final output 1630b. However, experiments show that sometimes improved results can be achieved by using a Hidden Markov Model (HMM). Figure 16C This is a state transition diagram illustrating a hypothetical set of task transition operations that may be implemented in some embodiments. Specifically, for four tasks T1, T2, T3, and T4, the model may assume the probability of transitioning to any other task (e.g., when predictions are made on a second-by-second basis) or returning to a task (e.g., when the same task is predicted for consecutive intervals) in the prediction of the next interval, given the task classification of the current interval. For example, when data is acquired at one-second intervals, the model may indicate the probability of the next second interval being a given task based on the classification of the current interval.

[0235] The HMM applied to smoothing can depend on several probabilities of its operation. Specifically, a "start task probability" can be assigned to each task, indicating the likelihood that the prediction for the first interval is for that task. For example, Figure 16E yes Figure 16C Example task state start probability set of the transition graph.

[0236] "Transition probability" can indicate the probability of remaining in the task state during the following interval or transitioning to another given task in the next interval. For example, Figure 16D yes Figure 16C The example state transition probability matrix of the transition diagram. Therefore, for example, the cell referenced by row T2 and column T4 indicates a probability of 0.002 of transitioning from task T2 in the first interval to task T4 in the next consecutive second.

[0237] The "launch probability" indicates how likely it is that a prediction for a given mission will become the actual mission within a given interval. Such probabilities can also be referenced in a matrix. For example, Figure 16E yes Figure 16CThe transition graph is an example of a task state emission probability matrix. Therefore, if the prediction is T2 (i.e., row T2), the probability that surgery actually occurs in T1 is 0.05, the probability that surgery actually occurs in T2 is 0.91, the probability that it occurs in task T3 is 0.01, and the probability that it occurs in task T4 is 0.03.

[0238] Using Hidden Markov Models (HMMs) and these probabilities, the system can iterate along the initial prediction 1630a and adjust the output 1630b to the most likely task classification based on the HMM, where the HMM differs from the initial prediction. For example, one would understand that HMMs can be used in conjunction with forward-backward algorithms to smooth the initial prediction 1630a.

[0239] Smoothing - HMM Probability Determination Method

[0240] In some embodiments, Figure 16C The probability of the model in can be based on Figure 17B , Figure 17C and Figure 17E The process is determined by this. Each of these methods can consider multiple task-annotated surgical procedures ordered by time, such as... Figure 17A As shown in the example. Typically, by considering the occurrence of tasks over time, the system can infer the frequency of task occurrence, transitions, and corresponding probabilities.

[0241] For example, Figure 17B The diagram is used to determine the starting probability (e.g., Figure 16F The process in block 1705. In block 1705a, the system can receive data annotated with tasks, such as... Figure 17A The data is shown in the diagram. In box 1705b, the system can initialize the start probability of each task to 0. In boxes 1705c and 1705d, the system can then consider... Figure 17A For each surgery in the procedure. In box 1705e, the system can consider the first task in the surgery and increase the corresponding start probability in box 1705f. Therefore, for surgery 1, the start probability of task T3 can be increased, for surgery 2, the probability of task T1 can be increased, and so on. After all surgeries have been considered, each probability value can be divided by the total number of surgeries (here, a total of NN surgeries) in box 1705g and output in box 1705h. Therefore, procedure 1705 is based on the task as the start task in the procedure. Figure 17A The probability of starting is determined by the occurrence of the words in the corpus.

[0242] Similarly, Figure 17C Depicting the probabilities used to determine the transition matrix (e.g.) Figure 16D The process 1710 (of the matrix). Similarly, in block 1710a, the system can receive surgical procedures (e.g., annotated with task descriptions) Figure 17A (Annotated surgical procedures). In box 1710b, each entry of the matrix can be initialized to zero and the sum counter is set to 0 in box 1710c. In boxes 1710d and 1710e, the system can then iterate through each surgical procedure. In boxes 1710f and 1710g, for each of these surgical procedures, all intervals (e.g., intervals of the same size used during training and inference in model 600) can be considered in the surgical procedure. Specifically, box 1710g considers interval pairs in the procedure. For example, refer to Figure 17D The surgical tasks are categorized such that at time intervals 1 and 2, the surgeon may be performing task 1, while at times 3, 4, and 5, the surgeon may be performing task 2. Therefore, the first traversal of box 1710g can consider tasks at time intervals 1 and 2, the second traversal of box 1710g can consider tasks at times 2 and 3, and so on. At each of these pairs, in box 1710h, the system can grow the corresponding transition matrix entries based on the baseline truth data and grow the SUM counter in box 1710i. After considering all surgical procedures in the corpus and the task assignment pairs for each surgical procedure across consecutive time intervals, in box 1710j, the system can divide each entry of the matrix by the value of the SUM counter, and output the result in box 1710k. Thus, the matrix again reflects the frequency of each transition occurring in the corpus.

[0243] People will understand, usually, especially Figure 17D When the time intervals between tasks are short, the self-transformation rate of many tasks will be quite high. In fact, as... Figure 16D As shown, values ​​on the diagonal are several orders of magnitude larger than those not on the diagonal. Nevertheless, these tiny differences can produce accurate results for the overall task order. For example, in cases where tasks are frequently performed in a specific order, that order can be captured in these values, and although their size is relatively small, they facilitate smoothing using an HMM.

[0244] Finally, some embodiments can be used Figure 17E The process 1715 is used to infer the emission probability (e.g., in...). Figure 16E (in the matrix). Here, the system can rely on the results from the fusion model 1305k during training instead of... Figure 17A Surgical procedures (although model 1305k can already be used) Figure 17A (Data training). Specifically, in box 1715a, the system can receive predictions from the fusion model and compute a confusion matrix in box 1715b (by comparing the predictions of the fusion model with true positive values ​​from the annotations of the training data). In box 1715c, normalizing this confusion matrix can then provide an emission probability matrix that can be used in the HMM.

[0245] People will understand that hmmlearn TM The library can be used to perform various operations among these. The described parameters can be learned from the data using Expectation-Maximization (such as the Baum-Welch algorithm). Some embodiments can leverage subject-specific knowledge about the task from experts to modify the probabilities determined in this way.

[0246] Example Implementation Results

[0247] Figure 18A This is a table of example datasets used for training and validating example implementations. Human annotators identify the task for each part of the cases (i.e., each case can be represented by a time series of task annotation identifiers per second, such as [0,0,0,1,1,1,1,1,1,1,1,...,2,2,2,2,2,2,...,0,0,0,…,3,3,3]). Each dataset can have its own corresponding task. After training and inference testing, recall precision values ​​are calculated for each dataset between human annotations and ML predictions for each task, and averaged to produce a result in... Figure 18B , Figure 18C , Figure 19A , Figure 19B and Figure 19C The average recall precision in the bar chart. Specifically, Figure 18B This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on the "prostatectomy" dataset. Figure 18C This is a bar chart showing the average recall accuracy results for various tasks on an example implementation model trained on the "Pig Training Lab" dataset. Figure 19A This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on the “cholecystectomy” dataset. Figure 19B This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on the "hysterectomy" dataset. Figure 19C This is a bar chart showing the average recall precision results for various tasks on an example implementation model trained on the "Inguinal Hernia" dataset.

[0248] A mean recall precision close to 100 indicates perfect performance of the machine learning model in identifying a specific task within a program type. Conversely, a value close to 0 indicates poor performance. As shown in the figure, performance depends on the program type, the task, and the amount of training data available.

[0249] Computer System

[0250] Figure 20 This is a block diagram of an example computer system that can be used in conjunction with some embodiments. The computing system 2000 may include interconnects 2005 that connect various components, such as one or more processors 2010, one or more memory units 2015, one or more input / output systems 2020, one or more storage systems 2025, one or more network adapters 2030, etc. The interconnects 2005 may be, for example, one or more bridges, traces, buses (e.g., ISA, SCSI, PCI, I2C, Firewire buses, etc.), wires, adapters, or controllers.

[0251] One or more processors 2010 may include, for example, Intel TM Processor chips, math coprocessors, graphics processors, etc. One or more memory components 2015 may include, for example, volatile memory (RAM, SRAM, DRAM, etc.), non-volatile memory (EPROM, ROM, flash memory, etc.), or similar devices. One or more input / output devices 2020 may include, for example, display devices, keyboards, pointing devices, touchscreen devices, etc. One or more storage devices 2025 may include, for example, cloud-based storage, removable USB storage, disk drives, etc. In some systems, memory components 2015 and storage devices 2025 may be the same component. Network adapter 2030 may include, for example, a wired network interface, a wireless interface, Bluetooth, etc. TM Adapters, line-of-sight interfaces, etc.

[0252] People will recognize that in some embodiments, only some, alternative, or all of the components may be present. Figure 20 Additional components beyond those depicted. Similarly, in some systems, components may be combined or used for dual purposes. These components can be implemented using dedicated hardwired circuit systems, such as one or more ASICs, PLDs, FPGAs, etc. Therefore, some embodiments may be implemented in, for example, programmable circuit systems (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in dedicated hardwired (non-programmable) circuit systems, or in a combination of these forms.

[0253] In some embodiments, data structures and message structures may be stored or transmitted via network adapter 2030 via a data transmission medium (e.g., signals on a communication link). Transmissions can occur across various media, such as the Internet, local area networks, wide area networks, or point-to-point dial-up connections. Therefore, "computer-readable medium" can include computer-readable storage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0254] One or more memory components 2015 and one or more storage devices 2025 may be computer-readable storage media. In some embodiments, one or more memory components 2015 or one or more storage devices 2025 may store instructions that can perform or cause to perform the various operations discussed herein. In some embodiments, the instructions stored in memory 2015 may be implemented as software and / or firmware. These instructions may be used to perform operations on one or more processors 2010 to perform the processes described herein. In some embodiments, such instructions may be provided to one or more processors 2010 by, for example, downloading instructions from another system via network adapter 2030.

[0255] Comment

[0256] The accompanying drawings and descriptions are illustrative. Therefore, neither the specification nor the drawings should be construed as limiting this disclosure. For example, titles or subtitles are provided merely for convenience and to facilitate understanding. Therefore, titles or subtitles should not be construed as limiting the scope of this disclosure, for example, by grouping or simply combining features presented in a particular order to facilitate understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the event of conflict, this document (including any definitions provided herein) shall prevail. The use of one or more synonyms herein does not preclude the use of other synonyms. The use of examples anywhere in this specification (including examples of any terms discussed herein) is merely illustrative and not intended to further limit the scope and meaning of this disclosure or any illustrative terms.

[0257] Similarly, although specific representations are made in the accompanying figures, those skilled in the art will understand that the actual data structures used to store information can differ from those shown. For example, data structures may be organized in different ways, may contain more or less information than shown, may be compressed and / or encrypted, etc. Common or well-known details may be omitted from the figures and disclosure to avoid confusion. Similarly, the figures may depict specific sets of operations to facilitate understanding, which are merely examples of a broader set of such operations. Thus, it will be readily apparent that the same purpose or effect depicted in some flowcharts can often be achieved using additional, alternative, or fewer operations. For example, data may be encrypted, although not shown in the figures, items may be considered different loop patterns (“for” loops, “while” loops, etc.), or ordered in different ways to achieve the same or similar effects, etc.

[0258] The references to “embodiment” or “an embodiment” herein mean that at least one embodiment of this disclosure includes the specific features, structures, or characteristics described in connection with that embodiment. Therefore, the phrase “in one embodiment” in various places herein does not necessarily refer to the same embodiment in every one of those places. Individual or alternative embodiments may not be mutually exclusive with other embodiments. It will be appreciated that various modifications can be made without departing from the scope of the embodiments.

Claims

1. A computer-implemented method for determining surgical task classifications of surgical data, the method comprising: A first task classification is determined using a first machine learning model to classify kinematics based on surgical data of a surgical procedure, the first task classification comprising one or more first possible task classifications and one or more corresponding first probability values. A second machine learning model is used to determine a second task classification to classify kinematics based on surgical data of the surgical procedure. The second task classification includes one or more second possible task classifications and one or more corresponding second probability values. The first task classification and the second task classification are used to generate a merged vector, the merged vector including the one or more first possible task classifications, the one or more corresponding first probability values, the one or more second possible task classifications, and the one or more corresponding second probability values; as well as The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

2. The computer-implemented method according to claim 1, the method further comprising: The kinematics derived from the surgical data is the first kinematics. A third machine learning model is used to determine a third task classification to classify the second kinematics based on surgical data of the surgical procedure. The third task classification includes one or more third possible task classifications and one or more corresponding third probability values. as well as A fourth machine learning model is used to determine a fourth task classification to classify system events based on surgical data. This fourth task classification includes one or more possible fourth task classifications and one or more corresponding fourth probability values. A merged vector is generated using the first task classification, the second task classification, the third task classification, and the fourth task classification. The merged vector includes one or more first possible task classifications, one or more corresponding first probability values, one or more second possible task classifications, one or more corresponding second probability values, one or more third possible task classifications, one or more corresponding third probability values, one or more fourth possible task classifications, and one or more corresponding fourth probability values. The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

3. The computer-implemented method according to claim 1, wherein the first machine learning model comprises: A multi-layer convolutional neural network, wherein the input of the multi-layer convolutional neural network is configured to communicate with at least one video frame feature input; A one-dimensional convolutional neural network, wherein the input of the one-dimensional convolutional neural network is configured to communicate with the output from the multi-layer convolutional neural network; as well as A recurrent neural network, wherein the input of the recurrent neural network is configured to communicate with the output from the one-dimensional convolutional neural network.

4. The computer-implemented method of claim 3, wherein the input of the one-dimensional convolutional neural network is configured to receive a connection of outputs from one or more multi-layer convolutional neural networks.

5. The computer-implemented method according to claim 3 or claim 4, wherein the multilayer convolutional neural network of the first machine learning model comprises a VGG19 neural network.

6. The computer-implemented method according to claim 2, wherein, The first machine learning model includes one or more neural networks. The third machine learning model includes one or more neural networks, and wherein Each of the one or more neural networks in the first machine learning model and the third machine learning model includes: One-dimensional convolutional layers; and Recurrent neural network layer.

7. The computer-implemented method according to claim 2 or claim 6, wherein the fourth machine learning model comprises: One or more basic machine learning models; as well as One or more fusion machine learning models.

8. The computer-implemented method of claim 7, wherein the one or more underlying machine learning models of the fourth machine learning model include at least one of a logistic regression classifier, a random forest, or a neural network.

9. The computer-implemented method of claim 8, wherein the one or more fusion machine learning models of the fourth machine learning model include at least one of random forest or extreme random tree.

10. The computer-implemented method according to claim 2 or claim 6, the method further comprising: The output of at least one of the first machine learning model, the second machine learning model, the third machine learning model, or the fourth machine learning model is replaced with a dummy value in the merged classification fusion machine learning model, and wherein... The merged classification fusion machine learning model is configured to generate predictive classifications of the surgical data in part based on the dummy values.

11. The computer-implemented method according to claim 2, the method further comprising: A smoothing operation is applied to the merged task classification prediction of the surgical data from the merged classification fusion machine learning model.

12. The computer-implemented method according to claim 11, wherein, The smoothing operation includes application-specific or hidden Markov model filters. The state of the hidden Markov model corresponds to the surgical task, and where, Each state of the hidden Markov model is associated with a start probability, a transition probability, and a firing probability.

13. The computer-implemented method according to claim 12, the method further comprising: Determine the uncertainty associated with the task classification predictions to be merged.

14. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, the method comprising: A first task classification is determined using a first machine learning model to classify kinematics based on surgical data of a surgical procedure, the first task classification comprising one or more first possible task classifications and one or more corresponding first probability values. A second machine learning model is used to determine a second task classification to classify video data based on surgical data of a surgical procedure, the second task classification including one or more second possible task classifications and one or more corresponding second probability values. The first task classification and the second task classification are used to generate a merged vector, the merged vector including the one or more first possible task classifications, the one or more corresponding first probability values, the one or more second possible task classifications, and the one or more corresponding second probability values; as well as The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

15. The non-transitory computer-readable medium of claim 14, wherein the method further comprises: The kinematics derived from the surgical data is the first kinematics. A third machine learning model is used to determine a third task classification to classify the second kinematics based on surgical data of the surgical procedure. The third task classification includes one or more third possible task classifications and one or more corresponding third probability values. as well as A fourth machine learning model is used to determine a fourth task classification to classify system events based on surgical data. This fourth task classification includes one or more possible fourth task classifications and one or more corresponding fourth probability values. A merged vector is generated using the first task classification, the second task classification, the third task classification, and the fourth task classification. The merged vector includes one or more first possible task classifications, one or more corresponding first probability values, one or more second possible task classifications, one or more corresponding second probability values, one or more third possible task classifications, one or more corresponding third probability values, one or more fourth possible task classifications, and one or more corresponding fourth probability values. The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

16. The non-transitory computer-readable medium of claim 15, wherein the second machine learning model comprises: A multi-layer convolutional neural network, wherein the input of the multi-layer convolutional neural network is configured to communicate with at least one video frame feature input; A one-dimensional convolutional neural network, wherein the input of the one-dimensional convolutional neural network is configured to communicate with the output from the multi-layer convolutional neural network; as well as A recurrent neural network, wherein the input of the recurrent neural network is configured to communicate with the output from the one-dimensional convolutional neural network.

17. The non-transitory computer-readable medium of claim 16, wherein the input of the one-dimensional convolutional neural network is configured to receive a connection of the outputs from one or more multi-layer convolutional neural networks.

18. The non-transitory computer-readable medium of claim 16 or claim 17, wherein the multilayer convolutional neural network of the first machine learning model comprises a VGG19 neural network.

19. The non-transitory computer-readable medium according to claim 15, wherein, The second machine learning model includes one or more neural networks. The third machine learning model includes one or more neural networks, and wherein Each of the one or more neural networks in the second machine learning model and the third machine learning model includes: One-dimensional convolutional layers; and Recurrent neural network layer.

20. The non-transitory computer-readable medium of claim 15 or claim 19, wherein the fourth machine learning model comprises: One or more basic machine learning models; as well as One or more fusion machine learning models.

21. The non-transitory computer-readable medium of claim 20, wherein the one or more underlying machine learning models of the fourth machine learning model include at least one of a logistic regression classifier, a random forest, or a neural network.

22. The non-transitory computer-readable medium of claim 21, wherein the one or more fusion machine learning models of the fourth machine learning model include at least one of random forest or extreme random tree.

23. The non-transitory computer-readable medium according to claim 15, claim 16, or claim 19, wherein the method further comprises: The output of at least one of the first machine learning model, the second machine learning model, the third machine learning model, or the fourth machine learning model is replaced with a dummy value in the merged classification fusion machine learning model, and wherein... The merged classification fusion machine learning model is configured to generate predictive classifications of the surgical data in part based on the dummy values.

24. The non-transitory computer-readable medium of claim 15, wherein the method further comprises: A smoothing operation is applied to the merged task classification prediction of the surgical data from the merged classification fusion machine learning model.

25. The non-transitory computer-readable medium according to claim 24, wherein, The smoothing operation includes application-specific or hidden Markov model filters. The state of the hidden Markov model corresponds to the surgical task, and where, Each state of the hidden Markov model is associated with a start probability, a transition probability, and a firing probability.

26. The non-transitory computer-readable medium of claim 25, wherein the method further comprises: Determine the uncertainty associated with the task classification predictions to be merged.

27. A computer system comprising: At least one processor; At least one memory, the at least one memory including instructions configured to cause the computer system to perform a method, the method comprising: A first task classification is determined using a first machine learning model to classify kinematics based on surgical data of a surgical procedure, the first task classification comprising one or more first possible task classifications and one or more corresponding first probability values. A second machine learning model is used to determine a second task classification to classify video data based on surgical data of a surgical procedure, the second task classification including one or more first possible task classifications and one or more corresponding second probability values. The first task classification and the second task classification are used to generate a merged vector, the merged vector including the one or more first possible task classifications, the one or more corresponding first probability values, the one or more second possible task classifications, and the one or more corresponding second probability values; and The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

28. The computer system of claim 27, wherein the kinematics from the surgical data is a first kinematics, and the method further comprises: A third machine learning model is used to determine a third task classification to classify the second kinematics based on surgical data of the surgical procedure. The third task classification includes one or more third possible task classifications and one or more corresponding third probability values. A fourth machine learning model is used to determine a fourth task classification to classify system events based on surgical data. This fourth task classification includes one or more possible fourth task classifications and one or more corresponding fourth probability values. A merged vector is generated using the first task classification, the second task classification, the third task classification, and the fourth task classification. The merged vector includes one or more first possible task classifications, one or more corresponding first probability values, one or more second possible task classifications, one or more corresponding second probability values, one or more third possible task classifications, one or more corresponding third probability values, one or more fourth possible task classifications, and one or more corresponding fourth probability values. The merged vectors are used as input to the merged task classification model to determine the surgical task classification prediction for the surgical procedure.

29. The computer system of claim 28, wherein the first machine learning model comprises: A multi-layer convolutional neural network, wherein the input of the multi-layer convolutional neural network is configured to communicate with at least one video frame feature input; A one-dimensional convolutional neural network, wherein the input of the one-dimensional convolutional neural network is configured to communicate with the output from the multi-layer convolutional neural network; as well as A recurrent neural network, wherein the input of the recurrent neural network is configured to communicate with the output from the one-dimensional convolutional neural network.

30. The computer system of claim 29, wherein the input of the one-dimensional convolutional neural network is configured to receive a connection of outputs from one or more multi-layer convolutional neural networks.

31. The computer system of claim 29 or claim 30, wherein the multilayer convolutional neural network of the first machine learning model comprises a VGG19 neural network.

32. The computer system according to claim 28, wherein, The second machine learning model includes one or more neural networks. The third machine learning model includes one or more neural networks, and wherein Each of the one or more neural networks in the second machine learning model and the third machine learning model includes: One-dimensional convolutional layers; and Recurrent neural network layer.

33. The computer system of claim 28 or claim 32, wherein the fourth machine learning model comprises: One or more basic machine learning models; as well as One or more fusion machine learning models.

34. The computer system of claim 33, wherein the one or more underlying machine learning models of the fourth machine learning model include at least one of a logistic regression classifier, a random forest, or a neural network.

35. The computer system of claim 34, wherein the one or more fusion machine learning models of the fourth machine learning model include at least one of random forest or extreme random tree.

36. The computer system according to claim 28, claim 29 or claim 32, wherein the method further comprises: The output of at least one of the first machine learning model, the second machine learning model, the third machine learning model, or the fourth machine learning model is replaced with a dummy value in the merged classification fusion machine learning model, and wherein... The merged classification fusion machine learning model is configured to generate predictive classifications of the surgical data in part based on the dummy values.

37. The computer system of claim 28, wherein the method further comprises: A smoothing operation is applied to the merged task classification prediction of the surgical data from the merged classification fusion machine learning model.

38. The computer system according to claim 37, wherein, The smoothing operation includes application-specific or hidden Markov model filters. The state of the hidden Markov model corresponds to the surgical task, and where, Each state of the hidden Markov model is associated with a start probability, a transition probability, and a firing probability.

39. The computer system according to claim 38, wherein the method further comprises: Determine the uncertainty associated with the task classification predictions to be merged.

Images