Systems and methods for evaluating surgical skills

Abstract

This invention provides a computer system and a computer-implemented method for measuring and monitoring surgical performance. [Solution] In the various embodiments disclosed, for example, the system may receive raw data acquired from an operating room, generate and select features from the data suitable for analysis, and use the selected features to train a machine learning classifier to facilitate subsequent evaluation of the performance of other surgeons. The generation and selection of features themselves may, in some embodiments, involve the application of a machine learning classifier. Some embodiments consider raw data acquired from a surgical robot system, while some embodiments facilitate evaluation with data acquired from a non-robot operating room.

Description

[Technical Field]

[0001] [Cross-Reference to Related Applications] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 121,220, filed on December 3, 2020 and entitled "SYSTEMS AND METHODS FOR ASSESSING SURGICAL ABILITY", which is hereby incorporated by reference in its entirety into this application for all purposes.

[0002] Various disclosed embodiments relate to computer systems and computer-implemented methods for measuring and monitoring surgical performance. [Background Art]

[0003] Many challenges complicate the assessment of surgical skills, making it extremely difficult to provide meaningful feedback on surgical performance to surgeons. For example, a specific surgical skill cannot actually be assessed based solely on postoperative outcomes, because multiple cumulative factors unrelated to skill contribute to the final result and obscure the impact of a single skill. Alternatively, a surgeon's skill can be observed directly during surgery or via recorded video, but such real-time and video-based reviews require a human expert, such as a senior surgeon, to recognize and assess the surgical skill presented in the operating room or on the video. Unfortunately, assessments by such human observers are often subjective, low in scale (at least because of the requirement for a human reviewer), and can be difficult to arrange since "specialist" surgeons for a given type of surgery are often far fewer in number than "novice" surgeons generating videos. Furthermore, many specialist surgeons are in high demand and are naturally reluctant to devote time to reviewing such videos instead of performing their own surgeries.

[0004] The data collection capabilities of new surgical tools and surgical robot systems have made vast amounts of surgical data available. However, this data not only fails to solve the aforementioned problems but also presents its own unique challenges that must now be overcome. For example, raw data rarely directly correlates with specific surgical techniques, so reviewers must struggle to infer the correlation between the techniques they want to review and the raw data available for review. Similarly, the aforementioned significant asymmetry between the "expert" and "novice" surgeons is often reflected in the collected data, complicating any effort to perform automated data analysis.

[0005] Therefore, there is a need for a scalable, automated surgical technique assessment system that reduces reliance on experts for manual review. Similarly, a system is needed that can account for the considerable asymmetry in available data between experts and non-experts. Such a system would also, ideally, conduct assessments in a way that is suitable for providing actionable feedback to surgeons. [Brief explanation of the drawing]

[0006] The various embodiments described here can be better understood by referring to the following detailed description in conjunction with the attached drawings. In the attached drawings, the same reference numerals indicate elements that are identical or functionally similar.

[0007] [Figure 1A] This is a schematic diagram of various elements that may appear in the operating room during a surgical procedure in relation to several embodiments.

[0008] [Figure 1B] This is a schematic diagram of various elements that may appear in the operating room during surgical procedures using a surgical robot, which may occur in relation to several embodiments.

[0009] [Figure 2A] This is a schematic Euler diagram illustrating the conventional grouping of machine learning models and methodologies.

[0010] [Figure 2B] Figure 2A is a schematic diagram illustrating various operations of an exemplary unsupervised learning method following the conventional grouping.

[0011] [Figure 2C] Figure 2A is a schematic diagram illustrating various operations of an exemplary supervised learning method following the conventional grouping.

[0012] [Figure 2D] Figure 2A is a schematic diagram illustrating various operations of an exemplary semi-supervised learning method following the conventional grouping.

[0013] [Figure 2E] Figure 2A is a schematic diagram illustrating various operations of an exemplary reinforcement learning method following the conventional grouping.

[0014] [Figure 2F] This is a schematic block diagram illustrating the relationships between machine learning models, machine learning model architectures, machine learning methodologies, machine learning methods, and machine learning implementations.

[0015] [Figure 3A] This is a schematic diagram illustrating the operation of various aspects of an exemplary Support Vector Machine (SVM) machine learning model architecture.

[0016] [Figure 3B] This is a schematic diagram illustrating various aspects of how an exemplary random forest machine learning model architecture operates.

[0017] [Figure 3C] This is a schematic diagram illustrating various modes of operation of exemplary neural network machine learning model architectures.

[0018] [Figure 3D]It is a schematic diagram of possible relationships between inputs and outputs at nodes of the exemplary neural network architecture of FIG. 3C.

[0019] [Figure 3E] It is a schematic diagram of exemplary input-output relationship variations that can occur in a Bayesian neural network.

[0020] [Figure 3F] It is a schematic diagram of various aspects of the operation of an exemplary deep learning architecture.

[0021] [Figure 3G] It is a schematic diagram of various aspects of the operation of an exemplary ensemble architecture.

[0022] [Figure 3H] It is a schematic block diagram showing various operations of an exemplary machine learning pipeline topology.

[0023] [Figure 4A] It is a schematic flow diagram showing various operations common to various machine learning model training methods.

[0024] [Figure 4B] It is a schematic flow diagram showing various operations common to various machine learning model inference methods.

[0025] [Figure 4C] It is a schematic flow diagram showing various iterative training operations that occur at block 405b in some architectures and training methods.

[0026] [Figure 4D] It is a schematic block diagram showing the operation of various machine learning methods that lack a strict distinction between training methods and inference methods.

[0027] [Figure 4E]This is a schematic block diagram illustrating an exemplary relationship between the architecture training method and the inference method.

[0028] [Figure 4F] This is a schematic block diagram illustrating the exemplary relationship between machine learning model training methods and inference methods, where the training method involves various data subset operations.

[0029] [Figure 4G] This is a schematic block diagram illustrating an exemplary decomposition of training data into training subsets, validation subsets, and test subsets.

[0030] [Figure 4H] This is a schematic block diagram illustrating various operations in a training method that incorporates transfer learning.

[0031] [Figure 4I] This is a schematic block diagram illustrating various operations in an exemplary training method that incorporates online learning;

[0032] [Figure 4J] This is a schematic block diagram illustrating the various components of an exemplary generative adversarial network method.

[0033] [Figure 5A] This is a schematic diagram of surgical data that may be received by the processing system in some embodiments.

[0034] [Figure 5B] This is a table of exemplary tasks that may be used in combination with various disclosed embodiments.

[0035] [Figure 6] This is a schematic diagram illustrating the information flow for performing surgical skill (or task) assessments that may occur in several embodiments.

[0036] [Figure 7A] This flowchart illustrates various operations in the process of generating and applying a skill model, which can be performed in several embodiments.

[0037] [Figure 7B] This is a schematic diagram illustrating the various components employed in an exemplary application of a skill model for determining surgical scores, which may be implemented in several embodiments.

[0038] [Figure 7C] This is a schematic diagram illustrating the application of an exemplary windowing score generation process that can be implemented in several embodiments.

[0039] [Figure 7D] This is a plot of exemplary skill score output over time, which may be created in several embodiments.

[0040] [Figure 8A] This is a schematic diagram illustrating the relationship between various metrics and data structures that may be used in several embodiments.

[0041] [Figure 8B] An exemplary raw data input, specifically a schematic diagram of the forceps translational motion in three-dimensional space, which may be used to generate one or more objective performance indicators (OPIs) in some embodiments.

[0042] [Figure 8C] An exemplary raw data input, specifically a schematic diagram of multiple rotations in three-dimensional space around multiple forceps component axes, which may be used to generate one or more OPIs in some embodiments.

[0043] [Figure 8D]This is a pair of tables showing exemplary mappings of skills to OPIs and skills to tasks, which may be applied in several embodiments.

[0044] [Figure 9A] This schematic diagram illustrates an exemplary set of relationships between skills, skill models, and OPIs, which can be implemented in several embodiments.

[0045] [Figure 9B] This is a schematic block diagram illustrating the operation of an OPI-related evaluation system for filtering an OPI subset of a given skill or task, which may be implemented in several embodiments.

[0046] [Figure 9C] This is a schematic diagram illustrating an exemplary filtering framework that may be implemented in an OPI-related evaluation system in several embodiments.

[0047] [Figure 10A] This flowchart illustrates an exemplary process for selecting OPIs by single OPI statistical distribution analysis (SOSDA) or multi-OPI statistical distribution analysis (MOSDA) filtering, which can be implemented in several embodiments.

[0048] [Figure 10B] These are illustrative schematic plots of similar and dissimilar expert and non-expert OPI value distributions, which may be considered in several embodiments.

[0049] [Figure 10C] This flowchart illustrates an exemplary process for performing OPI selection using a Multi-OPI Prediction Model (MOPM) filter, which can be implemented in several embodiments.

[0050] [Figure 11A]This flowchart illustrates various operations in an exemplary process for evaluating skill (or task) model configurations and OPI selections, which can be implemented in several embodiments.

[0051] [Figure 11B] This is an exemplary training dataset breakdown that may be applied when training according to the process shown in Figure 11A in some embodiments.

[0052] [Figure 12A] This is a schematic plot of the training feature vectors and inference feature vectors in the abstract feature space that may occur in some embodiments.

[0053] [Figure 12B] This flowchart illustrates various operations in an exemplary process for determining score mappings from model outputs based on a reference population, which can be performed in several embodiments.

[0054] [Figure 12C] This is a schematic diagram illustrating an exemplary application of reference group-based scoring, which may be performed in some embodiments.

[0055] [Figure 13A] This is a schematic block diagram showing a hierarchical input / output topology that may be used for score generation in some embodiments.

[0056] [Figure 13B] This is a schematic screenshot of a graphical user interface depicting a performance metric overlay on a video of a surgeon's performance, which can be implemented in several embodiments.

[0057] [Figure 13C] This flowchart illustrates various operations in an exemplary update process for a skills assessment system, which can be implemented in several embodiments.

[0058] [Figure 14A] This is a bar graph showing the types and quantities of data samples available for use in an exemplary reduction to the practice of one embodiment.

[0059] [Figure 14B] This table shows the mean cross-validated performance metrics, specifically equilibrium precision and Matthews correlation coefficient (MCC), for the skill task and the overall task logistic regression model, respectively, in an exemplary practical application of one embodiment.

[0060] [Figure 15] These are a pair of schematic dot plots showing the economy of motion values ​​for four instruments in the uterine horn task before and after the application of resampling in an exemplary implementation of one embodiment.

[0061] [Figure 16] This is a collection of schematic line plots showing the distribution of task duration by experience level in an example of practice reduction in an exemplary implementation of one embodiment, and cross-validation scores of various numbers of OPIs per skill using recursive feature extinction (RFE).

[0062] [Figure 17] This is a table listing an example collection of OPIs, each with a description, and their relationship to various skills and tasks.

[0063] [Figure 18] This is a table listing an example collection of OPIs, each with a description, and their relationship to various skills and tasks.

[0064] [Figure 19] This is a table listing an example collection of OPIs, each with a description, and their relationship to various skills and tasks.

[0065] [Figure 20] This is a table listing an example collection of OPIs, each with a description, and their relationship to various skills and tasks.

[0066] [Figure 21] This is a block diagram of an exemplary computer system used in combination with some of the embodiments.

[0067] The specific examples depicted in the drawings have been selected for ease of understanding. Therefore, the disclosed embodiments should not be limited to specific details or corresponding disclosures in the drawings. For example, the drawings may not be drawn to scale, the dimensions of some elements in the drawings may be adjusted for ease of understanding, and the behavior of embodiments related to the flowcharts may include additional, alternative, or fewer behaviors than those shown herein. Therefore, some components and / or behaviors may be separated into different blocks or combined into one block in ways other than those shown. The embodiments are intended to cover all modifications, equivalents, and alternatives within the scope of the disclosed examples, rather than limiting the embodiments to specific examples described or drawn. [Modes for carrying out the invention]

[0068] Example of an operating room overview Figure 1A is a schematic diagram of various elements that may appear in the operating room 100a during surgery in relation to several embodiments. In particular, Figure 1A shows a non-robot operating room 100a, in which a surgeon 105a on the patient's side performs surgery on a patient 120 with the assistance of one or more support members 105b, these support members themselves may be surgeons, physician's assistants, nurses, technicians, etc. The surgeon 105a may perform surgery using various tools, such as visualization tools 110b such as laparoscopic ultrasound or endoscopes, and mechanical end-effectors 110a such as scissors, retractors, and dissectors.

[0069] The visualization tool 110b provides the surgeon 105a with an internal view of the patient 120, for example, by displaying visualization output from a camera mechanically and electrically coupled to the visualization tool 110b. The surgeon can view the visualization output, for example, through an eyepiece coupled to the visualization tool 110b or on a display 125 configured to receive the visualization output. For example, if the visualization tool 110b is an endoscope, the visualization output may be a color or grayscale image. The display 125 may allow an assistant member 105b to monitor the progress of the surgeon 105a's surgery. The visualization output from the visualization tool 110b may be recorded and stored for future review, for example, by using the hardware or software of the visualization tool 110b itself, capturing the visualization output in parallel as it is provided to the display 125, or capturing the output from the display 125 as it appears on the screen. Two-dimensional video capture using visualization tool 110b is often discussed here, as is the case when visualization tool 110b is an endoscope. However, it will be understood that in some embodiments, visualization tool 110b may capture depth data instead of, or in addition to, two-dimensional image data (e.g., laser rangefinder, stereoscopic vision). Therefore, it will be understood that, if such three-dimensional depth data is available, the two-dimensional operations described herein can be applied to such three-dimensional depth data. For example, machine learning model inputs may be extended or modified to accept features obtained from such depth data.

[0070] A single surgery may involve the execution of several groups of actions, each group of actions forming a separate unit called a task. For example, locating a tumor may be the first task, excising the tumor may be the second task, and closing the surgical site may be the third task. Each task may involve multiple actions; for example, a tumor excision task may require several excision actions and several cauterization actions. Some surgeries require tasks to be performed in a specific order (e.g., excision is performed before closure), but some surgeries may allow changes to the order and presence of some tasks (e.g., elimination of prophylactic tasks or reordering excision tasks whose order does not affect them). Transitions between tasks may require the surgeon 105a to remove a tool from the patient, replace a tool with another, or introduce a new tool. Some tasks may require removing a visualization tool 110b and repositioning it relative to its position in the previous task. Some support members 105b may assist with surgery-related tasks, such as administering anesthesia 115 to the patient 120, while others may assist with transitions between these tasks, for example, anticipating the need for new tools 110c.

[0071] Technological advancements have made it possible to perform procedures such as those shown in Figure 1A using robotic systems, enabling procedures that cannot be performed in a non-robot operating room 100a. Specifically, Figure 1B is a schematic diagram of various elements that may appear in the operating room 100b during a surgical procedure using a surgical robot, such as the da Vinci® surgical system, which may occur in relation to several embodiments. Here, a patient-side cart 130 with tools 140a, 140b, 140c, and 140d attached to multiple arms 135a, 135b, 135c, and 135d, respectively, may occupy the position of the patient-side surgeon 105a. As before, the tools 140a, 140b, 140c, and 140d may include visualization tools 140d such as endoscopes and laparoscopic ultrasound. The operator 105c, who may be a surgeon, can view the output of the visualization tools 140d through a display 160a on the surgeon console 155. By operating the handheld input mechanism 160b and pedal 160c, the operator 105c can remotely communicate with the tools 140a-d on the patient-side cart 130 and perform surgical procedures on the patient 120. In fact, in some embodiments, communication between the surgeon console 155 and the patient-side cart 130 may occur via a communication network, so the operator 105c may or may not be in the same physical location as the patient-side cart 130 and the patient 120. The electronics / control console 145 may also include a display 150 that displays the patient's vital signs and / or the output of the visualization tool 140d.

[0072] Similar to task transitions in non-robot operating room 100a, surgical operations in operating room 100b may require not only removing and replacing tools 140a-d, including the visualization tool 140d, for various tasks, but also introducing new tools, such as a new tool 165. As before, one or more support members 105d can anticipate such changes and work with operator 105c to make necessary adjustments as the surgery progresses.

[0073] Furthermore, similar to the non-robot operating room 100a, the output from the visualization tool 140d can be recorded here from, for example, the patient-side cart 130, the surgeon console 155, and the display 150. While some tools 110a, 110b, and 110c in the non-robot operating room 100a can record additional data such as temperature, motion, conductivity, and energy levels, the presence of the surgeon console 155 and the patient-side cart 130 in operating room 100b makes it easier to record considerably more data than just the output from the visualization tool 140d. For example, the operation of the operator 105c's handheld input mechanism 160b, the operation of the pedal 160c, and eye movements in the display 160a can all be recorded. Similarly, the patient-side cart 130 can record the operation of tools during surgery (e.g., application of radiant energy, closing of scissors, etc.) and the movement of the end effector.

[0074] Basic Concepts of Machine Learning - Overview This section provides a basic description of the architectures and methods of machine learning models that may be relevant to the various embodiments disclosed. Machine learning constitutes a vast and heterogeneous landscape and has undergone many abrupt and overlapping developments. Given this complexity, practitioners have not always used terminology consistently or with strict clarity. Therefore, this section aims to provide a common ground to ensure a more reliable understanding of the entities of the disclosed embodiments. It will be found that it is not feasible to fully address all known machine learning models, as well as all known variations of their architectures, tasks, methods, and methodologies. Instead, it will be found that the examples discussed here are merely representative, and the various embodiments disclosed may employ many other architectures and methods than those explicitly discussed.

[0075] To guide the reader based on existing literature, Figure 2A shows a conventionally recognized group of machine learning models and methodologies, also known as methods, in the form of a schematic Euler diagram. The group in Figure 2A is explained in conventional terms with reference to Figures 2B-E before a more comprehensive explanation of the field of machine learning is provided with respect to Figure 2F.

[0076] The conventional grouping in Figure 2A typically distinguishes machine learning models and their methodologies based on the nature of the inputs a model is expected to receive, or the nature of the inputs on which the methodology is expected to operate. Unsupervised learning methodologies derive inference from input datasets that lack output metadata (also called "unlabeled data"), or by ignoring such metadata if it exists. For example, as shown in Figure 2B, an unsupervised K-nearest neighbor (KNN) model architecture may receive multiple unlabeled inputs represented by circles in feature space 205a. The feature space is the mathematical space of inputs on which a given model architecture is configured to operate. For example, if a 128x128 grayscale pixel image is provided as input to a KNN, it can be treated as a linear array of 16,384 "features" (i.e., raw pixel values). In this case, the feature space is a 16,384-dimensional space (for ease of understanding, a space of only two dimensions is shown in Figure 2B). Alternatively, for example, if a Fourier transform is applied to the pixel data, the resulting frequency magnitudes and phases could serve as "features" input to the model architecture. The input values ​​in a feature space are sometimes called feature "vectors," but it's important to note that not all model architectures expect feature inputs to be in a linear form (for example, some deep learning networks expect input features as matrices or tensors). Therefore, references to feature vectors, feature matrices, etc., should be seen as examples of possible forms that might be input to a model architecture, without context indicating otherwise. Similarly, references to "input" are understood to include feature types or forms that might be acceptable to the architecture. Continuing with the example in Figure 2B, the KNN classifier might output associations between the input vectors and the various groups determined by the KNN classifier, as represented by the squares, triangles, and hexagons shown in the figure. Thus, unsupervised methodologies may include, for example, determining clusters in the data as in this example, or reducing or changing the feature dimensions used to represent the data input.

[0077] A supervised learning model takes an input dataset with output metadata (called "labeled data"), modifies the model architecture parameters (such as neural network biases and weights, or SVM support vectors) based on this input data and metadata, and then better maps the received input to the desired output. For example, an SVM supervised classifier operates as shown in Figure 2C, taking multiple input feature vectors, represented as circles in feature space 210a, as training inputs, with output labels A, B, or C, as provided by the practitioner. According to the supervised learning methodology, the SVM modifies its parameters using these label inputs, and when the SVM receives a new input 210c in feature vector form in feature space 210a that was not previously seen, the SVM may output the desired classification "C" in its output. Thus, supervised learning methodologies can include, for example, performing classifications like this example, or performing regressions.

[0078] Semi-supervised learning methodologies inform parameter tuning of the model architecture based on both labeled and unlabeled data. For example, a supervised neural network classifier operates as shown in Figure 2D, receiving several training input feature vectors in a feature space 215a labeled with classification A, B, or C, and several such unlabeled training input feature vectors (as shown by circles without labels). Without considering unlabeled inputs, a simple supervised classifier can distinguish between B-class and C-class inputs based on a simple planar separation 215d in the feature space between the available labeled inputs. However, by considering labeled input feature vectors as well as unlabeled input feature vectors, a semi-supervised classifier can employ a more subtle separation 215e. Unlike simple separation 215d, subtle separation 215e can correctly classify a new input 215c as being in class C. Thus, semi-supervised learning methods and architectures can include applications for both supervised and unsupervised learning, where at least some of the available data is labeled.

[0079] Finally, the conventional group in Figure 2A distinguishes reinforcement learning methodologies as those in which an agent, such as a robot or digital assistant, performs some action (e.g., moving a manipulator, making a suggestion to the user) that influences the agent's environmental context (e.g., the location of objects in the environment, the user's characteristics, etc.), thereby promoting a new environmental state and some associated environment-based rewards (e.g., a positive reward when an environmental object approaches a target state, a negative reward when the user feels uncomfortable, etc.). Thus, reinforcement learning may include, for example, updating a digital assistant based on user behavior and expressed preferences, an autonomous robot navigating a factory, or a computer playing chess.

[0080] As mentioned above, many practitioners will recognize the conventional classification method in Figure 2A, but the grouping in Figure 2A obscures the rich diversity of machine learning and may fail to adequately characterize machine learning architectures and techniques that fall into multiple categories within those groups or fall entirely outside of them (for example, random forests and neural networks may be used for supervised or unsupervised learning tasks; similarly, some generative adversarial networks employ supervised classifiers but do not readily fit into any of the groups in Figure 2A). Therefore, while various terms from Figure 2A may be referenced here to facilitate the reader's understanding, this explanation should not be limited to the uniform conventions of Figure 2A. For example, Figure 2F offers a more flexible machine learning classification.

[0081] In particular, Figure 2F approaches machine learning as including model 220a, model architecture 220b, methodology 220e, method 220d, and implementation 220c. At a higher level, model architecture 220b can be considered a species of each genus model 220a (model A has possible architectures A1, A2, etc.; model B has possible architectures B1, B2, etc.). Model 220a refers to a description of a mathematical structure suitable for implementation as a machine learning architecture. For example, KNN, neural networks, SVM, Bayesian classifiers, and principal component analysis (PCA), represented by boxes such as "A", "B", and "C", are examples of models (ellipsis in the figure indicates the existence of additional items). A model can specify general computational relationships, such as an SVM containing a hyperplane or a neural network having layers or neurons, but it may not specify specific architectural structures, such as the selection of hyperparameters or dataflow for the architecture to perform a particular task, such as an SVM employing a radial basis function (RBF) kernel or a neural network being configured to accept a 256x256x3 dimensional input. These structural features may be selected by the practitioner or obtained as a result of the training or configuration process. Note that the universe of model 220a includes combinations of members, such as when creating an ensemble model (described later in relation to Figure 3G) or when using a model pipeline (described later in relation to Figure 3H).

[0082] To clarify, it's important to understand that many architectures include both parameters and hyperparameters. Architecture parameters refer to the configuration values ​​of the architecture, which can be adjusted directly based on the reception of input data (e.g., adjusting the weights and biases of a neural network during training). Different architectures may have different parameter choices and relationships between parameters, but changing parameter values ​​during training, for example, is not considered a change in architecture. In contrast, architecture hyperparameters refer to configuration values ​​of the architecture that are not adjusted directly based on the reception of input data (e.g., the number of K neighbors in a KNN implementation, the learning rate in a neural network training implementation, the kernel type in an SVM). Therefore, changing hyperparameters usually changes the architecture. The operation of some methods described below, such as validation, will show that it may be possible to adjust hyperparameters during training, and consequently adjust the architecture type. As a result, some implementations may consider multiple architectures, but only some of them may be configured to be used, or used at specific moments.

[0083] Similar to models and architectures, at a higher level, method 220d can be considered a species of methodology 220e within its genus (methodology I has methods I.1, I.2, etc.; method II has methods II.1, II.2, etc.). Methodology 220e refers to algorithms that are well-suited for adaptation as a way to perform tasks using one or more specific machine learning architectures, such as training an architecture, testing an architecture, validating an architecture, performing inference with an architecture, or using multiple architectures in a generative adversarial network (GAN). For example, gradient descent is a methodology that describes how to train a neural network, and ensemble learning is a methodology that describes how to train a group of architectures, etc. A methodology can specify general algorithmic operations, such as gradient descent performing iterative steps along a cost or error plane, or ensemble learning examining intermediate results of its architecture. However, a methodology can also specify how a particular architecture executes the methodology's algorithm, for example, gradient descent employing iterative backpropagation and stochastic optimization via Adam using specific hyperparameters in a neural network, ensemble systems consisting of a collection of random forests applying AdaBoost with specific configuration values, or training data organized into a specific number of folds. It will be understood that architectures and methods themselves may have sub-architectures and sub-methods, such as when adding or modifying existing architectures or methods (for example, a GAN architecture and GAN training method can be considered to include a deep learning architecture and deep learning training method). It will also be understood that not all possible methodologies apply to all possible models (for example, suggesting performing gradient descent on a PCA architecture without further explanation would seem meaningless). Methods may involve several actions by the practitioner, or they may be fully automated.

[0084] As demonstrated in the example above, when moving from models to architectures and from methodologies to methods, some methods may apply only to specific architectures, and certain architectures may apply only to specific methods; therefore, aspects of architectures may appear in methods, and aspects of methods may appear in architectures. Understanding this interaction, implementation 220c is a combination of one or more architectures and one or more methods to form a machine learning system configured to perform one or more specified tasks, such as training, inference, and generating new data with a GAN. To clarify, the architecture of an implementation does not need to actively perform its methods; it can simply be configured to perform methods (for example, if accompanying training control software is configured to pass input through the architecture). Applying a method performs a task such as training or inference. Thus, the virtual implementation A shown in Figure 2F (indicated as "Imp. A") contains a single architecture with a single method. This could correspond, for example, to an SVM architecture configured to recognize objects in a 128x128 grayscale pixel image using a hyperplanar support vector separation method employing an RBF kernel in a 16,384-dimensional space. The use of RBF kernels and the selection of feature vector input structures reflect aspects of both architecture selection and training and inference method selection. Therefore, it will be found that some descriptions of architectural structures may imply aspects of the corresponding methods, and vice versa. For example, virtual implementation B (indicated as "Imp.B") may correspond to training method II.1, which may switch between architectures B1 and C1 based on validation results before inference method III.3 is applied.

[0085] The close relationships between architectures and methods within an implementation contribute to much of the ambiguity in Figure 2A, as the groups do not readily capture the close relationships between methods and architectures within a given implementation. For example, very small changes in method or architecture may cause a practitioner to move model implementations between the groups in Figure 2A, such as training a random forest with a first method (supervised) incorporating labels and then applying a second method with the trained architecture to detect clusters in unlabeled data (unsupervised) rather than performing inference on the data. Similarly, the groups in Figure 2A may make it difficult to classify aggregation methods and architectures, as will be discussed later in relation to Figures 3F and 3G, for example, as techniques found in some, none, or all of the groups in Figure 2A may be applied. Therefore, the following sections will refer to Figures 3A-G and 4A-J to describe the relationships between various exemplary model architectures and exemplary methods in order to clarify the relationships between architectures, methods, and implementations and to facilitate the reader's understanding. The tasks being discussed are illustrative, and therefore, referring to, for example, classification operations to facilitate understanding, should not be interpreted as suggesting that the implementation must be used exclusively for that purpose.

[0086] To clarify, the above explanation regarding Figure 2F is provided solely for the reader's convenience and should therefore not be interpreted restrictively without explicit wording indicating it. For example, while “Method” 220d is obviously a computer-implemented method, not all computer-implemented methods are methods in the sense of “Method” 220d. A computer-implemented method may be logic without machine learning capabilities. Similarly, the term “methodology” is not always used in the sense of “methodology” 220e and may refer to an approach without machine learning capabilities. Likewise, the terms “model,” “architecture,” and “implementation” are used in 220a, 220b, and 220c above, but these terms are not limited to their distinction in Figure 2F and may be used to refer to the general topology of machine learning components where no such language exists.

[0087] Basic concepts of machine learning - illustrative implementations Figure 3A is a schematic diagram of the operation of an exemplary SVM machine learning model architecture. At a high level, given data from two classes (e.g., images of dogs, images of cats) as input features represented by circles and triangles in the schematic diagram of Figure 3A, the SVM attempts to determine a hyperplanar separator 305a that maximizes the minimum distance from members of each class to the separator 305a. Here, the training feature vector 305f has the minimum distance 305e from all peers to the separator 305a. Conversely, the training feature vector 305g has the minimum distance 305h from all peers to the separator 305a. The margin 305d formed between these two training feature vectors is therefore a combination of distances 305h and 305e (reference lines 305b and 305c are provided for clarity) and is a maximum-minimum separation, thus identifying the training feature vectors 305f and 305g as support vectors. This example demonstrates linear hyperplane separation, but different SVM architectures correspond to different kernels (e.g., RBF kernels) and may facilitate nonlinear hyperplane separation. The separator may be found during training, and subsequent inference may be achieved by considering where new inputs in the feature space are located relative to the separator. Similarly, this example shows two-dimensional feature vectors (in a two-dimensional plane of paper) for clarity, but you will see that the architecture may accept features of many more dimensions (e.g., a 128x128 pixel image can be input as 16,384 dimensions). While the hyperplane in this example only separates two classes, multi-class separation can be achieved in various ways, such as using an ensemble architecture of SVM hyperplane separations in configurations like one-to-one, one-to-all, etc. Practitioners often use the LIBSVM™ and scikit-learn™ libraries when implementing SVMs. Understand that many different machine learning models, such as logistic regression classifiers, attempt to identify separating hyperplanes.

[0088] In the exemplary SVM implementation described above, the practitioner determined the feature format as part of the implementation's architecture and methodology. For some tasks, an architecture and methodology that processes the input to determine a new or different feature format itself may be desirable. Some random forest implementations may substantially tune the feature space representation in this manner. For example, Figure 3B shows an exemplary random forest model architecture at a high level, including multiple decision trees 310b, each of which may receive all or part of the input feature vector 310a at its root node. While this exemplary architecture shows three trees with a maximum depth of three levels, it can be seen that forest architectures with different levels (even between trees in the same forest) are possible with a small or large number of trees. When each tree considers that part of the input, it refers all or part of the input to subsequent nodes, e.g., path 310f, based on whether the input part satisfies or fails to satisfy conditions relevant to various nodes. For example, when considering an image, a single node in the tree may query whether the pixel value at a position in the feature vector is above or below a certain threshold. In addition to threshold parameters, some trees may include additional parameters, and their leaves may include the probability of correct classification. Each leaf of a tree can be associated with a provisional output value 310c for consideration by a voting mechanism 310d that generates the final output 310e, for example, by taking a majority vote across trees or by a probability-weighted average of the predictions of each tree. This architecture may be suitable for various training methods, for example, so that different data subsets are trained on different trees.

[0089] The depth of the trees within a random forest, as with different trees, can facilitate the exploration of feature relationships by the random forest model beyond a direct comparison of those relationships in the initial input. For example, if the original features were pixel values, the tree might recognize relationships between task-relevant groups of pixel values, such as the relationship between "nose" and "ear" pixels for cat / dog classification. However, the relationships in a binary decision tree may impose limitations on its ability to identify these "higher-order" features.

[0090] As in the exemplary architecture in Figure 3C, a neural network can also infer higher-order features and relationships between initial input vectors. However, each node in the network can be associated with various parameters and connections to other nodes, facilitating more complex decisions and intermediate feature generation than the binary relationships of a conventional random forest tree. As shown in Figure 3C, a neural network architecture can include an input layer, at least one hidden layer, and an output layer. Each layer contains a collection of neurons that can receive several inputs and provide an output value, also called an activation value, which is the output value of the final output layer 315b, serving as the final result of the network. Similarly, the input 315a of the input layer can be received from the input data rather than from a previous neuron layer.

[0091] Figure 3D shows the relationship between the input and output at node 315c in Figure 3C. Specifically, the output n of node 315c out This can relate to its three (zero-based indexed) inputs as follows:

number

[0092] You will notice that many of the exemplary machine learning implementations discussed so far in this overview are “discriminative” machine learning models and methodologies (such as SVMs, logistic regression classifiers, and neural networks with nodes like those in Figure 3D). Generally, discriminative approaches assume a form that seeks to find the following probability in Equation 2:

number

number

[0093] It will be understood that such generative approaches are applied here to achieve the results presented in discriminative implementations, and vice versa. For example, Figure 3E shows an exemplary node 315d that may appear in a Bayesian neural network. Unlike node 315c, which simply accepts numerical values, a node in a Bayesian neural network like node 315d may accept weighted probability distributions 315f, 315g, 315h (e.g., parameters of such distributions) and may itself output distribution 315e. Thus, while the classification uncertainty of a discriminative model can be determined, for example, through various post-processing techniques (e.g., comparing the output to an iterative application of dropout to a discriminative neural network), it will be found that a similar measure of uncertainty can be achieved, for example, by considering the variance of distribution 315e, by employing a generative model that outputs a probability distribution. Therefore, just as referring to one particular machine learning implementation here is not intended to exclude substitution by an implementation that functions similarly, referring to a discriminative implementation here is not meant to exclude substitution by a generative counterpart where applicable, and vice versa.

[0094] Returning to the general discussion of machine learning approaches, Figure 3C shows an exemplary neural network architecture with a single hidden layer, but many neural network architectures can have more than one hidden layer. Some networks with many hidden layers have produced surprisingly effective results, and the term "deep learning" has been applied to these models to reflect the many hidden layers. Here, deep learning refers to architectures and methods that employ at least one neural network architecture with more than one hidden layer.

[0095] Figure 3F is a schematic diagram of the operation of an exemplary deep learning model architecture. In this example, the architecture is configured to receive a two-dimensional input 320a, such as a grayscale image of a cat. When used for classification as in this example, the architecture can generally be divided into two parts: a feature extraction part consisting of a series of layer operations, and a classification part that determines output values ​​based on the relationships between the extracted features.

[0096] Many different feature extraction layers are possible, such as convolutional layers, max pooling layers, dropout layers, and cropping layers, and many of these layers are themselves susceptible to variability. For example, 2D convolutional layers, 3D convolutional layers, and convolutional layers with different activation functions are different methods and methodologies for training and inference of networks. As illustrated, these layers can generate multiple intermediate values ​​320b-j of different dimensions, and these intermediate values ​​can be processed along multiple passes. For example, the original grayscale image 320a can be represented as a feature input tensor of dimension 128×128×1 (e.g., a grayscale image with width 128 pixels and height 128 pixels) or as a feature input tensor of dimension 128×128×3 (e.g., an RGB image with width 128 pixels and height 128 pixels). Multiple convolutions with different kernel functions in the first layer can precipitate multiple intermediate values ​​320b from this input. These intermediate values ​​320b could themselves be examined by two different layers to form two new intermediate values ​​320c and 320d along separate paths (two paths are shown in this example, but more paths or even a single path are possible in different architectures). Furthermore, data can also be provided in multiple “channels,” such as when an image has red, green, and blue values ​​for each pixel, resulting in, for example, a 128x128x3 feature tensor with “x3” dimensions (for simplicity, this input has three “tensor” dimensions but 49,152 individual “feature” dimensions). Different architectures may operate on channels individually or together in various layers. The ellipsis in the diagram indicates the presence of additional layers (e.g., some networks have hundreds of layers). As illustrated, intermediate values ​​can change in size and dimensions, for example, following a pool, as in value 320e. In some networks, intermediate values ​​may be considered at the inter-path layer, as shown between the intermediate values ​​of 320e, 320f, 320g, and 320h.Ultimately, the final set of feature values ​​appears in intermediate collections 320i and 320j, which are fed into collections of one or more classification layers 320k and 320l via, for example, flattened layers, softmax layers, fully connected layers, etc., and the output nodes of layer 320l generate output values ​​320m. For example, if N classes are recognized, there may be N output nodes to reflect the probability that each class is the correct class (e.g., here the network identifies one of three classes and indicates class "cat" as the most likely for a given input), although some architectures may have fewer or more outputs. Similarly, some architectures may accept additional inputs (e.g., some flood-fill architectures utilize evolving mask structures that may be received as input in addition to the input feature data and produced as output in a modified form in addition to the classification output values; similarly, some recurrent neural networks may store values ​​that are input from one iteration to subsequent iterations in parallel with other inputs) or include feedback loops, etc.

[0097] TensorFlow®, Caffe®, and Torch® are examples of common software library frameworks for implementing deep neural networks, but many architectures can be built "from scratch," simply representing layers as operations on matrices or tensors of values, and data as values ​​within such matrices or tensors. Examples of deep learning network architectures include VGG-19, ResNet, Inception, and DenseNet.

[0098] On the other hand, exemplary paradigm machine learning architectures are discussed with respect to Figures 3A to 3F, and there are many machine learning models and corresponding architectures formed by combining, modifying, or adding operations and structures to other architectures and methods. For example, Figure 3G is a schematic diagram of an ensemble machine learning architecture. Ensemble models include a wide variety of architectures, such as "meta-algorithmic" models that use multiple weak learning models to form a collectively stronger model, like AdaBoost. The random forest in Figure 3A can be seen as another example of such an ensemble model, although the random forest itself may be an intermediate classifier of the ensemble model.

[0099] In the example in Figure 3G, the initial input feature vector 325a may be input, in whole or in part, to various model implementations 325b, which may be from the same model or different models (e.g., SVM, neural network, random forest, etc.). The outputs from these models 325b can then be received by a "fused" model architecture 325d to produce the final output 325e. The fused model implementation 325d itself may be the same as or different from one of the implementations 325b. For example, in some systems, the fused model implementation 325d may be a logistic regression classifier and model 325b may be a neural network.

[0100] While it may be understood that ensemble model architectures may facilitate greater flexibility than the paradigm architectures in Figures 3A–3F, it should be recognized that modifications to the architecture or its methods, sometimes relatively minor, can facilitate new behaviors that are not readily suitable for the conventional grouping in Figure 2A. For example, PCA is generally described as an unsupervised learning method and corresponding architecture for identifying dimensionality-reduced feature representations of unlabeled input data. However, PCA has often been used with labeled inputs to facilitate classification in supervised methods, as in the EigenFaces application described in M. Turk and A. Pentland, "Eigenfaces for Recognition," J. Cognitive Neuroscience, vol.3, no.1, 1991. Figure 3H shows an exemplary machine learning pipeline topology of such a modification. Similar to EigenFaces, feature representations can be determined using an unsupervised method in block 330a (e.g., using PCA to determine principal components for each group of face images associated with one of several individuals). As an unsupervised method, the conventional grouping shown in Figure 2A may not typically be interpreted as "training" this PCA operation. However, by transforming the input data (e.g., facial images) into a new representation (principal component feature space) in block 330b, a data structure suitable for subsequent inference methods can be created.

[0101] For example, in block 330c, the new input feature vector (new face image) is transformed into an unsupervised form (e.g., principal component feature space), and then in block 330d, a metric (e.g., the distance between the principal components of each individual's face image group and the principal component representation of the new vector) or another subsequent classifier (e.g., SVM) can be applied to classify the new input. Thus, a model architecture (e.g., PCA) that does not follow a particular methodology (e.g., metric-based training and inference) can be made to be accepted in that way through modifications of the method or architecture, such as pipeline processing. Here again, it will be seen that this pipeline is just one example—the unsupervised architecture and method of KNN in Figure 2B can also be used for supervised classification by assigning the new inference input to a class of groups that have the closest first moment in the feature space to the inference input. Thus, these pipeline processing approaches can be considered here as machine learning models, although they may not be conventionally called so.

[0102] Some architectures may be used with training methods, and some of these trained architectures may then be used with inference methods. However, it will be understood that not all inference methods perform classification, and not all trained models are used for inference. Similarly, it will be understood that not all inference methods require the training method to be applied to the architecture beforehand to process new inputs for a given task (for example, as KNN directly examines the input data and generates classes). Regarding training methods, Figure 4A is a schematic flowchart showing the general operation of various training methods. Specifically, in block 405a, the practitioner or the architecture can assemble the training data into one or more training input feature vectors. For example, a user may collect images of dogs and cats with metadata labels for a supervised learning method, or, in the case of unsupervised clustering, unlabeled stock prices over time. As mentioned above, raw data can be converted into feature vectors through preprocessing, or it can be directly acquired as features in its raw form.

[0103] In block 405b, the training method allows for tuning the architecture's parameters based on the training data. For example, the weights and biases of a neural network can be updated via backpropagation, and an SVM can select support vectors based on hyperplane calculations. However, not all model architectures can update parameters within the architecture itself during "training," as discussed with respect to the pipeline architecture in Figure 3G. For example, in Eigenfaces, the determination of the principal components of a face ID group may be interpreted not as tuning existing parameters (e.g., tuning the weights and biases of the neural network architecture), but as creating new parameters (principal component feature space). Thus, here, the Eigenfaces determination of principal components from training images is still interpreted as a training method.

[0104] Figure 4B is a schematic flowchart illustrating various operations common to different machine learning model inference methods. As mentioned earlier, not all architectures or methods include inference capabilities. Where an inference method is applicable, in block 410a, the practitioner or architecture can assemble raw inference data, e.g., new images to be classified, into inference input feature vectors, tensors, etc. (e.g., in the same feature input format as the training data). In block 410b, the system can apply the trained architecture to the input inference feature vectors to determine outputs, e.g., classification, regression results, etc.

[0105] When "training," several methods and some architectures allow the input training feature data to be examined as a whole, either in a single pass or iteratively. For example, PCA decomposition can be implemented as a non-iterative matrix operation in some implementations. SVM can be trained with a single iteration through the input, depending on its implementation. Finally, some neural network implementations can be trained with multiple iterations on the input vector during gradient descent.

[0106] Regarding iterative training methods, Figure 4C is a schematic flowchart illustrating the iterative training behavior that may occur in block 405b for several architectures and methods, for example. A single iteration may apply the method in the flowchart once, while implementations that perform multiple iterations may apply the method in the diagram multiple times. In block 415a, the architecture parameters may be initialized to their default values. For example, in some neural networks, the weights and biases may be initialized to random values. In some SVM architectures, for example, the behavior in block 415a may not be applied. Since each of the training input feature vectors is considered in block 415b, the system may update the model parameters in 415c. For example, in an SVM training method, a new input feature vector is considered, and it is determined whether or not it affects the selection of support vectors, so a new hyperplane may or may not be selected. Similarly, a neural network method may update weights and biases according to backpropagation and gradient descent, for example. If all input feature vectors are considered, the model may be considered "trained" if the training method requires only one iteration. The method requiring multiple iterations applies the behavior of Figure 4C again (naturally avoiding initialization in block 415a again and prioritizing the parameter values ​​determined in the previous iteration), and training can be completed when a condition is met, for example, when the error rate between predicted labels and metadata labels decreases to below a threshold.

[0107] As mentioned above, the diverse range of machine learning architectures and methods includes those with explicit training and inference steps, as shown in Figure 4E, and those without, as generalized in Figure 4D. Figure 4E shows, for example, a method 425a to train a neural network architecture to recognize a newly received image in inference 425b, while Figure 4D shows, for example, an implementation 420b that reduces data dimensionality via PCA or performs KNN clustering, with implementation 420b taking input 420a and producing output 420c. To clarify, it will be understood that some implementations may take data input and produce an output (e.g., an SVM architecture with an inference method), some implementations may only take data input (e.g., an SVM architecture with a training method), and some implementations may only produce an output without taking data input (e.g., a trained GAN architecture with a random generator method to generate new data instances).

[0108] The operations in Figures 4D and 4E can be further extended in several ways. For example, some methods extend the training as shown in the schematic block diagram of Figure 4F, and the training methods further include various data subset operations. As shown in Figure 4G, some training methods can split the training data into a training data subset 435a, a validation data subset 435b, and a test data subset 435c. As shown in Figure 4F, when training the network in block 430a, the training method can first iteratively tune the network parameters using, for example, backpropagation based on all or part of the training data subset 435a. However, in block 430b, the effectiveness of the training can be evaluated using the subset of data reserved for validation 435b. Not all training methods and architectures are guaranteed to find the optimal architecture parameters or configuration for a given task; for example, they may get stuck at a local minimum and may employ inefficient learning step size hyperparameters. The method can predict such defects and validate the current hyperparameter configuration in block 430b using training data 435b, which is different from training data subset 435a, and adjust the architecture's hyperparameters or parameters accordingly. In some methods, the method iterates between training and validation, such as continuing to train the rest of training data subset 435a with validation feedback, restarting training on all or part of training data subset 435a, or adjusting the architecture's hyperparameters or architecture topology (if additional hidden layers are added to the neural network in meta-learning), as indicated by arrow 430f. Once the architecture is trained, the method can evaluate the effectiveness of the architecture by applying it to all or part of test data subset 435c.Using different data subsets for validation and testing can also help avoid overfitting, where the training method brings the architecture's parameters too close to the training data, mitigating the more optimal generalization when the architecture encounters a new inference input. If the test results are undesirable, the method may start training again with a different parameter configuration, a different hyperparameter configuration, and so on, as indicated by arrow 430e. The test in block 430c may be used to confirm the validity of the trained architecture. Once the model is trained, inference 430d may be performed on a newly received inference input. The existence of variations of this validation method is understandable, for example, when the method performs a grid search of the space of possible hyperparameters to determine the best architecture for the task.

[0109] Many architectures and methods may be modified to integrate with other architectures and methods. For example, some architectures that have been successfully trained for a certain task may be trained more effectively on a similar task than starting with, for example, randomly initialized parameters. Methods and architectures that adopt parameters from a first architecture in a second architecture (which may in some cases be the same architecture) are called “transfer learning” methods and architectures. Given a pre-trained architecture 440a (e.g., a deep learning architecture trained to recognize birds in images), a transfer learning method may perform additional training using data from a new task domain so that inference 440e can be performed on a new task domain (e.g., providing labeled data of car images to recognize cars in images). Transfer learning training methods may or may not distinguish between training 440b, validation 440c, and test 440d, as well as iterative operations 440f and 440g, as described above. It can be seen that the pre-trained model 440a can be received as the entire trained architecture, or as a list of trained parameter values ​​applied to, for example, parallel instances of the same or similar architecture. In some transfer learning applications, some parameters of the pre-trained architecture may be "frozen" to prevent adjustments during training, while other parameters are allowed to change during training with data from the new domain. This approach allows the architecture to be adapted to the new domain while retaining the general advantages of the architecture's original training.

[0110] The combination of architecture and method can also be extended over time. For example, an “online learning” method anticipates the application of an initial training method 445a to the architecture, subsequent application of an inference method using that trained architecture 445b, and periodic updates 445c by another training method 445d, possibly the same method as method 445a, but usually applied to new training data inputs. Online learning methods can be useful, for example, when a robot is deployed in a remote environment according to an initial training method 445a and encounters additional data that may improve the application of the inference method in 445b. For example, if several robots are deployed in this way, when one robot encounters a “true positive” recognition (e.g., a new core sample with a classification validated by a geologist; a new patient characteristic during surgery validated by a surgeon), the robot can send that data and result to its peer robots for use in method 445d as new training data input. The neural network can then perform backpropagation adjustments using the true positive data in training method 445d. Similarly, SVM can consider whether new data in training method 445d influences the selection of support vectors and settles the adjustment of its hyperplane. Online learning is often part of reinforcement learning, but it can also appear in other methods such as classification, regression, and clustering. The initial training method may or may not include the training 445e, validation 445f, and test 445g submethods in training method 445a, as well as the iterative adjustments 445k and 445l. Similarly, online training may or may not include the training 445h, validation 445i, and test submethods, 445j, and the iterative adjustments 445m and 445n, and if included, they may differ from the submethods 445e, 445f, 445g, and the iterative adjustments 445k and 445l. In fact, the subsets and proportions of training data allocated to validation and testing may differ between each training method 445a and 445d.

[0111] As mentioned earlier, many machine learning architectures and methods do not need to be used exclusively for a single task, such as training, clustering, or inference. Figure 4J shows one such exemplary GAN architecture and method. In a GAN architecture, the generator subarchitecture 450b may interact competitively with the discriminator subarchitecture 450e. For example, the generator subarchitecture 450b is trained to generate synthetic "fake" challenges 450c, such as synthetic portraits of non-existent individuals, while in parallel, the discriminator subarchitecture 450e is trained to distinguish the "fake" challenges from real, true positive data 450d, such as true portraits of real people. Such a method can be used, for example, to generate synthetic assets that resemble real-world data for use as additional training data. Initially, the generator subarchitecture 450b can be initialized with random data 450a and parameter values ​​to induce a highly unconvincing challenge 450c. The discriminator subarchitecture 450e can initially be trained with true positive data 450d, and therefore may initially easily distinguish the fake challenge 450c. However, in each training cycle, the generator loss 450g can be used to improve the generator subarchitecture's 450b training, and the discriminator loss 450f can be used to improve the discriminator subarchitecture's 450e training. Such competitive training may eventually generate a synthetic challenge 450c that is very difficult to distinguish from the true positive data 450d. To clarify, in the context of GANs, "adversarial" networks refer to the generator and discriminator competition described above, while "adversarial" inputs refer to inputs that are specifically designed in an implementation to produce a particular output, and in some cases an output that the implementation's designers did not intend.

[0112] Data Overview Figure 5A is a schematic diagram of surgical data that may be received by the processing system in some embodiments. Specifically, the processing system may receive raw data 510, such as video from a visualization tool 110b or 140d, which constitutes a sequence of individual frames over time 505. In some embodiments, the raw data 510 may include video and system data from multiple surgical procedures 510a, 510b, 510c, or from a single surgical procedure only.

[0113] As mentioned above, each surgical procedure can include a group of actions, each group forming a separate unit called a task. For example, surgical procedure 510b may include tasks 515a, 515b, 515c, and 515e (the ellipsis 515d indicates the possibility of more intervening tasks). Note that some tasks may be repeated in surgery and their order may be changed. For example, task 515a involves locating the fascial segment, task 515b involves dissecting the first portion of the fascia, task 515c involves dissecting the second portion of the fascia, and task 515e involves cleaning and cauterizing the area of ​​fascia before closure.

[0114] Each of the tasks 515 may be associated with a set of corresponding frames 520a, 520b, 520c, and 520d, and a device dataset including operator kinematic data 525a, 525b, 525c, 525d, patient-side device data 530a, 530b, 530c, 530d, and system event data 535a, 535b, 535c, 535d. For example, in the case of video acquired from a visualization tool 140d in the operating room 100b, the operator-side kinematic data 525 may include translational and rotational values ​​of one or more handheld input mechanisms 160b at the surgeon console 155. Similarly, the patient-side kinematic data 530 may include data from a patient-side cart 130, data from sensors placed on one or more tools 140a-d, 110a, rotational and translational data from arms 135a, 135b, 135c, and 135d, etc. System event data 535 may include data on discrete parameters such as the operation of one or more pedals 160c, tool operation, system alarm activation, energy application, button presses, and camera movement. Depending on the situation, task data may include one or more of the following, rather than all four: frameset 520, operator-side kinematics 525, patient-side kinematics 530, and system events 535.

[0115] For clarity and ease of understanding, kinematic data are presented here as waveforms and system data as continuous state vectors. However, it should be understood that some kinematic data may be discrete values ​​over time (for example, an encoder measuring the position of a continuous component may be sampled at fixed intervals), while some system values ​​may be continuous values ​​over time (for example, values ​​may be interpolated, such as when a parametric function can be fitted to individually sampled values ​​from a temperature sensor).

[0116] Furthermore, for ease of understanding, surgeries 510a, 510b, and 510c are shown here as being immediately adjacent to tasks 515a, 515b, and 515c, but you will understand that in actual surgical videos, there may be gaps between surgeries and tasks. Therefore, some videos and data may be related to tasks that are not related to the task or are not the subject of the current analysis. In some embodiments, these “non-task” / “irrelevant task” regions of the data may be shown as tasks in the annotations themselves, for example, in a “gap” task, the “real” task does not occur.

[0117] A discrete set of frames associated with a task can be determined by the task's start and end points. Each start and end point can be determined by either a tool action or a state change of a body affected by the tool. Thus, data captured between these two events can be associated with the task. For example, the start and end point actions of task 515b may occur with timestamps associated with locations 550a and 550b, respectively.

[0118] Figure 5B is a table showing examples of tasks with corresponding start and end points that may be used in combination with various disclosed embodiments. Specifically, the data associated with the task "Mobilize Colon" is the data acquired between the time the tool first interacts with the colon or surrounding tissue and the time the tool last interacts with the colon or surrounding tissue. Thus, any of the framesets 520, operator-side kinematics 525, patient-side kinematics 530, and system events 535 with timestamps between these start and end points are data associated with the task "Mobilize Colon". Similarly, the data associated with the task "Endopelvic Fascia Dissection" is the data acquired between the time the tool first interacts with the pelvic fascia (EPF) and the timestamp of the last interaction with the EPF after the prostate has been degreased and separated. The data associated with the task "Apical Dissection" corresponds to the data acquired between the time the tool first interacts with the prostate tissue and the time the task ends when the prostate is released from all attachments to the patient's body. It will be understood that task start and end times may be chosen to allow for temporal overlap between tasks, or to avoid such overlap. For example, in some embodiments, tasks may be "paused," such as when a surgeon engaged in a first task moves on to a second task before completing the first, and then returns to and completes the first task after completing the second. Thus, it will be understood that while start and end points can define the boundaries of tasks, the data can be annotated to reflect timestamps related to multiple tasks.

[0119] Examples of additional tasks include completing four horizontal interruption sutures using two-handed technique (i.e., the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle exits the tissue with only two hands, e.g., when no one-handed suturing motion occurs in between). The "Uterine Horns" task includes incising the broad ligament from the left and right uterine horns and dissecting the uterine body (here, you will understand that some tasks have multiple conditions or events that determine the start or end time, such as when the incision tool touches either the uterine horn or the uterine body and the task ends when both the uterine horn and body are separated from the patient). The "One-Handed Suturing" task includes completing four vertical interruption sutures using one-handed technique (i.e., the start time is when the suture needle first penetrates the tissue, and the stop time is when the suture needle exits the tissue with only one hand, e.g., when no two-handed suturing motion occurs in between). The task "Suspensory Ligaments" involves incising the lateral lobules of each suspensory ligament to expose the ureter (i.e., the start time is when the incision of the first lobe is started, and the end time is when the incision of the last lobe is completed). The task "Running Suture" involves performing a running suture with four bits (i.e., the start time is when the suture needle first penetrates the tissue, and the stop time is when the needle exits the tissue after all four bits are completed). As a final example, the task "Rectal Artery / Vein" involves incising and ligating the upper rectal artery and vein (i.e., the start time is when the incision is started in either the artery or the vein, and the end time is when the surgeon ceases contact with the ligation after it has been performed).

[0120] Example of an operator evaluation process topology A surgeon's technical skills are a crucial element in providing optimal patient care. Unfortunately, many existing methods for assessing operator skills remain subjective, qualitative, or resource-intensive. The various embodiments disclosed herein aim to provide a more effective assessment of surgical skills by analyzing operator skills using objective performance indicators (OPIs), quantitative metrics generated from surgical data, which are suitable for examining an operator's individual skill performance, task-level performance, and overall surgical performance. It will be understood that OPIs may be generated not only from data values ​​but also from other OPIs (for example, the ratio of two OPIs can be considered an OPI). Skills are actions or groups of actions performed during surgery that are perceived to influence the efficiency or outcome of the surgery. Initially, for the purposes of automated operation, skills are “defined” or represented by an initial assignment of OPIs (e.g., proposed by a specialist), but such initial assignments may be adjusted using the various systems and methods described herein.

[0121] Figure 6 is a schematic topology diagram showing the information flow for performing surgical skill assessments that may occur in several embodiments. Specifically, the “reference” data 605a may be data obtained from a real-world non-robot operating room 635a, a real-world robotic operating room 635b, and a simulated surgery 635c (although a robotic simulator is shown, it will be understood that non-robot surgeries can also be simulated, for example, using appropriate dummy patient materials). The reference dataset 605a may include data from both “experienced” users (e.g., operators with more than 100 hours of experience performing the skill or task) and “novice” users (e.g., operators with less than 100 hours of experience performing the skill or task). The reference dataset 605a may be used as part of the performance assessment system 625 to train a machine learning model classifier (e.g., one or more skill or task models described here).

[0122] Later, the “target” dataset 605b can be obtained and may also include data provided by a real-world non-robot operating room 640a, a real-world robotic operating room 640b, and a simulated surgery 640c (again, a robotic simulator is shown, but it will be understood that non-robot surgeries may also be simulated, for example, using appropriate dummy patient materials). The target dataset 605b can also be provided to a classifier of a performance evaluation system 625 trained on the “reference” data 605a to generate performance metrics 630 (e.g., skill score, task score, etc.) of the target dataset 605b. Selecting the “capture case” button 640 may have the same effect as selecting the submit button 635 in some embodiments.

[0123] In some embodiments, the reference dataset 605a and the target dataset 605b may be stored in data storage 610a and 610b, respectively, before being consumed by the performance evaluation system 625. In some embodiments, data storage 610a and 610b may be the same data storage. In some embodiments, data storage 610a and 610b may be offsite from where the data was acquired, for example, on a cloud-based network server. Processing systems 615a and 615b may process the data stored in data storage 610a and 610b (e.g., recognizing individual surgeries captured in the data stream, separating surgeries recognized in the stream into separate datasets, providing metadata annotations for datasets, or simply ensuring appropriate data storage without additional action). In some embodiments, human annotators may assist, modify, or verify the results of processing systems 615a and 615b. In some embodiments, processing systems 615a and 615b may be the same processing system.

[0124] Subsequently, the processed reference data 620a and target data 620b in the data storage 610a and 610b may be used by the performance evaluation system 625 to determine the performance metrics 630, specifically, the processed data 620a may be used to train the classifier of the performance evaluation system 625, and then the classifier may be used to generate the performance metrics 630 of the processed data 620b.

[0125] Example of a process overview Figure 7A is a flowchart illustrating various operations in process 700 for generating and applying a skill (or task) model, which may be performed in several embodiments. Generally, the operations include either training the skill model with annotated training data 705a, or inference with new, unannotated data using such a model 705b. Specifically, in block 705c, the processing system (e.g., performance evaluation system 625) may receive raw data, e.g., visualization tool data 520, operator-side kinematic data 525, patient-side kinematic data 530, and system event data 525 appearing in dataset 605a, although, as mentioned above, in some embodiments, less data than all of these types may be available. Since this data is used to train a skill (or task) model used to determine a surgeon's skill score, the data may be annotated to indicate whether the data was generated by a “professional” or “non-professional” surgeon. The training data may contain asymmetry, such as having more non-professional data values ​​than professional data values, as described here. As a result, resampling methods such as the Synthetic Minority Oversampling Technique (SMOTE) (e.g., using the imblearn(trademark) library function imblearn.over_sampling.SMOTE) may be applied to the raw training data in block 705c or the metrics generated in block 705d. You will also understand that variations of SMOTE, such as SMOTE with edited nearest neighbor cleaning (SMOTEENN) and SMOTE using Tomek links (SMOTETomek), may also be employed.

[0126] In block 705d, as will be explained in more detail here, the system can convert raw data into metrics, such as OPI. Simply put, all metrics could be used when evaluating all conceivable surgeon skills, and indeed, several embodiments can take this approach. However, in block 705e, various embodiments select a specific type of metric for each model to use when evaluating the corresponding skill (or task). Not only does this reduce future computational overhead, but it can also fine-tune the models and ensure that the models operate with feature vectors better suited to each skill. In block 705f, the system can train each skill or task model using the metric feature vectors generated in block 705d, which were selected for each skill or task model in block 705e. For simplicity, a skill model is a machine learning model trained to distinguish between expert and non-expert OPI values ​​associated with a skill, and a task model is a machine learning model trained to distinguish between expert and non-expert OPI values ​​associated with a task (although, as discussed here, for example with respect to Figure 13A, the score results from the skill model can also be used to infer the score results for the task). Since both the task model and the skill model operate based on a collection of OPI data to generate scores, you will find that the descriptions here regarding OPI selection, training, and application for the skill model also apply to the task model (in some cases, only the skill model is described for clarity).

[0127] These trained models can then be used to evaluate the performance of other surgeons (for example, so that they are reflected in the target dataset 605b). Specifically, when the system receives additional raw data in block 705g (which, for inference purposes, is not annotated as being associated with either experts or non-experts), the system can iterate through such data, convert it into appropriate metrics for each skill model in block 705h, and generate skill scores in block 705i (in this example, separate tasks are associated with separate skills, but in some embodiments, the same skill may apply to the entire surgery).

[0128] Figure 7B is a schematic diagram showing various components used in application examples of a skill model for determining surgical scores, which may be performed in several embodiments. Specifically, given a skill model 710d trained according to blocks 705c, 705d, 705e, and 705f, the performance evaluation system 625 can perform the actions of blocks 705h and 705i. For example, given new raw data 710a which may include system data, patient-side or console-side kinematics, or video frame data, a transformation component 710b (e.g., software, hardware, or firmware logic) can transform the raw data into various metric values ​​710c. In this example, the OPI (e.g., selected for this particular skill model 710d in block 705e) is represented as an array of individual OPI values ​​associated with the timestamp value of each frame. Once these metrics are available, the system can select all or a subset of the OPI for consideration by the skill model 710d (in these embodiments, each skill is associated with its own model, but embodiments will understand that one model is trained to output multiple skills). Applying OPI values ​​to model 710d may generate model output value 710e. Since the model is trained to recognize “experts” and “non-experts” based on the OPI feature vector input, output 710e can have two values, for example, the probability derived from the input OPI 710c being generated by an expert or a non-expert from the data 710a. Here, for example, the result may indicate that there is a 55% probability that the data creator is an expert and a 45% probability that the creator is a non-expert. As will be discussed later, model 710d can be any suitable classifier that can generally distinguish between experts and non-experts for a given OPI data, such as a logistic regression classifier, SVM, neural network, or random forest. In some embodiments, model 710d may be configured to accept OPI values ​​for a single subset of available raw data (e.g., data associated with 1-second intervals) and thus can be repeatedly applied to the raw data.However, in some embodiments, Model 710d may be configured to receive all raw data 710a as a single input and generate an output pair 710e for each point in time in the data (for example, at each timestamp of a frame in the data).

[0129] As will be discussed later, the raw model probability output 710e is not necessarily directly suitable for determining the "score" of the surgeon who generated the data 710a with respect to the skill in question (i.e., the skill associated with model 710d). Therefore, the system may include a post-processed model 710f that can map the model output value 710e to the final score value 710g (for example, by associating the output with a score from a reference population, as described here with reference to Figures 12B and 12C).

[0130] From the discussion in Figure 7B, you will understand how the system is applied to generate scores for multiple sets of data (e.g., all frames of a surgical video), but for clarity, Figure 7C shows one “window” approach for generating such data. Specifically, the entire raw data can be organized into data “segments” 750a-e, which can correspond to raw data that fits within consecutive discrete time intervals. For example, data segment 750a might be system and kinematic data from the first 30 seconds of the recording, data segment 750b might be system and kinematic data acquired in the 30 seconds following segment 750a, and data segment 750c might similarly follow data segment 750b.

[0131] Model 710d may be configured to receive OPI values ​​generated from three consecutive segments of data. Thus, a three-segment "window" may be temporarily applied to the entire segments 750a-e. For example, three OPI values ​​750f may be generated using segments 750a, 750b, and 750c; three OPI values ​​750g may be generated using segments 750b, 750c, and 750d; and three OPI values ​​750h may be generated using segments 750c, 750d, and 750e. Each of the OPI values ​​750f, 750g, and 750h may act as a feature vector (i.e., OPI710c) fed into Model 710d to generate the corresponding predictive outputs 750i, 750j, and 750k (i.e., each instance of output 710e). Each of these outputs can be processed by the post-processing component 710f to produce final scores of 0.73, 0.72, and 0.75, respectively. You will understand the variations, such as the window size being adjusted as processing continues, if the window considers individual data points rather than data segments.

[0132] This method allows plotting score evaluations over time by associating the final score with the timestamp of the corresponding data segment. For example, a skill score of 0.73 could be associated with the timestamp of segment 750b, a skill score of 0.72 with the timestamp of segment 750c, and a skill score of 0.73 with the timestamp of segment 750d. In this example, a 3-segment window is used, generating three OPI values ​​based on these three segments, but you will soon realize that this is just one possible value chosen for ease of understanding. Shorter / longer segment windows or more / fewer segment windows can be used. For very short windows, the predicted outputs 750i, 750j, and 750k can be combined, for example, averaged, to facilitate inspection by a human reviewer. Conversely, for long windows, intermediate score values ​​can be generated by interpolation.

[0133] Such data can be organized into plots like the one shown in Figure 7D, where the score for the retraction skill in the "uterine horn" task is shown for each timestamp of the corresponding segment (you will notice that this can be organized by analogy to task scores over time from a task model, rather than skill scores from a skill model). Corresponding scores to timestamps in this method can facilitate correlation between specific times in surgery and score values, for example, skill scores between tasks 515a, 515b, 515c, and 515e. Corresponding plots like those in Figure 7D can be very useful when it is known that a task requires proficiency in this skill. For example, the portion of the plot corresponding to a task can be highlighted to the surgeon, providing feedback indicating whether their performance is "good" or "bad" compared to their colleagues. Of course, since a single task may require multiple skills, multiple plots like those in Figure 7D can be presented together to the surgeon. Such plots related to the time when a task occurred can also help contextualize score values ​​for human reviewers. For example, when a surgeon reviews a video of a surgery, such detailed results could allow the surgeon to jump to the good or bad points in the video to quickly identify the "highlights" of the performance, rather than reviewing the entire video.

[0134] Objective Performance Metrics - Application Overview For ease of understanding, Figure 8A is a schematic diagram illustrating the relationship between various metrics and data structures that may be used in several embodiments. Specifically, a surgical procedure 805a may consist of multiple tasks, e.g., tasks 805b, 805c, and 805d. Each task may itself represent several skills. For example, task 805c may depend on skills 805e, 805f, and 805g, respectively. In a similar manner, each skill itself may be evaluated based on one or more OPI metric values ​​(however, as mentioned above, in some embodiments, the OPI values ​​may be directly related to the task without the mediation of skills). For example, skill 805f may be evaluated by OPI metrics 805h, 805i, and 805j. Each OPI metric may be derived from one or more raw data fields. For example, OPI metric 805i may depend on raw data values ​​805k, 805l, and 805m. Therefore, care may be taken to divide the surgery into meaningful task segments, evaluate the skills involved in each task, determine the OPI and associate it with various skills, and define the OPI from the available data.

[0135] As an example of raw data (specifically, kinematic data), Figure 8B shows the translational motion 845a of the forceps 840 in three-dimensional space, which in some embodiments may be used to generate one or more OPIs. Figure 8C shows an example of raw data input, specifically multiple rotations in three-dimensional space around multiple forceps component axes, which in some embodiments may be used to generate one or more OPIs. The forceps 840 may rotate its various components 845b, 845c, and 845d around their respective axes 850a, 850b, and 850c. The translations and rotations in Figures 8B and 8C can be captured as raw kinematic data over time, forming raw data values ​​805k, 805l, and 805m. The OPI metric 805i could be the “forceps tip movement velocity” OPI, which may represent the velocity of the forceps tip based on the raw values ​​805k, 805l, and 805m (for example, the OPI may infer tip velocity from the Jacobian matrix derived from the raw data in Figures 8B and 8C). The OPI metric 805i may be one of several OPI metrics used as part of the model's feature vector to generate a skill score for skill 805f (or, again, a task score for task 805c). In some embodiments, a collection of skill scores may be used to assess the surgeon's performance on task 805c, and finally, the surgeon's overall performance 805a by considering all tasks.

[0136] Here again, for clarity, if we want to evaluate a surgeon's performance in one of tasks 805b, 805c, and 805d, in some embodiments, the task may be scored by considering the task's constituent skill score obtained from a skill-based model. Alternatively, in some embodiments, we may instead simply assign OPIs directly to the task and then train a task-based (not skill-based) model with the OPIs and the systems and methods disclosed herein with respect to skills (i.e., having the professional select task OPIs rather than skill OPIs, and perform OPI filtering against the set of OPIs selected for the task rather than the set of OPIs selected for skills).

[0137] OPI-Skill / Task Mapping Example Figure 8D is a pair of tables 835a and 835b, illustrating examples of OPIs to skills and skill-to-task mappings that may be applied in several embodiments (e.g., following the OPI selection process in Figure 11A, or as an initial mapping based on expert intuition and experience). For multiple skills 855c, the shaded cells in Table 835b indicate the corresponding OPI 855b. Similarly, Table 835a shows how a task 855a may correspond to a skill 855c through the shaded cells.

[0138] To clarify, in the correspondence example shown in Figure 8D, for example, all six tasks shown depend on the "Camera Use" skill, but only the "Uterine Horn" task depends on the "Two-Hand Arm Retraction" skill. Similarly, the "Dominant Arm Wrist Articulation" OPI is associated with the "Suture" skill. From these tables, transitive inferences can also be made; for example, the "Camera Control Rate" OPI is associated with the "Uterine Horn" task (since "Camera Use" is common to both tables). Thus, tables like those in Figure 8D can be used to select OPIs for both skill and task models. You will understand that many more skills, tasks, and OPIs may be applicable than those shown in this example. Also note that a single skill may be applicable to multiple tasks. Here again, the initial OPI-versus-skill correspondence may be enhanced through data-driven selection, which is described in more detail here.

[0139] As mentioned above, the initial OPI-to-skill, OPI-to-task, or skill-to-task correspondence can be determined through inspection or consultation with experts. However, as described here, selecting the appropriate OPI for a skill or task through manual inspection alone is often difficult, and therefore the automated systems shown here may be used.

[0140] Specifically, while it is true that machine learning models trained on all OPI features may naturally focus processing on more prominent OPIs (for example, neural networks may reduce weights associated with irrelevant features, or SVMs may ignore irrelevant dimensions when generally selecting hyperplanar separation), reference to such models can complicate interpretability because it may be unclear to practitioners exactly how the model selected up or down a given OPI. Instead, efficiently mapping OPIs to skills, as shown in Figure 8D, can make the skill scores reported to surgeons more generalizable and interpretable. Therefore, selecting a more efficient subset of OPIs rather than associating many OPIs with skills or tasks more than necessary makes it easier to group data into categories that surgeons can understand more easily, which in itself may facilitate a more meaningful breakdown in surgeon feedback. Furthermore, including fewer OPIs in the input features can also reduce the computational overhead between training and inference for each skill model. In fact, by relying on fewer OPIs, the system can continue to generate at least some skill scores and produce a complete set of OPI values, even if not all available data types are available (or not all data can be synchronized).

[0141] OPI Selection - Example Overview Figure 9A is a schematic diagram illustrating an exemplary set of relationships between skills, skill models, and OPIs that may be implemented in several embodiments. Here, each skill score 905a, 910a, 915a, etc., may be derived from the corresponding machine learning models 905b, 910b, 915b, respectively (however, as previously stated, in some embodiments, post-processing of the model outputs may be applied to determine the final skill score values). While some embodiments consider providing the entire set of OPIs for all skill models (in fact, some embodiments may use a monolithic model that provides the outputs for all skills considered), here, each model 905b, 910b, 915b may instead consider subsets 920a, 920b, 920c from the entire corpus 900 of available OPIs. Specifically, a human annotator (such as a specialist surgeon) may select initial subsets 905c, 910c, 915c based on their expertise and intuition associated with each skill. For example, an expert might consider whether a given OPI is related to the skill in question. The skill "camera management" might include OPIs related to camera speed, but it is unlikely to depend on OPIs related to, for example, the activation of scissors. Therefore, the first OPI subset selection for camera-related skills could include all OPIs derived from camera-related data.

[0142] Therefore, in some embodiments, the OPI values ​​of subsets 905c, 910c, and 915c may be provided to models 905b, 910b, and 915b and used to determine the skill score. However, as mentioned above, there may be merit in removing redundant or uninformative OPIs. Therefore, in some embodiments, automated filtering is applied to 905c, 910c, and 915c to determine the final set of OPIs 920a, 920b, and 920c. However, it will be understood that automated filtering may coincide with the initial subset selection (e.g., subsets 905c and 920a are the same). Similarly, some embodiments may omit initial human annotations and rely entirely on automated filtering (e.g., set 900 and the initial subset 905c are the same).

[0143] You will understand that each of the subsets 920a, 920b, and 920c, spanning different tasks, may or may not contain one or more of the same OPIs. In fact, in some cases, two or more of the subsets 920a, 920b, and 920c may share the same set of OPIs. In some embodiments, the same skill may be evaluated by different machine learning models when the skill is used on different tasks. Alternatively, in some embodiments where the training data is annotated at the task level (i.e., the portion of the data relevant to a task is identified in that way), models 905b, 910b, and 915b may be configured to receive additional input indicating a task (thus prompting the generation of task-specific skill evaluations). Thus, you will understand that in some embodiments, different task-specific skill models may receive different task-specific OPI subsets.

[0144] To effectively select an OPI subset in a reproducible and meaningful manner, various embodiments consider applying an OPI relevance evaluator 970c (e.g., software, hardware, or firmware logic), as shown in Figure 9B. Such a component may receive annotated training data 970a (i.e., surgical data annotated as being from either an expert or a non-expert with respect to a particular skill or task in question) and an initial OPI selection 970b for a given skill (or task). The OPI relevance evaluator component 970c may apply one or more filtering operations to the initial OPI selection 970b to determine a final filtered selection 970d (for example, if the initial OPI selection 970b is set to 905c, the final set 970d may be set 920a).

[0145] Figure 9C shows a high-level visualization of exemplary operation of the OPI relevance evaluator component 970d. Here, the initial OPI corpus 970b of the skill, e.g., a set 905c selected by a domain expert, may contain eight different OPIs (e.g., "OPI1", "OPI2"). The OPI relevance evaluator component 970d may pass corpus 970b to one or more filters, e.g., each of several filters 950a, 950b, and 950c. Each of these filters may return all or a subset of the members of corpus 970b by examining the corresponding data 970a for each OPI. Here, for example, filter 950a removes OPI1, 4, 6, and 8 to generate subcorpus 950d. Similarly, filter 950b generates subcorpus 950e, and filter 950c generates subcorpus 950f. Here too, some of filters 950a, 950b, and 950c may consider the expert / non-expert annotated training data 970a when performing filtering (as described in more detail here, for example, with respect to the examples in Figures 10A and 10C). Examples of filters 950a, 950b, and 950c include Mann-Whitney U tests, correlation-based filtering, linear discriminant analysis, chi-squared test analysis, t-tests (and Welch's variants), least absolute contraction and selection operator (LASSO) regression, random forests or decision trees, RFE / recursive feature removal (logistic regression-based estimators, standard ML models as base estimators, etc.), PCA / sparse PCA / Fourier methods (e.g., by retaining the OPI with the largest principal component or signal contribution in the training data), t-distributed stochastic neighborhood embedding (t-SNE), homogeneous manifold approximation and projection (UMAP) dimensionality reduction-based methods, relief feature selection, and the Boruta algorithm (e.g., implemented in the Boruta R™ language package).

[0146] After filters 950a, 950b, and 950c generate subcorpora 950d, 950e, and 950f, the OPI relevance evaluator component 970c may integrate subcorpora 950d, 950e, and 950f into a final set 970d. This integration may take different forms in different embodiments. In this example, component 970c takes a logical OR (or equivalent union) of subcorpora 950d, 950e, and 950f to generate a final subcorpus 970d on which the skill model can be trained (i.e., the model consumes OPI data for training and inference corresponding to selected OPIs in set 970d). For example, OPI2 appears only once in the final set 970d because it appears in both subsets 950d and 950e. In contrast, since OPI8 is not included in any of sets 950d, 950e, or 950f, OPI8 does not appear in the final set 970d. Here, filters are treated equally, but in some embodiments, a subset of some filters takes precedence over others, and the OPIs appearing in set 970d may be selected, for example, by a weighted vote of sets 950d, 950e, or 950f (e.g., selecting the four most common OPIs from sets 950d, 950e, or 950f). Similarly, in some embodiments, a logical AND of sets (e.g., the intersection of each corpus) may be used instead of a logical OR. In this example, the logical AND produces the final set 970d, which has only OPI2.

[0147] The best way to integrate subsets (e.g., weighted voting, logical OR, logical AND) may depend on the skill model used, the nature of the OPIs, and the computational constraints imposed. For example, if computational resources are abundant, the skill model is robust (e.g., a deep learning model), and can identify the relationships between multiple OPIs, or if the OPIs do not necessarily capture large amounts of data, then logical OR may be more suitable. On the other hand, if computational resources are limited, the skill model is not very robust, or the OPIs capture large amounts of data, then logical AND may be more suitable. In the various experimental reduction practices described here, it was found that when the model is a logistic regression classifier, the OPIs are as shown in Figures 17, 18, 19, and 20, and logical OR yields favorable results.

[0148] In some embodiments, filters can also be selected by performing preliminary validation on the data to be processed. For example, if the data is tested for assumptions such as normality and equal variances, and it is found that the data source satisfies various independence requirements, then various filters can be applied individually or in combination accordingly.

[0149] As mentioned above, filters 950a, 950b, and 950c can select OPIs according to various methods. In general, filters can be assumed to be one of three forms: single OPI statistical distribution analysis (SOSDA) filters, multiple OPI statistical distribution analysis (MOSDA) filters, and multiple OPI predictive model (MOPM) filters. SOSDA and MOSDA filters can be used to determine whether the expert distribution and the non-expert distribution of the set of OPI values ​​are sufficiently different for the selection of one or more OPIs, respectively, so that one or more OPIs may be useful for distinguishing expert and non-expert data (e.g., according to the method in Figure 10A described here). Specifically, the respective "differences" of the SOSDA and MOSDA filters can be determined according to statistical tests applied to the expert and non-expert distributions. Examples of statistical tests and analyses used in SOSDA filters include, for example, Mann-Whitney hypothesis tests, t-tests, Welch's t-tests, correlation methods, generalized linear models, and chi-squared tests. Similarly, examples of statistical tests and analyses that may be used with MOSDA filters include, for example, Wald tests, ANOVA, generalized linear models, PCA, sparse PCA, t-SNE, UMAP / other dimensionality reduction techniques, and correlation. In contrast, MOPM filters may examine the effectiveness of OPI values ​​from a selection of one or more OPIs when distinguishing expert / non-expert data in predictive models, and accordingly, may include, for example, recursive feature removal (RFE) with log registration (log reg) or other base estimators, relief feature selection, linear discriminant analysis, LASSO regression, random forests, decision trees, and Volta feature selection. Thus, you will understand that filters 950a, 950b, and 950c may all be SOSDA filters, all be MOSDA filters, or all be MOPM filters, and that some filters may be one type of filter while others are of a different type.Similarly, both MOSDA and MOPM filters will recognize that, as a statistical test, clustering methods such as K-means, K-nearest neighbors, hierarchical clustering, and density-based spatial clustering of applications with noise (DBSCAN) may be employed, or that iterative model predictions may subsequently be used to evaluate the dissimilarity or predictive effectiveness of the identified clusters.

[0150] OPI Selection - Example of SOSDA / MOSDA OPI Filtering Process Figure 10A is a flowchart illustrating an exemplary process 1010 of SOSDA-style OPI filtering, which can be implemented in several embodiments, but it can be seen that the steps can be applied to MOSDA filtering by considering multiple OPIs at once, rather than one OPI at a time. Here, in blocks 1010a and 1010b, the system may iterate through the OPIs of set 970b (again, you will understand that in MOSDA-style filtering, a collection of OPIs, rather than a single OPI, may be considered in block 1010b). For each of these OPIs, the system may, in block 1010c, refer to the corresponding expert / non-expert annotated (again, “expert” on the skill in question) data 970a to determine the respective distributions of OPI values ​​in the expert and non-expert datasets for the skill. In block 1010d, the system may then consider whether the distributions are similar or dissimilar.

[0151] For example, if the skill is "camera movement" and the OPI considered in block 1010b is "duration of camera movement," the distributions may be very dissimilar, as novice users may often take longer to position the camera with greater variability than experts who can position the camera quickly and accurately. Conversely, if the OPI considered in block 1010b is "focus" (e.g., evaluated by finding the widest frequency variety from the Fourier transform of the video image), both experts and non-experts may be able to achieve good focus quickly. Therefore, the data distributions may be very similar. Dissimilar OPI distributions may be more useful in distinguishing between experts and non-experts, so they are retained in block 1010e, while similar distributions may result in the OPI being removed in block 1010f. For clarity, Figure 10B shows an example of an OPI value distribution. If the expert distribution of an OPI (e.g., "focus") is distribution 1015a and the non-expert distribution is distribution 1015b, then the OPI may be an insufficient means of distinguishing between experts and non-experts and may be removed from the set. In contrast, if the expert distribution of an OPI (e.g., "camera motion duration") is distribution 1015c and the non-expert distribution is distribution 1015d, then the OPI may be an appropriate means of distinguishing between experts and non-experts and may be retained in the set. Once all OPIs have been considered in block 1010a, the final set of retained OPIs (e.g., subcorpus 950d) may be output in block 1010g.

[0152] You will understand various mechanisms for evaluating the quantitative determination of "similarity" between distributions. For example, some embodiments may directly compare the means and variances of distributions, perform a t-test, and evaluate the p-value. Nonparametric tests that do not assume a normal distribution, such as the Mann-Whitney U test, are particularly useful when dealing with imbalanced data, as in the case of the OPI value distribution considered here. You will understand that there are various libraries for performing many of these tests, including the Mann-Whitney U test. For example, the SciPY(trademark) library provides the scipy.stats.mannwhitneyu function. In a practical application, we have found that using this function to identify distributions with a U statistic p-value less than 0.05 as "dissimilar" (e.g., in block 1010d) yields good results. In some embodiments, family-wise error correction such as Bonferroni and Bonferroni-holm may also be applied to reduce false conclusions (e.g., Bonferroni may reduce false positives at the expense of potentially increasing false negatives).

[0153] OPI Selection - Example of the MOPM OPI Filtering Process For clarity, Figure 10C is a flowchart illustrating an exemplary process 1005 for performing OPI selection using an MOPM filter, which may be used, for example, according to an RFE. Specifically, in block 1005a, the processing system may split the expert / non-expert annotated data 970a into a training subset and a validation subset for all initial sets of OPI values ​​970b. In block 1005b, the processing system may train a machine learning model (e.g., a logistic regression classifier model, an SVM model, etc.) on the OPI data in this training set to recognize OPI values ​​associated with expert and non-expert surgeons. In block 1005c, the processing system may validate the model using the validation subset. In block 1005d, the processing system may determine the order of OPIs based on their importance in impacting the correct classification during validation. For example, each of the initial OPIs may be selectively removed, and the model performance may be re-evaluated, with these OPI removals being deemed more important if they increase the variability of the output. You will understand that some machine learning models, such as Random Forest, can provide importance scores as part of their processing (e.g., showing how much accuracy would be lost if an OPI were excluded). After determining the importance order of the OPIs, the system can remove less important OPIs in blocks 1005f through 1005m (although various embodiments may omit these operations to simply select the most important OPIs above a threshold).

[0154] Specifically, in block 1005e, the subset size counter CNTS may be initialized to 1 (the use of the counter here is merely for ease of understanding, and it will be understood that equivalent functionality can be easily implemented in various ways). This counter tracks how many S OPIs are considered from set 970b (in order of importance as determined in block 1005d). Thus, the number of OPIs may be incremented in blocks 1005f and 1005g until all S of the most important OPIs (as determined in block 1005d) are considered in block 1005f.

[0155] In each iteration, the most important OPI of CNTS in block 1005g may be considered and used to train the machine learning model in block 1005h. The system can then validate the trained model in block 1005i (e.g., with the same raw data and splits as in block 1005a). The effectiveness of this validation can be used as an estimate of the fit of the selected set of OPIs. Thus, the result, for example, the model accuracy in validation, may be recorded in block 1005j. CNTS is incremented in block 1005k, and additional iterations may be performed.

[0156] Once all desirable sets of OPI have been evaluated through validation in block 1005i, the system may determine the performance profile of each set selection in block 1005l. The system may then select a final set of OPI, e.g., the smallest set that achieves an acceptable or best validation result, in block 1005m.

[0157] You will find that many statistical packages can facilitate the application of SOSDA, MOSDA, or MOPM filters. For example, RFE can be implemented using the "sklearn.feature_selection.RFE" class from the scikit-learn™ library. The following code line listing C1 is used in a practical application of one embodiment and has been found to produce results suitable for this purpose. rfe = RFE (estimator = SVC(kernel="linear"), step=1) (C1) To reiterate, while SVM was used in the example in code listing C1, you will see that RFE can also be used with other models, such as random forests and logistic regression classifiers.

[0158] Example of a skill model training process using OPI filtering Figure 11A is a flowchart illustrating various operations in an example of process 1100 for evaluating skill (or task) model configurations and OPI selections and generating expertise models (e.g., skill models or task models), which can be implemented in several embodiments. Generally, process 1100 can facilitate parameter selection for expertise machine learning model classifiers by integrating OPI selections and cross-validation operations. Cross-validation will be understood as a method of iteratively training multiple model configurations to achieve more robust model configurations than those that could be generated by training on all or part of the available data. Specifically, for reference and clarification, refer to the example of training data 1115a in Figure 11B, where training data 1115a is in the form of selected features that can be annotated as described here (e.g., raw data values ​​or OPI values ​​annotated as being associated with expert or non-expert). This data can be divided into a training portion 1115b and a test portion 1115c (in some embodiments, the test portion 1115c may be omitted, and all available training data may be used as the training portion 1115b). The training portion 1115b itself may be used to determine each of the model's hyperparameters and to validate the model, while the test portion 1115c may be reserved to provide a final validation evaluation of the generated model or the final model derived from the generated model. For this purpose, the training portion 1115b itself may be divided into "folds" of roughly equal groups of data (three such folds are shown here). In each training iteration, the version of the model's hyperparameters may be determined by training the model using some or all of the folds from the training portion 1115b (for example, the first trained model may be generated using folds 2 and 3, with fold 1 being used to validate the model; the second model may be trained with folds 1 and 3, with fold 2 being used for validation, etc.).Each of the generated models and their validation results can then be analyzed to evaluate the effectiveness of the selected model parameters (e.g., layer selection in a neural network, kernel selection in an SVM). The most preferred parameters can then be applied to the new model and the trained model, for example, the entire dataset 1115b, and evaluated using a separate reserved test portion 1115c.

[0159] Using process 1100, multiple such intermediate models with validation results can be generated, and if necessary, a final model can be generated in a similar manner if the parameters produce satisfactory results. As discussed with respect to Figure 9A, an expert selection subset of the OPI corpus (or the entire OPI corpus) may be received in block 1190. Training data (i.e., expert / non-expert annotated raw data values ​​or OPI values) can also be provided. This data may be allocated in block 1186 to a training portion (corresponding to portion 1115b) and a test portion (corresponding to portion 1115c). As previously stated, the training portion itself can be divided into a desired number of folds, from which a desired number of selections T2 can be derived in each iteration (e.g., T2=5). For example, the first selection may be trained in folds 1 and 2 and validated in fold 3, the second selection may be trained in folds 2 and 3 and validated in fold 1, and so on. Therefore, in block 1105, the outer counter Cnt2 may be initialized to 1 and used to iterate over the fold selection in block 1187 (again, the use of counters here is merely for ease of understanding, and you will understand that equivalent functionality can be easily implemented in various ways).

[0160] For each selection of folds considered, in block 1110, the system may determine a first OPI subset using a SOSDA or MOSDA OPI filtering method, e.g., filtering by the Mann-Whitney U test. For clarity, it will be understood that the raw data referenced by the OPI values ​​of the SOSDA or MOSDA OPI filtering for each performance in block 1110 is that of the folds from the current selection from block 1187 (e.g., all folds within the selection range, however, in some embodiments, fewer folds than all folds may be adopted). Once a first set of OPIs is selected via the SOSDA or MOSDA filter in block 1110, the system may attempt to generate additional OPI subsets via the MOPM filter in block 1145. Since the MOPM filter may employ its own machine learning model, an internal loop cross-validation approach may be desirable here as well. Specifically, the current fold selection from block 1187 (e.g., both the folds allocated for training and the folds allocated for validation) can be split into a subfold and a desired iterative subfold selection for T1, which is considered via an internal counter Cnt1 as shown in blocks 1120, 1125, and 1188. Cnt1 can be incremented in block 1125 while it remains below a first threshold T1 (e.g., a desired number of subfold selections to be considered) in block 1120.

[0161] However, here, instead of referring to the "actual" OPI value distribution from the original data received in block 1190 (as in the case of the SOSDA or MOSDA filters in block 1110), each OPI selection in block 1145 may instead be determined based on a synthetic dataset created through the operations of blocks 1130, 1135, and 1140 (collectively referred to as the "intermediate synthetic dataset" generation operation 1195a).

[0162] In other words, for each iteration of T1, upsampling via SMOTE, for example, may be applied in block 1130 to the expert-annotated portion of the subfold selection, for example, to upsample underrepresented expert data. Expert data can be upsampled, while non-expert data can be downsampled. For example, downsampling may proceed in two stages. Firstly, in block 1135, a neighbor cleaning rule (NCR), a downsampling method, may be used to reduce noisy data in larger non-expert groups by removing outlier points that are far from other non-expert samples (effectively cleaning noisy samples). NCR may be implemented in some embodiments using a library, for example, the function "imblearn.under_sampling.NeighbourhoodCleaningRule" from the imblearn(trademark) library. Secondly, in block 1140, the processing system may randomly downsample any remaining non-expert values ​​that were not removed by NCR. This may have the effect of balancing the expert and non-expert groups to a ratio of approximately 50 / 50. Because the NCR method may not sample up to a certain ratio, block 1140 can compensate for the operation of the NCR. For small class sizes, this combination of blocks 1135 and 1140 may perform better than random downsampling alone, although in some embodiments, block 1140 may be used instead and block 1135 may be omitted, or vice versa. Figure 15 shows examples of OPI data before and after the downsampling process of blocks 1130, 1135, and 1140.

[0163] To clarify, you will understand that these operations can be integrated through various libraries, such as scikit-learn (commercial) and imblearn (commercial), as shown in the example code line listing from C2 to C8. [Table 1] Here, "upratio" could be, for example, the ratio of SMOTE expressed as 3x up sample. Again, these rows are just examples, and it can be seen that different numbers of folds or models may be used in other embodiments.

[0164] As previously stated, in block 1145, an MOPM filter, such as the RFE described here, may be applied to the synthetic dataset generated by operation 1195a to determine a second subset of OPIs. An MOPM filter, such as an RFE, may in block 1145 (as described with respect to Figure 10C, for example) determine the set of OPIs to use for a skill or task that satisfies both the maximum cross-validation (CV) score criterion and the minimum feature count criterion (thus, if two sets of OPIs of different sizes are found to perform the same function, the filter may select the smaller of the two sets). For clarity, the SOSDA or MOSDA filter in block 1110 may act as a check for OPIs omitted in block 1145 that are worth including (i.e., by taking a logical OR in block 1150, as the sets may be combined or compared in other ways as discussed elsewhere). Again, it is shown here that block 1110 occurs before the internal cross-validation loop for clarity, but you will understand that in some embodiments the order may be reversed, or these two may be pursued in parallel.

[0165] When Cnt1 exceeds the threshold T1 in block 1120 (i.e., all subfold selections have been considered), in block 1150, the system may take the logical OR of the subset generated in block 1110 and each subset generated in each iteration of block 1145. For example, if T1=5, block 1150 may combine six OPI corpus sets: the set generated in block 1110 and the five sets generated in the five iterations up to block 1145. However, it will be interesting to understand the variations, as various embodiments may instead cross-validate the results from the inner loop and generate the best single set based on each set generated in block 1145. In these variations, naturally, only two sets are joined in block 1150 (the best set from the internal cross-validation of the set generated in block 1145 and the set generated in block 1110). The joined corpus in block 1150 can then be used to train a machine learning model on the current fold selection (selected in block 1187) for the T2 fold selection.

[0166] Specifically, in block 1155, the external counter Cnt2 may be incremented. Another “training” synthetic dataset may be generated again, as a model trained on balanced data is more likely to provide more robust inference results, but this time from the entire selection of training data folds (or more) from block 1187. This training synthetic dataset may be generated using operation 1195b, for example, a sequence of SMOTE, NCR, and random downsampling applied in blocks 1160, 1165, and 1170 respectively (for example, based on fold selection rather than subfold selection, in the same way as described above with respect to blocks 1130, 1135, and 1140). As described above with respect to the internal validation loop, here several embodiments may similarly omit various blocks 1160, 1165, and 1170. However, retaining the blocks may be beneficial, as block 1170 may again compensate for the operation of NCR.

[0167] In block 1175, the system may train an experience classifier (task or skill model) using data from a synthetic training dataset corresponding to the OPI values ​​of the merged corpus from block 1150. Again, for clarity, it will be understood that these operations can be integrated via various libraries, such as scikit-learn® and imblearn®, as shown in code line listings C9 through C14. [Table 2] Here, similarly, the "upratio" could be, for example, the ratio of SMOTE expressed as 3x up sample. Again, these lines are just examples, and it can be seen that different numbers of folds or models may be used in other embodiments.

[0168] Subsequently, the system can evaluate the effectiveness of the classifier in block 1180. Unlike training a model using a synthetic training dataset, validating the model in block 1180 will reveal that the data for the current fold selection may be used in its original, unbalanced form. This ensures that the model's performance is evaluated against operations on real-world data (e.g., input data not synthetically balanced during inference), as synthetically increasing the expert sample size can generate inaccurate performance metrics.

[0169] Again, for clarity, each iteration in blocks 1175 and 1180 may generate a new trained model and a new corresponding evaluation of that model. For example, in some embodiments, in block 1175, the filtered OPI training data (from block 1150) is used to train a logistic regression classifier (or other suitable model, e.g., one discussed here), so that multiple such trained classifiers may be generated in each iteration (and their corresponding evaluations may be performed in block 1180). From these evaluations, a model configuration and OPI selection that yields a favorable result can be inferred. For example, the "best" model configuration and OPI selection may then be used for inference, or to guide the creation of a new model with similar or the same parameters. In some embodiments, performance in block 1180 may be measured using two methods: a balanced accuracy score to account for large differences in sample size (e.g., the average of recalls between the two groups, such as ROC-AUC, F1 score, or Jaccard), and the Matthews correlation coefficient (MCC), which is a balanced quality measure of classification ranging from -1 (poor prediction) to 1 (perfect prediction). In block 1185, if Cnt2 has not yet exceeded the outer threshold T2, the outer loop may be run again.

[0170] To evaluate the relative merits of the selections, process 1100 may be run multiple times, except for block 1189, for multiple parameter selections (e.g., changing the initial OPI selection in block 1190, changing the model selection and configuration parameters in block 1175, or the selection between MOPM and SOSDA / MOSDA). Once parameters that produce a favorable result are identified, the final model may be trained using those parameters, as shown in block 1189, or alternatively, for example, the entire training portion 1115b for training the model and the reserved portion 1115c for testing (or, for example, the model may be trained with all available data without testing).

[0171] If sample size is limited, a small number of cross-validation folds (e.g., T1=5, T2=5) may be performed. This prevents overestimating the model's performance due to the reuse of the same set of data multiple times. If more expert data is available, more folds (e.g., T1=10, 20, etc.) may be used. Note that in some embodiments, down (or up) sampling may be intentionally avoided in block 1110, which may result in false positives. In fact, Mann-Whitney may handle asymmetric datasets well. This test may use conservative multiple-test corrections for some reasons in order to provide stricter filtering (e.g., Bonferroni, Bonferroni-Holm, etc.) in some embodiments.

[0172] Therefore, combining the SOSDA / MOSDA and MOPM approaches in this manner may yield synergistic effects that neither approach can achieve alone (although, as mentioned above, in some embodiments, one or more SOSDA / MOSDA or one or more MOPM filters can be applied individually or together). Rather than simply identifying the minimum effective number of OPIs for distinguishing experts from non-experts, employing two or more SOSDA, MOSDA, or MOPM approaches may also help to obtain OPIs that intuitively distinguish expert / non-expert groups and work well for modeling, even if they are greater than the minimum. Such various OPI captures may themselves affect the training of the model during cross-validation. These combined approaches may be particularly effective when used in combination with the resampling techniques disclosed herein (e.g., SMOTE, random downsampling, NCR, etc.).

[0173] Example of score generation In some embodiments, the direct output from a skill model may be used to provide feedback to a surgeon. For example, knowing that the model believes a surgeon's performance is 80% likely to be professional performance may be a meaningful statement. However, the nature of some models, such as logistic regression classifiers and support vector machines, may not provide an output that directly maps to skill levels.

[0174] For example, as shown in the abstract feature space 1205b in Figure 12A, in the first situation, the group of expert OPI values ​​1205a may be located at a large distance 1205e in the feature space from the group of non-expert OPI values ​​1205i. Similarly, in the second situation, the group of expert OPI values ​​1205d may be located at a much shorter distance 1205g in the feature space from the group of non-expert OPI values ​​1205h. In either situation, some machine learning models can determine the separator 1205f. If the model is an SVM, the distance from the hyperplane separator to the new feature point 1205c can be used as a proxy for the probability that the point is in a class (you will find that many binary classifiers, like some SVMs, can usually only output the predicted class without any percent prediction). However, the distance in the feature space is not intuitively analogous to performance as a surgeon. Similarly, if separator 1205f is the separation plane of a logistic regression classifier, the distance from separator 1205f may correspond to the value of the sigmoid function and the corresponding output probabilities. Intuitively, one would expect a new feature 1205c to receive different probability assignments in the first instance of the group (i.e., 1205a and 1205i) compared to a second instance of the group (i.e., 1205d and 1205h). However, one can imagine a situation where the properties of the sigmoid function precipitates similar or identical probability values ​​for feature 1205c in each instance. Similarly, the sigmoid function may plateau, providing similar probabilities for new features that are quite far from the separator. Thus, such sigmoid mapping may not have an intuitive correspondence with skill levels. While examples of SVM and logistic regression classifiers are shown here, similar discontinuities between scores and predicted probabilities can be imagined based on the feature spaces of other models.

[0175] To compensate for discontinuities between predicted probabilities from model outputs and possible skill scores, various embodiments consider a post-processing step (e.g., in post-processing module 710f) to map model probabilities to surgeon skill levels. For example, Figure 12B is a flowchart illustrating the process of determining score mappings from model outputs based on a reference population, which may be performed in several embodiments. In block 1210a, the component may review outputs from a skill model of a known population of surgeons, which may be annotated to indicate expert and non-expert values, such as training data used to train the model (although another randomized population may be used instead). In block 1210b, the component may generate mappings between model outputs and surgeon skill levels. In block 1210c, the system may record the mappings for future use, for example, when considering new subject data. For example, if future results are generated during inference, post-processing module 710f may index the model outputs based on the mappings recorded in block 1210c, interpolating as needed.

[0176] To clarify the process in Figure 12B, consider a hypothetical sample population of reference surgeons 1215b in Figure 12C. Each of these surgeons may have provided surgical performance captured in raw data used to generate the corresponding OPI value, which then passed through a skills (or task) model, thereby generating the corresponding probability of being an expert 1215a. In this example, the mapping in block 1210b could order the model output 1215a to an order of decreasing magnitude 1215c, and then map a linear scoring metric 1215d to the values ​​shown (a linear metric is used here for ease of understanding, but you will understand that in some embodiments, a bell curve or mapping corresponding to the proportion of experts and non-experts in the reference population 1215b could be used instead). The values ​​generated during inference between the ranked outputs can generate corresponding values ​​from the scoring metric. For example, if the performance of a new surgeon 1215e were applied to the same skills model, a probability of being an expert of 0.73 might be generated. Since this probability corresponds to position 1215f in the ranked order 1215c, the final skill score may be output based on the corresponding position of metric 1215d, for example, the mean of 87.5 and 75 (81.25), the mean scaled by the linear position between the model output boundary values ​​(i.e., (0.73-0.5) / (0.75-0.5)=0.92 and 0.92*(87.5-75)+75=86.5), etc.

[0177] The above approach may be particularly effective when the probability distribution of the model output is well separated across the ranking data. In some embodiments, other approaches may be employed when the variance of groups of probabilities in the data is small. For example, some embodiments may estimate kernel densities to find local maximums for probability groupings and associate those maximums with ranks or sets of ranks (e.g., in the case of a single maximum, the majority of the sample may show a 50% score). In embodiments, the standard deviation of such distributions may be estimated to determine when a sample deviates large enough to constitute a large change in rank. Without such estimation, very small changes in machine learning output can lead to undesirable large changes in the final score.

[0178] By estimating a mixture of distributions, embodiments can correlate clusters of rankings, rendering scores more robust to variations in model predictions and giving them more intuitive meaning for human interpreters. Jenk's natural break optimization, similar to the application of a one-dimensional K-means algorithm, can be applied similarly in several embodiments.

[0179] Example of score display and maintenance To provide feedback on a surgeon's performance, it will be helpful to understand several ways in which the above results may be presented to the surgeon. Figure 13A is a schematic block diagram showing a typical hierarchical input / output topology that may be used for score generation in several embodiments. As previously mentioned, the output of each skill model (or, if a monolithic model is used, the skill-specific output of the model) may be post-processed to generate skill scores. The skill scores can then be associated with corresponding tasks from the raw data from which the skill scores were derived. For example, task B may depend on skills A, B, and C. Therefore, data from a surgeon performing task B may be used to generate scores 1305a, 1305b, and 1305c. These scores themselves may be combined to form a score 1310b of the surgeon's task performance (alternatively, or in a complementary way, a separately trained task model, as described here, may be used to create the score, and the final task score may be, for example, the average of the score determined by this task model and the score 1310b determined by cumulative skills). For example, the scores may be weighted by their relative importance to the task, summed, and normalized to form a score 1310b. Such scores from all tasks, e.g., 1310a, 1310b, etc., can similarly be combined to form an overall surgical score 1315. For example, the combined score 1315 may be the average of the task scores weighted by the relative importance of each task score to the surgical procedure (and / or surgical time).

[0180] The scores in Figure 13A may allow surgeons to receive both granular and overall feedback, enabling them to track their progress at various levels of detail over time. As an example of providing such feedback, Figure 13B is a schematic representation of a graphical user interface screenshot 1320a that depicts a performance metric overlay on a video of a surgeon's performance. Specifically, when a recorded video 1320f plays back the surgeon's performance, the overlay may include icons indicating the surgeon's score for the portion of the surgery being depicted, such as a pie chart shown in icon 1320e (for example, according to the windowed application of the skill model described with respect to Figure 7C). For example, portions of the pie chart may be shaded in different colors depending on the percentage value of the corresponding score. Icons such as icons 1320b and 1320c may allow the user to select the scores they wish to display. As discussed with respect to Figures 7C and 7D, if segment-specific scores are available, the score value shown in icon 1320e may change during the video presentation (similarly, icons 1320b and 1320c may change as the video depicts new tasks and their corresponding skills). Alternatively, in some embodiments, icon 1320e may reflect a final score value determined based on the data available up to the moment currently depicted in the video. In some embodiments, a plot, such as that shown in Figure 7D, may be overlaid on one or more skills and on icons, such as arrows, used to indicate where the currently displayed frame of video 1320f corresponds to the plot.It will be understood that such scored feedback can be extended beyond mere surgical scores, including manual review results (e.g., the Global Evaluative Assessment of Robotic Skills, The Journal of Urology, 187(1):247-252, 2012, described by Goh et al. in Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills), grouping skill score results into other expertise categories (showing a physician's relative performance compared to other physicians who have performed a similar number of procedures), and searching for data for potential skill groupings (e.g., performing unsupervised methods / clustering on the population of skill or task score results).

[0181] Examples of continuous training and score mapping updates Figure 13C is a flowchart illustrating various operations in an example of the update process for a skill assessment system (e.g., performance assessment system 625) that may be implemented in several embodiments. Specifically, as more annotated data becomes available in block 1325a, the system may update the skill (and / or task) model accordingly in block 1325b (e.g., applying online learning methods to the neural network skill model, retraining the logistic regression classifier with new population data, etc.). The system may update the score mapping in block 1325c, for example, as described above with respect to Figures 12B and 12C.

[0182] Similarly, as a new sensor becomes available in the operating room in block 1325d, various embodiments may modify the OPI corpus in block 1325e. This may make new feature vectors available, allowing for the updating of the corresponding score mapping in block 1325g, as well as the retraining or updating of the skill (and / or task) model in block 1325f.

[0183] Examples of applying the results to practical situations The effectiveness of the system and method described herein has been demonstrated through the implementation of various embodiments. Figure 14A is a bar graph showing the types and quantities of data samples available for use in exemplary reductions of the embodiments into practice. As shown, annotated data for “Experts,” “Trainees,” and “Training Specialists” were obtained for each skill related to various tasks: “Bilateral Suturing,” “One-Handed Suturing,” “Continuous Suturing,” “Uterine Horn,” “Suspension Ligament,” and “Rectal Artery / Venus 1.” Trainees were surgeons with no experience in robotic surgery and were therefore grouped as “Non-Experts.” Expert surgeons performed over 1000 da Vinci® robotic surgeries. Training specialists were expert operators who were not surgeons but had gained experience in training task exercises evaluated by ~300-600 hours of practice or use on the robotic platform. Therefore, training specialists were also treated as “Experts.” There were 7-9 tasks from the datasets of expert surgeons and training specialists ("Experts") and 93-122 tasks from the trainee group ("Non-Experts"). When there were many trainee participants, five trainees were randomly selected for each task to be excluded from the training process, leaving 88–117 for feature selection. Each skill task combination of mapped OPIs started with 4–20 OPIs, and RFE reduced the OPI set to 1–18 OPIs. The RFE of the task models showed high, balanced accuracy, which typically plateaued early. The Mann-Whitney test with Bonferroni correction generated overlapping feature sets, adding 0–11 OPIs to the final model.

[0184] Data from "trainees" and "training specialists" were grouped as "non-expert" data, while "expert" data formed a separate "expert" data group. As mentioned earlier and reflected in this real-world example, there was considerably more "non-expert" data than "expert" data. The datasets were acquired from recording devices on the da Vinci Xi® and da Vinci Si® robotic surgical systems.

[0185] As shown in Figure 11A, six logistic regression classifiers were trained as task models, and eighteen logistic regression classifiers were trained as skill models (tasks identified in Figure 14B, with corresponding skills associated with each task). Figure 14B is a table showing the mean cross-validation performance metrics, balanced accuracy, and MCC for each of the skill task and the overall task logistic regression model (i.e., models trained on the overall task OPI rather than the task skills, as described here) in an exemplary reduction to practice of one embodiment. Accuracy here refers to a balanced accuracy score, such as that which can be generated using the scikit-learn® library function sklearn.metrics.balanced_accuracy_score. MCC can similarly be generated using the scikit-learn® library function sklearn.metrics.matthews_corrcoef.

[0186] Figure 15 is a pair of schematic dot plots showing the economy of motion OPI values ​​for the four instruments in the uterine horn task before and after the application of resampling in exemplary practice reduction examples, as described with respect to blocks 1130, 1135, and 1140 or blocks 1160, 1165, and 1170 of Figure 11A. Figure 16 is a collection of a set of schematic line plots showing the distribution of task duration by experience level in practice reduction examples and cross-validated scores of various numbers of OPIs per skill using RFE. Specifically, plots 1605a, 1605b, 1605c, 1605d, 1605e, and 1605f show task duration for different groups, and plots 1610a, 1610b, 1610c, 1610d, 1610e, and 1610f show RFE performance. As shown, the lines in the RFE plots represent different skills. For plots 1610a-f, the vertical axis represents the balanced predictive accuracy of the cross-validation of the model, and the horizontal axis reflects the number of OPIs used in the model for each skill shown in the plot.

[0187] Figures 17, 18, 19, and 20 are tables listing example collections of OPIs that can be used in various embodiments, their descriptions, and their relationships to various skills and tasks. For robot arms, "SCE" refers to "Surgeon Console," "Cam" to the arm holding the camera, "D" to the dominant arm of the robot system, "ND" to the non-dominant arm of the robot system, and "Ret" to retracting the robot arm. For skills, "E" refers to "Energy," "S" to "Suture," "D" to "Incision," "CU" to "Camera Use," "AR" to "Arm Retraction," "1-HD" to "One-Handed Incision," and "2-HAR" to "Two-Handed Arm Retraction." For tasks, "SL" refers to the "Suspension Ligament" task, "2-HS" to the "Two-Handed Suture" task, "1-HS" to the "One-Handed Suture" task, "RS" to the "Continuous Suture" task, "UH" to the "Uterine Horn" task, and "RA / V" to the "Rectal Artery / Venus" task.

[0188] Based on this implementation example, and based on the odds ratio per OPI calculated from the coefficients of the logistic regression model, it became clear that surgeons could improve their energy skills by reducing unnecessary energy activation (reducing total events) while simultaneously applying energy more frequently in shorter periods of time (increasing frequency). Similarly, the results showed that not only increasing the frequency of adjusting the camera to improve the surgeon's field of view, but also adjusting it at a faster speed, could improve their camera skills.

[0189] In this practical application, for many skills, only a small subset (2-10) of the OPI was needed to achieve the best model accuracy (80-95%) for estimating technical skills. Most skill-specific models had similar accuracy to models trained to predict expertise across the entire task (80-98%).

[0190] Computer system Figure 21 is a block diagram of an example computer system that may be used in combination with some of the embodiments. The computing system 2100 may include an interconnect 2105 that connects several components, such as one or more processors 2110, one or more memory components 2115, one or more input / output systems 2120, one or more storage systems 2125, and one or more network adapters 2130. The interconnect 2105 may be, for example, one or more bridges, traces, buses (e.g., ISA, SCSI, PCI, I2C, Firewire bus, etc.), wires, adapters, or controllers.

[0191] One or more processors 2110 may include, for example, Intel® processor chips, numerical coprocessors, graphics processors, etc. One or more memory components 2115 may include, for example, volatile memory (RAM, SRAM, DRAM, etc.), non-volatile memory (EPROM, ROM, flash memory, etc.), or similar devices. One or more input / output devices 2120 may include, for example, display devices, keyboards, pointing devices, touchscreen devices, etc. One or more storage devices 2125 may include, for example, cloud-based storage, removable USB storage, disk drives, etc. Depending on the system, memory components 2115 and storage devices 2125 may be the same component. Network adapters 2130 may include, for example, wired network interfaces, wireless interfaces, Bluetooth® adapters, line-of-sight interfaces, etc.

[0192] In some embodiments, it can be seen that only some of the components, alternative components, or additional components may be present compared to those shown in Figure 21. Similarly, in some systems, components may be combined or serve dual purposes. Components can be implemented using, for example, one or more special-purpose hardwired circuits such as ASICs, PLDs, or FPGAs. Thus, some embodiments can be implemented, for example, with programmable circuits programmed with software and / or firmware (e.g., one or more microprocessors), with entirely special-purpose hardwired (non-programmable) circuits, or in a combination of such forms.

[0193] In some embodiments, data structures and message structures can be stored or transmitted via the network adapter 2130, over a data transmission medium such as signals on a communication link. Transmission can take place over various media, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, “computer-readable media” can include computer-readable storage media (e.g., “non-temporary” computer-readable media) and computer-readable transmission media.

[0194] One or more memory components 2115 and one or more storage devices 2125 may be computer-readable storage media. In some embodiments, one or more memory components 2115 or one or more storage devices 2125 may store instructions that perform or can perform the various operations discussed herein. In some embodiments, instructions stored in memory 2115 may be implemented as software and / or firmware. These instructions may be used to perform operations on one or more processors 2110 in order to perform the processes described herein. In some embodiments, such instructions may be provided to one or more processors 2110 by downloading instructions from other systems, for example, via a network adapter 2130.

[0195] remarks The drawings and descriptions herein are illustrative. Therefore, neither the descriptions nor the drawings should be interpreted as limiting the disclosure. For example, titles and subtitles are provided simply for the convenience and ease of understanding of the reader. Therefore, titles and subtitles should not be interpreted as limiting the scope of the disclosure, for example, by grouping features presented in a particular order or simply by summarizing them for ease of understanding. Unless otherwise specifically defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by an ordinary technician in the art relating to this disclosure. In case of any conflict, this document, including the definitions provided herein, shall prevail. Reference of one or more synonyms herein does not preclude the use of other synonyms. Any use of examples anywhere in this specification, including examples of terms discussed herein, is illustrative only and is not intended to further limit the scope and meaning of the terms disclosed or illustrated.

[0196] Similarly, despite the specific presentation in the diagrams herein, those skilled in the art will understand that the actual data structures used to store the information may differ from those shown. For example, the data structures may be composed in different ways, contain more or less information than shown, or be compressed and / or encrypted. Diagrams and disclosures may omit common or well-known details to avoid confusion. Likewise, diagrams may illustrate a particular set of actions for ease of understanding, which is merely an example of a broader class of such actions. It is therefore readily apparent that additional, alternative, or fewer actions may often be used to achieve the same purpose or effect as shown in some flowcharts. For example, although not shown in the diagrams, the data may be encrypted, items may be considered in different loop patterns ("for" loops, "while" loops, etc.), or items may be sorted in different ways to achieve the same or similar effect.

[0197] The references herein to “one embodiment” or “one embodiment” mean that at least one embodiment of this disclosure includes certain features, structures, or characteristics described in relation to that embodiment. Therefore, the expression “in one embodiment” in different places here does not necessarily refer to the same embodiment in each of those different places. Different or alternative embodiments are not necessarily mutually exclusive with other embodiments. It will be recognized that various modifications can be made without departing from the scope of the embodiments. The following note is added. (Note 1) A computer-implemented method for generating a score based on surgical data, wherein the method is: Steps to obtain surgical data; A step of converting the aforementioned surgical data into multiple metric values; and The steps include: generating scores related to the surgical data using a machine learning model, at least partially; A computer-implemented method. (Note 2) The steps to generate the score are: The steps of providing the metric values ​​to the machine learning model in order to generate predictions; and The step of mapping the aforementioned prediction to a score; The computer-implemented method described in Appendix 1. (Note 3) The aforementioned machine learning model is: Neural network; Support vector machine; Random Forest; Logistic regression classifier; and Logic configured to predict classification based on data patterns; Including one or more of the following: The computer-implemented method described in Appendix 2. (Note 4) The aforementioned machine learning model is a logistic regression classifier, The step of generating the score includes inputting the metric value into the logistic regression classifier and applying the output of the logistic regression classifier to the mapping of the classifier result to the skill score, wherein the mapping of the classifier result to the skill score is generated based on surgical data from a reference population. The computer-implemented method described in Appendix 2. (Note 5) The surgical data includes one or more of the following: patient-side kinematic data, surgeon-side kinematic data, visualization tool video, and system events. Computer-implemented methods as described in Appendix 2 or 4. (Note 6) The metrics were selected, at least in part, using a single OPI statistical distribution analysis filter. The computer-implemented method described in Appendix 2. (Note 7) The metric was selected, at least in part, using multiple OPI statistical distribution analysis filters. The computer-implemented method described in Appendix 2. (Note 8) The metric was selected, at least in part, using multiple OPI prediction model filters. The computer-implemented method described in Appendix 2. (Note 9) The metric was selected by cross-validation during the training of the machine learning model. Computer-implemented methods as described in Appendix 2, 6, 7, or 8. (Note 10) The above method further: The step includes presenting the score as one of several scores generated throughout the process of a surgeon's surgical performance, The computer-implemented method described in Appendix 2. (Note 11) The score mentioned above is a skill score, and the machine learning model mentioned above is a skill model. The computer-implemented method described in Appendix 10. (Note 12) The score mentioned above is a task score, and the machine learning model is a task model. The computer-implemented method described in Appendix 10. (Note 13) A non-temporary computer-readable medium containing instructions configured to cause a computer system to execute a method, wherein the method is: Steps to obtain surgical data; A step of converting the aforementioned surgical data into multiple metric values; and The steps include: generating scores related to the surgical data using a machine learning model, at least partially; Non-temporary computer-readable media. (Note 14) The steps for generating the score are: The steps of providing the metric values ​​to the machine learning model in order to generate predictions; and The step of mapping the aforementioned prediction to a score; Non-temporary computer-readable media as described in Appendix 13. (Note 15) The aforementioned machine learning model is: Neural network; Support vector machine; Random Forest; Logistic regression classifier; and Logic configured to predict classification based on data patterns; Including one or more of the following: Non-temporary computer-readable media as described in Appendix 14. (Note 16) The aforementioned machine learning model is a logistic regression classifier, The step of generating the score includes inputting the metric value into the logistic regression classifier and applying the output of the logistic regression classifier to the mapping of the classifier result to the skill score, wherein the mapping of the classifier result to the skill score is generated based on surgical data from a reference population. Non-temporary computer-readable media as described in Appendix 14. (Note 17) The surgical data includes one or more of the following: patient-side kinematic data, surgeon-side kinematic data, visualization tool video, and system events. Non-temporary computer-readable media as described in Appendix 14 or 16. (Note 18) The metric was selected, at least in part, using a single OPI statistical distribution analysis filter. Non-temporary computer-readable media as described in Appendix 14. (Note 19) The metric was selected, at least in part, using multiple OPI statistical distribution analysis filters. Non-temporary computer-readable media as described in Appendix 14. (Note 20) The metric was selected, at least in part, using multiple OPI prediction model filters. Non-temporary computer-readable media as described in Appendix 14. (Note 21) The metric was selected by cross-validation during the training of the machine learning model. Non-temporary computer-readable media as described in Appendix 14, 18, 19, or 20. (Note 22) The above method further: The step includes presenting the score as one of several scores generated throughout the process of a surgeon's surgical performance, Non-temporary computer-readable media as described in Appendix 14. (Note 23) The score mentioned above is a skill score, and the machine learning model mentioned above is a skill model. Non-temporary computer-readable media as described in Appendix 22. (Note 24) The score mentioned above is a task score, and the machine learning model is a task model. Non-temporary computer-readable media as described in Appendix 22. (Note 25) Computer system: at least one processor; and Having at least one memory, the at least one memory includes instructions configured to cause the computer system to perform a method, the method being: Steps to obtain surgical data; A step of converting the aforementioned surgical data into multiple metric values; and The steps include: generating scores related to the surgical data using a machine learning model, at least partially; Computer system. (Note 26) The steps for generating the score are: The steps of providing the metric values ​​to the machine learning model in order to generate predictions; and The step of mapping the aforementioned prediction to a score; The computer system described in Appendix 25. (Note 27) The aforementioned machine learning model is: Neural network; Support vector machine; Random Forest; Logistic regression classifier; and Logic configured to predict classification based on data patterns; Including one or more of the following: The computer system described in Appendix 26. (Note 28) The aforementioned machine learning model is a logistic regression classifier, The step of generating the score includes inputting the metric value into the logistic regression classifier and applying the output of the logistic regression classifier to the mapping of the classifier result to the skill score, wherein the mapping of the classifier result to the skill score is generated based on surgical data from a reference population. The computer system described in Appendix 26. (Note 29) The surgical data includes one or more of the following: patient-side kinematic data, surgeon-side kinematic data, visualization tool video, and system events. The computer system described in Appendix 26 or 28. (Note 30) The metric was selected, at least in part, using a single OPI statistical distribution analysis filter. The computer system described in Appendix 26. (Note 31) The metric was selected, at least in part, using multiple OPI statistical distribution analysis filters. The computer system according to Supplementary Note 26. (Supplementary Note 32) The metric is selected, at least in part, using a plurality of OPI prediction model filters, The computer system according to Supplementary Note 26. (Supplementary Note 33) The metric is selected by cross-validation during training of the machine learning model, The computer system according to Supplementary Note 26, 30, 31, or 32. (Supplementary Note 34) The method further comprises: comprising the step of presenting said score as one of a plurality of scores generated over the course of a surgeon's surgical performance, The computer system according to Supplementary Note 26. (Supplementary Note 35) The score is a skill score, and the machine learning model is a skill model, The computer system according to Supplementary Note 34. (Supplementary Note 36) The score is a task score, and the machine learning model is a task model, The computer system according to Supplementary Note 34. (Supplementary Note 37) A computer-implemented method for selecting a surgical evaluation metric, the method comprising: obtaining a first corpus of surgical data metric types; applying a first filter to the first corpus of surgical data metric types to generate a second corpus of surgical data metric types, wherein the second corpus is a subset of the first corpus; applying a second filter to the first corpus to generate a third corpus of surgical data metric types; taking the union of the second corpus and the third corpus to generate a fourth corpus of surgical data metric types; generating a plurality of metric values from surgical data of the surgical data metric types of the fourth corpus; and using the plurality of metric values to train a machine learning model for distinguishing metric values of surgical data between experts and non-experts; A computer-implemented method. (Supplementary Note 38) The first corpus of surgical data metric types is an objective performance indicator (OPI) metric type, the second corpus of surgical data metric types is an OPI metric type, the third corpus of surgical data metric types is an OPI metric type, the fourth corpus of surgical data metric types is an OPI metric type, the first filter is one of a statistical distribution analysis (SOSDA) filter and a multi-OPI statistical distribution analysis (MOSDA) filter, the second filter is a multi-OPI prediction model (MOPM) filter, The computer-implemented method according to Supplementary Note 37. (Supplementary Note 39) The first filter comprises comparing OPI expert distributions and non-expert distributions by Mann-Whitney U test, and the second filter comprises recursive feature elimination, The computer-implemented method according to Supplementary Note 38. (Supplementary Note 40) The score is one of a skill score and a task score, and the machine learning model is one of a skill model and a task model, The computer-implemented method according to Supplementary Note 39. (Supplementary Note 41) A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, wherein the method comprises: obtaining a first corpus of surgical data metric types; applying a first filter to the first corpus of surgical data metric types to generate a second corpus of surgical data metric types, wherein the second corpus is a subset of the first corpus; A step of applying a second filter to the first corpus to generate a third corpus of surgical data metric types; The step of generating a fourth corpus of surgical data metric types by taking the union of the second corpus and the third corpus; The steps of generating multiple metric values ​​from surgical data of the surgical data metric type in the fourth corpus; and The steps include: training a machine learning model to distinguish between metric values ​​of surgical data from experts and non-experts using the aforementioned multiple metric values; Non-temporary computer-readable media. (Note 42) The first corpus of the surgical data metric type is an objective performance indicator (OPI) metric type, The second corpus of the aforementioned surgical data metric type is the OPI metric type, The third corpus of the aforementioned surgical data metric type is the OPI metric type, The fourth corpus of the aforementioned surgical data metric type is the OPI metric type, The first filter described above is one of the following: a statistical distribution analysis (SOSDA) filter and a multi-OPI statistical distribution analysis (MOSDA) filter. The second filter mentioned above is a Multi-OPI Prediction Model (MOPM) filter. Non-temporary computer-readable media as described in Appendix 41. (Note 43) The first filter includes a comparison of the expert and non-expert distributions of OPI using the Mann-Whitney U test, and the second filter includes recursive feature erasure. Non-temporary computer-readable media as described in Appendix 42. (Note 44) The score is one of the skill score and the task score, and the machine learning model is one of the skill model and the task model. Non-temporary computer-readable media as described in Appendix 43. (Note 45) A computer system: at least one processor; and Having at least one memory, the at least one memory includes instructions configured to cause the computer system to perform a method, the method being: Steps to obtain the first corpus of surgical data metric types; A step of generating a second corpus of surgical data metrics by applying a first filter to a first corpus of surgical data metrics type, wherein the second corpus is a subset of the first corpus; A step of applying a second filter to the first corpus to generate a third corpus of surgical data metric types; The step of generating a fourth corpus of surgical data metric types by taking the union of the second corpus and the third corpus; The steps of generating multiple metric values ​​from surgical data of the surgical data metric type in the fourth corpus; and The steps include: training a machine learning model to distinguish between metric values ​​of surgical data from experts and non-experts using the aforementioned multiple metric values; Computer system. (Note 46) The first corpus of the surgical data metric type is an objective performance indicator (OPI) metric type, The second corpus of the aforementioned surgical data metric type is the OPI metric type, The third corpus of the aforementioned surgical data metric type is the OPI metric type, The fourth corpus of the aforementioned surgical data metric type is the OPI metric type, The first filter described above is one of the following: a statistical distribution analysis (SOSDA) filter and a multi-OPI statistical distribution analysis (MOSDA) filter. The second filter mentioned above is a Multi-OPI Prediction Model (MOPM) filter. The computer system described in Appendix 45. (Note 47) The first filter includes a comparison of the expert and non-expert distributions of OPI using the Mann-Whitney U test, and the second filter includes recursive feature erasure. The computer system described in Appendix 46. (Note 48) The score is one of the skill score and the task score, and the machine learning model is one of the skill model and the task model. The computer system described in Appendix 47.

Claims

[Claim 1] A computer-implemented method for generating a score based on surgical data, wherein the method is: Steps to obtain surgical data; A step of converting the surgical data into multiple metric values; and The steps include: generating scores related to the surgical data using a machine learning model, at least partially; A computer-implemented method.

Images