Methods for Artificial Intelligence (AI) Model Selection

Abstract

A computational method and system for training artificial intelligence (AI) models with improved translatability or generalizability (robustness) includes training multiple artificial intelligence (AI) models using a common validation data set over multiple epochs. During training of each model, at least one reliability metric is calculated over one or more epochs, and for each model, the best reliability metric value over the multiple epochs and the associated epoch number for the best reliability metric are stored. An AI model is then generated by selecting at least one of the multiple trained AI models based on the stored best reliability metric and calculating a reliability metric for the selected at least one trained AI model that is applied to a blind test set. The resulting AI model is saved and deployed if the best reliability metric exceeds an acceptance threshold.

Description

[Technical Field]

[0001] [Priority document] This application claims priority from Australian Provisional Patent Application No. 2020901042, entitled "METHOD FOR ARTIFICIAL INTELLIGENCE (AI) MODEL SELECTION", filed on April 3, 2020, the contents of which are incorporated herein by reference in their entirety.

[0002] [Technical field] FIELD OF THE DISCLOSURE The present disclosure relates to artificial intelligence. In a particular aspect, the present disclosure relates to a method for training an AI model and classifying data. [Background technology]

[0003] Advances in artificial intelligence (AI) are enabling the development of new products that are reshaping businesses and the future of many important industries, including healthcare. Underlying these changes is the rapid growth of machine learning and deep learning (DL) technologies.

[0004] Both machine learning and deep learning are two subsets of artificial intelligence (AI). Machine learning is a technique or algorithm that enables a machine to self-learn a task (e.g., create a predictive model) without human intervention or being explicitly programmed. Supervised machine learning (or supervised learning) is a classification technique that learns patterns from labeled (training) data, where the label or annotation of each data point is associated with a set of classes, to create a (predictive) AI model that can be used to classify new, unknown data. In the context of this specification, AI is used to refer to both machine learning and deep learning methods.

[0005] Using embryo viability identification in IVF as an example, an image of an embryo may be labeled "viable" if the embryo resulted in a pregnancy (viable class) or non-viable if the embryo did not result in a pregnancy (non-viable class). Supervised learning can be used to train a large dataset of labeled embryo images to learn patterns associated with viable and non-viable embryos. These patterns are incorporated into an AI model. The AI ​​model can then be used to classify new, unknown images to identify (via inference on the embryo image) whether the embryo is likely to be viable (i.e., likely to result in a pregnancy and therefore a candidate for implantation into a patient in IVF treatment) or non-viable (i.e., unlikely to result in a pregnancy and therefore should not be implanted into a patient).

[0006] Deep learning is similar to machine learning in terms of learning objectives, but goes beyond statistical machine learning models to better mimic the functioning of the human nervous system. Deep learning models typically consist of an artificial "neural network" containing multiple intermediate layers between input and output, with each layer considered a submodel and providing a different interpretation of the data. While machine learning generally accepts only structured data as its input, deep learning does not necessarily require structured data as its input. For example, to recognize images of dogs and cats, traditional machine learning models require pre-defined user features from those images. Such machine learning models learn from specific numerical features as input and can then be used to identify features or objects from other, unknown images. Raw images are sent layer by layer through a deep learning network, with each layer learning to define specific (numerical) features of the input image.

[0007] To train an AI model (including machine learning and / or deep learning models), the following steps are typically performed: a) exploring data in the context of the problem domain and the desired AI solution or application, which may include identifying what type of problem is being solved, e.g., a classification problem or a segmentation problem, and then precisely defining the problem to be solved, e.g., precisely defining which subset of data should be used to train the model and into which category the model will output results; b) Pre-processing the data, including data quality techniques / data cleaning to remove any label noise or bad data, to prepare the data ready to be used for AI training and validation; c) extracting features (e.g., by using computer vision methods) if required by the model; d) selecting a model configuration, including model type, model architecture, and machine learning hyperparameters; e) splitting the data into a training data set, a validation data set and / or a test data set; f) Training an AI model by using machine learning and / or deep learning algorithms on a training dataset. Typically, during the training process, many models are generated by adjusting and tuning the model configuration to optimize the model's performance according to a accuracy metric. Each training iteration is called an epoch. At the end of each epoch, the accuracy is estimated and the model is updated. g) Selecting the best “final” model or ensemble of models based on the model’s performance on the validation dataset, which is then applied to an “unknown” test dataset to validate the performance of the final AI model.

[0008] Machine learning or deep learning algorithms discover patterns in training data and map them to targets. The resulting trained model from this process is then able to capture these patterns.

[0009] As AI-driven technologies become more widespread, the demand for high-quality (e.g., accurate) AI predictive models has become more evident. The current state of the literature on machine learning applications for image classification (the field of computer vision) has primarily focused on accuracy, measured by dividing the total number of correctly identified images that fall into those categories by the total number of images in a blind test set. While accuracy is a useful metric for establishing model performance in certain cases where the dataset is extremely large (tens to hundreds of thousands of images) and the problem set is very well-contained, many commercial applications of machine learning suffer from challenges such as a lack of generalizability (i.e., robustness) required for AI to scale and apply globally to different and diverse users, or a failure to translate to data sourced from real-world industrial datasets.

[0010] One reason for this discrepancy between performance on standard artificial and well-curated datasets and real-world industrial performance is that models tend to be "weak" or unable to generalize (or translate) from the set on which they were trained to datasets that extend outside their narrow, constrained domain of applicability. Dataset characteristics, such as handling bad data, poorly labeled or misleading data, and adversarial examples, while studied in the literature, are typically not featured in major computer vision competitions (e.g., Kaggle) and / or their industry-specific counterparts, and therefore many techniques are not typically implemented as part of protocols for training, validating, and testing robust, scalable AI models (required for commercially scalable AI products) and learning which metrics are most appropriate.

[0011] This is especially true in certain industries, such as healthcare / medical imaging datasets, which differ in several ways from other well-studied computer vision datasets. First, medical images can contain important very fine information associated with features within the image, and the distribution of this information can differ from standard image datasets. This means that while transfer learning is a useful technique that has shown many benefits for medical applications, it is not sufficient on its own; retraining on a new medical training set (specific to the problem at hand) must be accomplished on the medical dataset before predictive power can be demonstrated.

[0012] Second, high-quality, well-labeled medical data is typically much scarcer than other types of imaging data, meaning that using a single coarse metric such as accuracy is potentially susceptible to either a) large statistical uncertainty due to the small validation and test sets available for reporting the metric, and / or b) the details of the distribution of the model's output, i.e., the strong dependence of model performance on the score for classifying images. This scarcity of high-quality, well-labeled medical data means that greater care must be taken in understanding the distribution of a model's output, its prediction score, and whether the distribution is good. It also requires care in understanding other important metrics that may prove to be better indicators of translatability to new blind (unknown) datasets, or double-blind datasets (blind datasets sourced from clinics, localities, or datasets with different sources or distributions than the training and validation sets).

[0013] Focusing on accuracy as the single metric that defines the performance of an AI model in the field, at the expense of all other metrics, can have adverse effects, as such AI models or AI products often fail to generalize well to new datasets and therefore produce poor decision-making outcomes when used in practice.

[0014] There is therefore a need to provide methods for generating AI models that perform well on new datasets (i.e., generalize well), or at least provide useful alternatives to existing methods. Summary of the Invention

[0015] 1. A computational method for generating an artificial intelligence (AI) model, the method comprising: training a plurality of artificial intelligence (AI) models using a common validation data set over a plurality of epochs, wherein during the training of each model, at least one reliability metric is calculated at one or more epochs, and for each model, the best reliability metric value over the plurality of epochs and the associated epoch number for the best reliability metric are stored; generating an AI model, selecting at least one of the plurality of trained AI models based on the stored best reliability metric; calculating a reliability metric for the selected at least one trained AI model applied to a blind test set; and If the best reliability metric exceeds an acceptable threshold, deploying the AI ​​model. Includes.

[0016] In a further aspect, at least one reliability metric is calculated at each epoch.

[0017] In one form, generating the AI ​​model includes generating an ensemble AI model using at least two of the plurality of trained AI models based on the stored best reliability metric, wherein the ensemble model uses a reliability-based voting strategy.

[0018] In a further aspect, the step of generating the ensemble AI model comprises: selecting at least two of the plurality of trained AI models based on the stored best reliability metric; generating a plurality of distinct candidate ensemble models, each candidate ensemble model combining results of at least two selected trained AI models from the plurality of trained AI models according to a confidence-based voting strategy; calculating a reliability metric for each candidate ensemble model applied to a common ensemble validation dataset; selecting a candidate ensemble model from the plurality of distinct candidate ensemble models and calculating a reliability metric for the selected candidate ensemble model applied to the blind test set; Includes.

[0019] In one form, the common ensemble validation data set may be a common validation data set, or the common ensemble validation data set may be an intermediate test set that is not used in training the multiple artificial intelligence (AI) models.

[0020] In one form, the confidence-based voting strategy may be selected from the group consisting of maximum confidence, mean confidence, majority-mean confidence, majority-max confidence, median confidence, or weighted mean confidence.

[0021] In one form, generating the AI ​​model includes generating the student AI model using a distillation method to train the student model using at least two of the plurality of trained AI models using at least one reliability metric.

[0022] In one form, selecting at least one of the plurality of trained AI models based on the stored best reliability metric includes selecting at least two of the plurality of trained AI models, comparing each of the at least two of the plurality of trained AI models using the reliability-based metric, and selecting the best trained AI model based on the comparison.

[0023] In one form, the at least one reliability metric includes one or more of a log-loss, a combined class log-loss, a combined data source log-loss, and a combined class and data source log-loss.

[0024] In one embodiment, multiple evaluation metrics are calculated, including accuracy, mean class accuracy, sensitivity, specificity, confusion matrix, sensitivity-to-specificity ratio, precision, negative predictive value, balanced accuracy, log loss, combined class log loss, combined data-source log loss, combined class and data-source log loss, tangent score, bounded tangent score, per-class ratio of tangent score vs log loss, sigmoid score, epoch number, mean of square error (MSE), and root MSE. MSE), mean of average error, mean average precision (mAP), confidence score, Area-Under-the-Curve (AUC) threshold, Receiver Operating Characteristic (ROC) curve threshold, Precision-Recall curve. In a further aspect, the plurality of evaluation metrics includes a primary metric and at least one secondary metric, wherein the primary metric is a confidence metric and the at least one secondary metric is used as a tie-breaker metric.

[0025] In one embodiment, the plurality of AI models includes a plurality of separate model configurations, each model configuration including a model type, a model architecture, and one or more preprocessing methods. In a further embodiment, the one or more preprocessing methods may include segmentation, and the plurality of AI models includes at least one AI model applied to unsegmented images and at least one AI model applied to segmented images. In another embodiment, the one or more preprocessing methods may include one or more computer vision preprocessing methods.

[0026] Embodiments of the method may be used in healthcare applications, and therefore in one form the validation dataset is a healthcare dataset consisting of a plurality of healthcare images.

[0027] According to a second aspect, there is provided a computing system comprising one or more processors, one or more memories, and a communications interface, wherein the one or more memories store instructions for configuring the one or more processors to computationally generate an artificial intelligence (AI) model according to the method of the first aspect. The computing system may be a cloud-based system. According to a third aspect, there is provided a computing system comprising one or more processors, one or more memories, and a communications interface, wherein the one or more memories are configured to store an AI model trained using the method of the first aspect, the one or more processors are configured to receive input data via the communications interface and process the input data using the stored AI model to generate model results, and the communications interface is configured to transmit the model results to a user interface or a data storage device. [Brief explanation of the drawings]

[0028] Embodiments of the present disclosure will be described with reference to the accompanying drawings. [Figure 1A] 1 is a schematic flowchart of generating an artificial intelligence (AI) model, according to one embodiment. [Figure 1B] 1 is a schematic flowchart of generating an ensemble artificial intelligence (AI) model, according to one embodiment. [Figure 2A] FIG. 1 is a schematic architectural diagram of a cloud-based computing system configured to generate and use AI models, according to one embodiment. [Figure 2B] 1 is a schematic flowchart of a model training process on a training server, according to one embodiment. [Figure 3] For C, we show the scores and score gradients for metrics such as accuracy, log loss, tangent score, and sigmoid score, which provide a measure of the marginal sensitivity of the various metrics. [Figure 4A] A histogram plot associated with the distribution of scores using recall as the primary metric for positively pregnant (viable) embryos of a single machine learning model in the validation set, where bars with a thick forward diagonal line are correct model predictions (true positives) and bars with a thin backward diagonal line are incorrect model predictions (false negatives). [Figure 4B] Plot of a histogram associated with the distribution of scores using recall as the primary metric for negatively pregnant (non-viable) embryos of a single machine learning model in the validation set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 4C] Plot of a histogram associated with the distribution of scores using recall as the primary metric for positively pregnant (viable) embryos of a single machine learning model on the combined blind / double-blind test set, where bars with a thick forward diagonal line are correct model predictions (true positives) and bars with a thin backward diagonal line are incorrect model predictions (false negatives). [Figure 4D]Plot of a histogram associated with the distribution of scores using recall as the primary metric for negatively pregnant (non-viable) embryos for a single machine learning model on the combined blind / double-blind test set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 5A] Plot of a histogram associated with the distribution of scores of positively pregnant (viable) embryos for the ensemble models selected based on the average accuracy rate in the shared validation set, where green is the correct model prediction (true positive) and bars with a thin backward diagonal line are the incorrect model prediction (false negative). [Figure 5B] Plot of a histogram associated with the distribution of scores for negatively pregnant (non-viable) embryos of the ensemble model, selected based on the average accuracy rate, in the shared validation set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 5C] 10 is a plot of a histogram associated with the distribution of scores of positively pregnant (viable) embryos for ensemble models selected based on their average accuracy rate on the shared blind test set, where bars with a thick forward diagonal line are correct model predictions (true positives) and bars with a thin backward diagonal line are incorrect model predictions (false negatives). [Figure 5D] Plot of a histogram associated with the distribution of scores for negatively pregnant (non-viable) embryos of the ensemble models selected based on the average accuracy rate on the shared blind test set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 6A] Plot of a histogram associated with the distribution of scores of positively pregnant (viable) embryos of the ensemble model, selected based on log-loss, in the shared validation set, where bars with a thick forward diagonal line are correct model predictions (true positives) and bars with a thin backward diagonal line are incorrect model predictions (false negatives). [Figure 6B] Plot of histogram associated with the distribution of scores of negatively pregnant (non-viable) embryos of the ensemble model, selected based on log-loss, in the shared validation set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 6C] Plot of a histogram associated with the distribution of scores of positively pregnant (viable) embryos for the ensemble model, selected based on log-loss, on the shared blind test set, where bars with a thick forward diagonal line are correct model predictions (true positives) and bars with a thin backward diagonal line are incorrect model predictions (false negatives). [Figure 6D] Plot of histogram associated with the distribution of scores of negatively pregnant (non-viable) embryos of the ensemble model, selected based on log-loss, on the shared blind test set, where bars with a thick forward diagonal line are correct model predictions (true negatives) and bars with a thin backward diagonal line are incorrect model predictions (false positives). [Figure 7A] Plot of a histogram associated with the distribution of scores using class-specific tangent score versus log loss ratio as the primary metric for positively pregnant (viable) embryos of a single machine learning model in the validation set, where bars with horizontal lines are correct model predictions (true positives) and solid bars are incorrect model predictions (false negatives). [Figure 7B] Plot of a histogram associated with the distribution of scores using class-specific tangent score versus log loss ratio as the primary metric for negatively pregnant (non-viable) embryos of a single machine learning model in the validation set, where bars with horizontal lines are correct model predictions (true negatives) and solid bars are incorrect model predictions (false negatives). [Figure 7C]Plot of histograms associated with the distribution of scores using class-specific tangent score versus log loss ratio as the primary metric for positively pregnant (viable) embryos of a single machine learning model on the combined blind / double-blind test set, where bars with horizontal lines are correct model predictions (true positives) and solid bars are incorrect model predictions (false negatives). [Figure 7D] Plot of histograms associated with the distribution of scores using class-specific tangent scores versus log loss ratio as the primary metric for negatively pregnant (non-viable) embryos for a single machine learning model on the combined blind / double-blind test set, where bars with horizontal lines are correct model predictions (true negatives) and solid bars are incorrect model predictions (false negatives).

[0029] In the following description, like reference characters designate like or corresponding parts throughout the figures. DETAILED DESCRIPTION OF THE INVENTION

[0030] Referring now to FIG. 1A, an embodiment of a method for training an AI model using a metric that takes into account confidence rather than just accuracy will be described.

[0031] Most prior art AI training methods focus on the overall accuracy rate or the variance of the overall accuracy rate to judge the performance of an AI model. These may include the accuracy rate of the model for individual classes (of classification categories), i.e., "class accuracy rate," and variations of accuracy rate, such as weighting the accuracy rate by the total number of images in each category or class, i.e., "average accuracy rate." However, the problem with these metrics that focus on accuracy rate is that these quantities do not directly measure the translatability or generalizability of the AI ​​model.

[0032] The embodiments described herein can be used to create well-performing AI models guided by the confidence (or distribution of confidences / scores) that the AI ​​model can correctly classify a particular image / data. While accuracy rates can be calculated and used in the final report, the method incorporates one or more confidence metrics that accurately measure this confidence as an intermediate step in selecting the best AI model from among many potential models prior to reporting. As outlined below, using a performance metric (or simply a metric) that takes confidence into account is more directly useful in establishing the translatability of an AI model.

[0033] FIG. 1 is a schematic flow chart of the generation of an artificial intelligence (AI) model 100 according to one embodiment.

[0034] At 101, multiple artificial intelligence (AI) models are trained using a common validation data set over multiple epochs. During training of each model, at least one reliability metric is calculated over one or more epochs, and for each model, the best reliability metric value over the multiple epochs and the associated epoch number at which it was best reliable are stored. Preferably, the reliability metric is calculated every epoch or every few epochs.

[0035] The at least one reliability metric may include a primary evaluation metric and one or more secondary evaluation metrics. The secondary metric may be used as a tiebreaker metric. In some embodiments, at least one of the metrics is a reliability metric and at least one is an accuracy metric. The metrics may include accuracy rate, average class accuracy rate, sensitivity, specificity, confusion matrix, sensitivity-to-specificity ratio, precision, negative predictive value, average accuracy rate, log-loss, combined class log-loss, combined data source log-loss, combined class and data source log-loss, tangent score, bounded tangent score, class-specific tangent score-to-log-loss ratio, sigmoid score, number of epochs, mean squared error (MSE), root MSE, mean of mean error, mean average precision (mAP), confidence score, area under the curve (AUC) threshold, receiver operating characteristic (ROC) curve threshold, and precision-recall curve. These metrics are further described below.

[0036] The multiple AI models may include multiple separate model configurations. Each model configuration includes a model type (e.g., binary classification, multi-class classification, regression, object detection, etc.) and a model architecture or methodology (machine learning including random forests, support vector machines, clustering; deep learning / convolutional neural networks including ResNet, DenseNet, or InceptionNet, including specific implementations such as different numbers of layers and inter-layer connections, e.g., ResNet-18, ResNet-50, ResNet-101). The concept of separate model configurations also extends to include the use of separate model inputs, hyperparameters, or (where relevant) preprocessing methods such as segmentation. In one embodiment, the AI ​​models may include at least one AI model applied to unsegmented images and at least one AI model applied to segmented images.

[0037] The one or more preprocessing methods may include computer vision preprocessing methods for generating feature descriptors for the images. Computer vision models rely on identifying important features of an image and representing them with descriptors. These descriptors may encode properties such as pixel variability, gray level, texture roughness, fixed corner points, or image gradient orientation, and are implemented in OpenCV or similar libraries. By selecting such features to search for in each image, a model can be built by finding which arrangements of features are good indicators of a desired class (e.g., embryo viability). This step is best performed by a machine learning process, such as a random forest or support vector machine, which can separate images with respect to descriptions from the computer vision analysis.

[0038] Deep learning and neural networks "learn" features rather than relying on manually designed feature descriptors like machine learning models. This allows them to learn "feature representations" tailored to the desired task. These methods are well-suited for image analysis because they can pick up both fine details and overall morphological shapes to perform global classification. Various deep learning models are available, each with different architectures (i.e., different numbers of layers and inter-layer connections), such as residual networks (e.g., ResNet-18, ResNet-50, and ResNet-101), densely connected networks (e.g., DenseNet-121 and DenseNet-161), and other variants (e.g., InceptionV4 and Inception-ResNetV2). Training involves trying different combinations of model parameters and hyperparameters, including input image resolution, optimizer selection, learning rate value and scheduling, momentum value, dropout, and weight initialization (pre-training). A loss function can be defined to evaluate the performance of the model, and during training, the deep learning model is optimized by varying the learning rate to drive the update mechanism for the network's weight parameters and minimize the objective / loss function.

[0039] The multiple trained AI models are then used to generate a final AI model (102). In one embodiment, this includes selecting at least one of the multiple trained AI models based on the stored best reliability metric (103) and calculating a reliability metric for the selected at least one trained AI model to be applied to the blind test set (104). Generating the final AI model (102) may be performed using an ensemble method that uses at least two of the trained AI models based on the stored best reliability metric and a reliability-based voting strategy, a distillation method that uses at least two of the trained AI models to train a student model using at least one reliability metric, or some other selection method, such as selecting at least two of the multiple trained AI models, comparing each of at least two of the multiple trained AI models using a reliability-based metric, and selecting the best trained AI model based on this comparison.

[0040] FIG. 1B is a flowchart of an ensemble model 110 for generating a final AI model (102). Two or more (including all) of the trained AI models are selected for inclusion in the ensemble model (113) based on a reliability metric. Each model is considered only once at its peak performance, and multiple epochs of the same model are not included. To select AI models for inclusion, details may be ranked based on a primary reliability metric. In one embodiment, all models exceeding a threshold are selected for inclusion in the ensemble model. In some embodiments, other selection criteria may be used in addition to the primary reliability metric, such as a secondary metric (reliability-based or accuracy-based) and / or the number of epochs. Additionally or alternatively, models may be selected to ensure that the AI ​​models in the ensemble include a range of different model architectures and computer vision preprocessing or segmentation techniques. That is, if two models have similar model configurations (e.g., architectures) and similar primary metrics, only one is selected as representative of that model configuration.

[0041] The selected AI models are used to generate 114 multiple distinct candidate ensemble models, each of which combines the results of the selected trained AI models into a single result according to a confidence-based voting strategy.

[0042] The voting strategy defines how model scores are combined. When selecting an ensemble, each voting strategy is considered as part of the ensemble model, so that the ensemble model consists of: A collection (or sub-collection) of AI models, and · Voting strategies.

[0043] Voting strategies may include confidence-based strategies such as maximum confidence, average confidence, majority-average confidence, majority-maximum confidence, median confidence, weighted average confidence, and other strategies that combine predictions from multiple models into a single score.

[0044] A reliability metric (and any secondary evaluation metric) is calculated for each candidate ensemble model applied to a common ensemble validation dataset (115). The common ensemble validation dataset may be a common validation dataset or an intermediate test set not used to train the multiple artificial intelligence (AI) models (and different from the final blind test set). The best candidate ensemble model is selected based on the reliability metric for the common ensemble validation dataset (116). The optional secondary metric may be used as a tiebreaker between similar reliability metrics or to help select the best model, for example, if multiple metrics pass an associated threshold, where at least one of the multiple metrics is a reliability metric. Similarly, if for a first model the primary reliability metric is good but the secondary metric is poor, and for a second model the primary reliability metric is smaller but equally good than for the first model, and the secondary metric is also good or at least much better than the secondary metric for the first model, the second model can be selected.

[0045] The best candidate ensemble models are then applied to a blind test set (unmodified, i.e., with the same configuration and hyperparameters), and a reliability metric is calculated and reported. For example, the report may include a distribution of scores associated with the final model, as well as a breakdown of individual data points, classes, and data sources (i.e., for medical applications, a breakdown for each patient, each class, such as viable or non-viable embryos for IVF, and each clinic). This is important because a well-generalized model would be expected to have a high accuracy metric on the blind test set, even if it was not selected using that metric. Selecting a model based on a reliability metric may actually lead to improved performance not only in that metric but also in other metrics, such as accuracy, that are more commonly reported and understandable to people outside the field of AI.

[0046] Then, if the best reliability metric (on the blind test set), e.g., the primary evaluation metric, exceeds an acceptance threshold (e.g., 50%, 70%, 90%, 95%, etc.), an AI ensemble model is deployed for use with the new data set (105). If the model does not reach the threshold, the process can be repeated with new training data or a different distribution of model configurations.

[0047] A model may be defined by its network weights, and deployment may involve exporting these network weights and loading them into a computational system (e.g., a cloud computing platform) to run the final trained AI model 100 on new data. In some embodiments, this may involve exporting or saving a checkpoint file or model file using appropriate functions of the machine learning code / API. A checkpoint file may be a file generated by the machine learning code / library and having a defined format that can be exported and read back (reloaded) using standard functions provided as part of the machine learning code / API (e.g., ModelCheckpoint() and load_weights()). The file format may be sent or copied directly (e.g., via ftp or a similar protocol), or may be serialized and sent using JSON, YAML, or a similar data transfer protocol. In some embodiments, additional model metadata may be exported / saved and sent along with the network weights, such as model accuracy, number of epochs, etc., that can further characterize the model or otherwise assist in building the model on another computational device (e.g., a cloud platform, server, or user computing device).

[0048] The computational generation of AI model 100 can be further understood with reference to Figure 2A, which is a schematic architectural diagram of a cloud-based computing system 1 configured to generate and use AI model 100, according to one embodiment. Referring to Figure 1, the AI ​​model generation method is handled by model monitor 21.

[0049] The model monitor 21 requires that a user 40 provide data (including data items and / or images) and metadata 14 to a data management platform, including a data repository. Data preparation steps are performed, for example, to move or rename data items or images to specific folders and perform preprocessing on any images, such as object detection, segmentation, alpha channel removal, padding, cropping / localization, normalization, and scaling. Feature descriptors can be calculated and augmented images can be generated in advance. However, additional preprocessing, including augmentation, can also be performed during training (i.e., on the fly). Images can also undergo quality assessment so that obviously bad images can be rejected and replacement images can be captured. Data, such as patient records or other clinical data, is processed (prepared) to extract classification results, such as survival or non-survival in binary classification, output classes in multiclass classification, or other outcome measures in the case of non-classification, which are linked or associated with each image or data item for use in training and / or evaluating the AI ​​model. The prepared data is loaded 16 onto a cloud provider (e.g., AWS) template server 28 that has the latest version of the training algorithm. The template server is stored and multiple copies are made across various training server clusters 37 (which can be CPU, GPU, ASIC, FPGA, or TPU (tensor processing unit) based) that form training servers 35.

[0050] The model monitor web server 31 then applies a training server 37 from multiple cloud-based training servers 35 for each job submitted by a user 40. Each training server 35 executes pre-prepared code (from the template server 28) to train an AI model using libraries such as PyTorch, Tensorflow, and may use computer vision libraries such as OpenCV. PyTorch and OpenCV are open-source libraries with low-level commands for building CV machine learning models. The AI ​​models may be deep learning models or machine learning models, including CV-based machine learning models.

[0051] The training server 37 manages the training process. This may include, for example, dividing the data or images into a training set, a validation set, and a blind validation set using a random assignment process. Additionally, during training / validation cycles, the training server 37 may randomize the set of images at the beginning of the cycle so that a different subset of images is analyzed or analyzed in a different order in each cycle. If preprocessing was not previously performed or was incomplete (e.g., during data management), additional preprocessing may be performed, including object detection, segmentation and generation of masked datasets, calculation / estimation of CV feature descriptors, and generation of data augmentation. Preprocessing may also include image padding, normalization, etc., as needed. Similar processing may also be performed on non-image data. That is, preprocessing may be performed before training, during training, or some combination (i.e., distributed preprocessing). The number of running training servers 35 can be managed from a browser interface. As training progresses, logging information about the status of training is recorded to a distributed logging service, such as CloudWatch 60 (62). Metrics are also calculated and information is parsed from the logs and stored in a relational database 36. The model is also periodically saved 51 to a data store (e.g., AWS Simple Storage Service (S3) or similar cloud storage service) 50 so that it can be retrieved and loaded at a later date (e.g., for restart in the event of an outage such as an error). Users 40 are sent email updates 44 on the status of the training server when the training server job is complete or if an error is encountered.

[0052] Within each training cluster 37, several processes take place. When the cluster is launched via web server 31, a script automatically runs, reading prepared images and patient records and initiating the specific Pytorch / OpenCV training code requested (71). Input parameters 28 for model training are supplied by user 40 via browser interface 42 or via a configuration script. The training process 72 is then initiated for the requested model parameters, which can be a lengthy and intensive task. Therefore, to avoid losing progress while training is in progress, logs are periodically saved to a logging (e.g., AWS CloudWatch) service 60 (62), and the current version of the model (in training) is saved to a data (e.g., S3) storage service 51 for later retrieval and use (51). An embodiment of a schematic flowchart of the model training process on the training server is shown in FIG. 3B. With access to various trained AI models on the data storage service, multiple models can be combined using ensemble, distillation, or similar techniques, for example, to incorporate a range of different deep learning models (e.g., PyTorch) and / or target computer vision models (e.g., OpenCV), to generate a robust AI model 100 that is then deployed to the distribution platform 80. As outlined above, a model may be defined by its network weights, and deployment may involve exporting these network weights and loading them into the distribution platform 80 to run the final trained AI model 100 on new data. The distribution platform may be a cloud-based computing system, a server-based computing system, or other computing system, and the same computing system used to train the AI ​​model may be used to deploy the AI ​​model.In some embodiments, the same computing system used to train the AI ​​model may be used to deploy the AI ​​model, so that deployment involves storing the trained AI model, for example, in memory of a web server 31, or exporting the model weights for loading onto a distribution server.

[0053] The delivery platform 80 is a computing system comprising one or more processors 82, one or more memories 84, and a communications interface 86. The memory 84 is configured to store trained AI models, which may be received from the model monitor web server 31 via the communications interface 86 or loaded from a model export stored on an electronic storage device. The processor 82 is configured to receive input data (e.g., images for classification from a user 40) via the communications interface and process the input data using the stored AI models to generate model results (e.g., classifications), and the communications interface 84 is configured to transmit the model results to a user interface 88 or export them to a data storage device or electronic report. The processor is configured to receive the input data and process the input data using the stored trained AI models to generate model results. The communications module 86 is configured to receive the input data and transmit or store the model results. The communications module may receive the input data and communicate with a user interface 88, such as a web application, to display the model results, e.g., classifications, object bounding boxes, segmentation boundaries, etc. The user interface 88 may run on a user computing device and be configured to allow the user(s) 40 to drag and drop data or images directly onto the user interface (or other local application) 88, which may trigger the system to perform any pre-processing of the data or images (if necessary), or pass the data or images to the trained / validated AI model 100 to obtain classification or model results (e.g., object bounding boxes, segmentation boundaries, etc.) that can be immediately returned to the user in a report and / or displayed within the user interface 88.The user interface (or local application) 88 allows users to store data such as images and patient information in a data storage device such as a database, generate various reports on that data, and generate audit reports on an organization's, group's, or specific user's use of the tool, as well as manage billing and user accounts (e.g., create users, delete users, reset passwords, change access levels, etc.). The delivery platform 30 may be cloud-based and may also allow product administrators to access the system to create new customer accounts and users, reset passwords, and access customer / user accounts (including data and screens) to facilitate technical support.

[0054] Various metrics may be used for the primary and secondary evaluation metrics. Accuracy-based metrics include accuracy, mean class accuracy, sensitivity, specificity, confusion matrix, sensitivity-to-specificity ratio, precision, negative predictive value, and mean accuracy, which are typically used for classification model types, and mean squared error (MSE), root MSE, mean of mean errors, and mean average precision (mAP), which are typically used for regression and object detection model types.

[0055] Confidence-based metrics include log-loss, combined class log-loss, combined data source log-loss, combined class and data source log-loss, tangent score, bounded tangent score, class-specific tangent score to log-loss ratio, and sigmoid score. Other metrics include the number of epochs, area under the curve (AUC) threshold, receiver operating characteristic (ROC) curve threshold, and precision-recall curves that indicate stability and transferability.

[0056] These metrics are further described below, however, it should be understood that they are merely representative and that variations and other accuracy or confidence-based metrics may be used.

[0057] Accuracy This metric is defined as the total number of correctly identified data (regardless of class) divided by the total number of data in the set for which the accuracy rate was cited. This is typically a validation set, a blind test set, or a double-blind test set. While this is the most common metric cited in the literature and is appropriate for large, well-curated datasets, it is a poor measure of translatability for real-world industrial datasets, especially when the data comes from a different distribution than the original training and validation sets. Accuracy also suffers as a metric when a model is applied to a highly imbalanced class distribution. That is, in the case of a strong majority and minority class, a high accuracy rate may be achieved by simply predicting only the majority class.

[0058] Mean class accuracy This metric is simply defined as the sum of the accuracy rates for each class divided by the total number of classes. Because each class accuracy rate is expressed as a percentage, a model that performs well overall for a heterogeneous dataset (e.g., where the data is mostly of one class and the model is biased toward that class, such as most embryo images being viable in an embryo dataset) will not score highly for this metric. This provides a quick assessment of whether the model is correctly capturing many examples across each class. In particular, when the total number of examples within each class in the validation or test sets is similar, the performance is often very similar to the average accuracy rate below. For highly imbalanced sample datasets, reporting average class accuracy can be misleading, as it strongly favors models that perform well for smaller classes (i.e., if the model performs unusually well or poorly for smaller classes that have greater statistical variability due to the small amount of data).

[0059] Sensitivity or recall (true positive rate - TPR) Sensitivity, TPR, and recall are synonyms and take the following forms: TPR=TP / (TP+FN) Equation 1 where TP is the total number of true positive examples in the set of measurements (prediction was positive and result was positive) and FN is the total number of false negatives in the set of measurements (prediction was negative and result was positive).

[0060] This quantity represents the model's ability to detect "positive" examples of the classification on which the model was trained, such as embryonic viability, PGT-A aneuploidy, or cancer detection. What constitutes a positive example or class depends on the classification problem on which the model was trained, and different industry problems will demonstrate different levels of utility in focusing on the sensitivity or recall metric. In some cases, it can represent a more reliable indicator of a model's translational success, but only if the model is not highly imbalanced or its class accuracy does not fluctuate widely, and if sensitivity is not significantly affected by label noise, such as in the case of embryonic viability (where label noise is more dominant in the non-viable embryo class). As an example, if a model classifies a high percentage (more than 90%) of embryos as viable and a low percentage (less than 20%) as non-viable, it is a poor indicator of translatability. Therefore, combining this metric with other metrics is useful. In the binary embryo classification example above, this ensures that the model does not a) decrease accuracy on non-viable embryos, or b) luck out and land on a very easily classified set of viable embryos at a particular epoch (which would be misleading about its overall model performance).

[0061] Specificity (true negative rate-TNR) Specificity or TNR takes the form of: TNR=TN / (TN+FP) Equation 2 where TN is the total number of true negative examples in the set of measurements (prediction was negative and result was negative), and FP is the total number of false positives in the set of measurements (prediction was positive and result was negative).

[0062] This quantity represents the model's ability to detect "negative" examples of the classification on which it was trained. For binary classification models, sensitivity and specificity are the only two class-specific accuracy rates available. It is important to examine the class accuracy rates for all classes across the complete set, as well as the breakdown for each distinct data source. In the case of the embryo viability problem above, it is important to look at the non-viability accuracy rates not only for the overall test set, but also for the distinct clinic breakdown of the complete test set. For embryo non-invasive PGT-A models, specificity relates to the euploid class of embryos, and for cancer detection, it relates to non-cancerous samples.

[0063] Confusion matrix A confusion matrix is ​​simply a tabular representation of the total number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) defined above. Note that the calculation of the confusion matrix and each of the four quantities requires the establishment of a threshold. The threshold is the value above which the output from the model (i.e., the predicted score) is considered positive and below which it is considered negative. For binary classification problems such as embryo viability classification, it is common to train models so that the threshold is set at 50% of 100% (i.e., normalized and with equal weighting between the two classes), but this need not be the case. For ensemble models, the overall combined ensemble model may have a different threshold than the individual models that compose it. To establish the best-performing threshold, this procedure should be performed on the validation set to avoid overfitting on the test set. Threshold evaluation methods involve scanning through all possible thresholds, which can take the form of an area under the curve (AUC) or receiver operating characteristic (ROC) curve, or a precision-recall (PR) curve. This metric is explained below.

[0064] Sensitivity-to-specificity ratio Data from several localities may be more difficult to stabilize, but attempting to simultaneously equalize accuracy rates across classes and different localities presents competitive effects that can be difficult to achieve. In some cases, the ratio between class accuracy rates may be preferentially unbalanced, especially when noise or other bad data is not uniformly distributed across the classes to be classified. For embryo viability classification, the sensitivity-to-specificity ratio has been shown to optimally translate to greater than 1. Therefore, the combined metric, the sensitivity-to-specificity ratio, can be defined as sensitivity / specificity and is a useful metric, although its best value depends on the problem being solved.

[0065] Precision rate (positive predictive value - PPV) PPV can take the following forms: PPV=TP / (TP+FP) Equation 3

[0066] This quantity represents the proportion of all positive predictions that are correctly classified. It is often used in conjunction with recall as a way to characterize a model's performance in a way that is less susceptible to bias in highly imbalanced datasets (see graphical information below). It can be calculated directly from the confusion matrix.

[0067] Negative predictive value - NPV NPV can take the form: PPV = TN / (TN+FN) Equation 4

[0068] This quantity represents the proportion of all correctly classified negative predictions and is the counterpart to PPV, which can be calculated directly from the confusion matrix.

[0069] F1-Score: The F1-score is defined as follows: 2*Precision*Recall / (Precision+Recall) Equation 5

[0070] This metric provides a joint metric between precision and recall that is less susceptible to highly imbalanced datasets.

[0071] Average correct answer rate: The average accuracy rate is defined as: (Sensitivity + Specificity) / 2 Equation 6

[0072] This metric is an overall accuracy metric, as an alternative to accuracy defined above, and gives equal weight to specificity and sensitivity.

[0073] (Negative) Log Loss The log-loss for a classification model where predictions are values ​​between 0 and 1 is defined as: -log(C) Equation 7 where C=1-|y prediction -ytarget | provides a measure of the level of "accuracy" of the prediction, where C=1 means that the prediction perfectly matches the target label, and C=0 means that the prediction is completely opposite to the target label.

[0074] Log-loss is the most direct measure of a model's performance in terms of itself, as it relates to the cross-entropy loss function used to optimize the model itself during training. It measures the performance of a classification model where predictions are values ​​between 0 and 1. Log-loss therefore inherently takes into account the uncertainty in the predicted score based on how far it deviates from the correct classification. Log-loss is a class of confidence metric.

[0075] The confidence metric takes into account (1) for each data point, the confidence in predicting its class, which is the distance in the distribution between the score for a correct classification (which should be higher) and the score for an incorrect classification (which should be lower), and (2) the confidence in predicting each class across all classes, which ensures a balanced and high distribution of confidence scores across classes.

[0076] In practice, analysis of well-performing models according to the reliability metric correlates somewhat with models selected based on accuracy (or average accuracy or average class accuracy) metrics. The reliability metric tends to favor results from higher epochs, but often produces similar per-epoch behavior compared to other metrics. This makes sense because it selects models that perform on a highly separated AI score distribution, i.e., there is a clear distinction between correct and incorrect predictions. This does not imply that the model will perform well on images with unexpected features (resolution, color balance) or that the model will behave stably across the breakdown of data sources that make up the complete dataset. However, it is an indication that the model has generalized well at a particular epoch.

[0077] Also, important in selecting a stable model is that the model loss (or other metric) remains consistent and stable over multiple epochs (or to the point of overtraining). To reveal this, graphical (per epoch) information can be considered.

[0078] Log-loss for individual classes (and combined): Combined class log-loss We propose that log-loss can be calculated for each distinct class individually, which can provide distributional information for each category. This is useful when the classes are imbalanced or contain different amounts of noise. In these cases, the log-loss of one class may provide a better indication of generalization than the log-loss of another class. In general, the log-loss associated with the less noisy class provides the best measure of generalization.

[0079] The log-losses of the individual classes can then be summed to give a combined class log-loss, which is different from the total log-loss (as it gives equal weight to each class regardless of the total number of samples represented in each class).

[0080] Log-loss for individual data sources (and combined): Log-loss for combined data sources It is proposed that log-loss can also be calculated for separate data sources individually, providing distributional information for each data source and ensuring that the selected model generalizes well across different (and possibly diverse) data sources and is not biased towards any individual or subset of data sources, which can be a good measure of AI generalization.

[0081] This is also useful when the data sizes among the data sources are disproportionate, or when the sources contain different amounts of noise. In these cases, the logarithmic loss on one data source may provide a better indication of generalization than the logarithmic loss on another class. In general, the logarithmic loss associated with the less noisy data source provides the best measure of generalization.

[0082] The log-losses of the individual data sources can then be summed to give the combined data source log-loss, which is different from the total log-loss (as it gives equal weight to each data source regardless of the total number of samples represented in each data source).

[0083] Log-loss for individual classes and data sources (and combined): Log-loss for combined classes and data sources Considering both class and generalization across (different and diverse) data sources, it is proposed that the combined class log-loss and combined data source log-loss described above be combined to ensure maximum generalizability.

[0084] The log-losses of the individual classes and data sources can then be summed to give the combined class and data source log-loss, which is different from the total log-loss (as it gives equal weight to each data source regardless of the total number of samples represented in each class and data source).

[0085] Tangent Score The tangent score for a classification model, where the prediction is a value between 0 and 1, is defined as:

number

[0086] Bounded Tangent Score One practical adjustment to the tangent score function is to avoid ±∞ runoff scores as C → 0 or C → 1, so that the metric is bounded:

Mathematics

Mathematics

[0087] The tangent score is used to counteract the undesirable trend of logarithmic loss, which unfairly "penalizes" model predictions that are surely wrong by rewarding model predictions that are surely correct. When the argument is asymptotic

Mathematics

[0088] Class-wise tangent score to logarithmic loss ratio When a binary dataset contains incorrect labels in one class, the class-wise tangent score to logarithmic loss ratio metric can balance the undesirable effects of both logarithmic loss (which unfairly penalizes models trained on low-quality data) and tangent score (which can lead to a high probability of false confident predictions in clean classes).

[0089] It is proposed to provide a metric that can counteract the adverse effects of individual metrics by calculating the ratio between the tangent score for unclean classes (classes with a significant label error rate) and the logarithmic loss for clean classes (classes with a very low label error rate). This situation applies only when the label error rate of one class is clearly at a much higher level.

[0090] Using the class-specific tangent score versus log-loss ratio as the primary metric, Figures 3A and 3C show histograms of the ratio, ranging from 0.0 to 1.0, with a binary threshold of 0.5 for live embryos (indicated by the vertical dashed line). Correctly classified embryos are shown as bars with a thick horizontal line (true positives) 32, and incorrectly classified embryos are shown as black columns (false negatives) 31. Figures 3B and 3D show equivalent histograms for non-live embryos, where correctly classified embryos are shown as bars with a horizontal line (true negatives) 34, and incorrectly classified embryos are shown as bars with a thick backward diagonal line (false positives) 33.

[0091] Sigmoid score The sigmoid score for a classification model, where the prediction is a value between 0 and 1, is defined as:

number

[0092] The sigmoid score is a "soft" alternative to other accuracy metrics in that it provides a gradual measure of model performance rather than a sharp cutoff.

[0093] Score gradient (also known as threshold sensitivity): Figure 3 shows the scores and score gradients for the metrics accuracy, log loss, tangent score, and sigmoid score against C, illustrating the critical sensitivity of various metrics. Depending on the particular problem and underlying data distribution (or suspect distribution), an appropriate confidence-based metric (i.e., the one that best fits the data) can be selected.

[0094] A variety of other model selection criteria may also be used.

[0095] Number of epochs A very coarse measure of a model's performance during training is the number of passes (i.e., epochs) it achieves through the training set. While this information does not provide richer analysis and insight into the balance between classes or the distribution of predicted scores obtained from the model that other metrics can provide, it nevertheless provides high-level information about the model, namely, whether it has converged, i.e., whether it has reached a steady state where continued training of the model will not lead to improvement. This is related to the graphical representation of loss on the training and validation sets, which is explained more fully below. Furthermore, models trained to higher epochs are also more likely to have been exposed to all available data augmentation available in the training process and are more likely to be confident in their predictions (i.e., their distribution of predicted scores includes more reliable examples). Models trained to extremely high epochs may also exhibit a loss of generality due to overtraining. Therefore, this metric should only be used as a very coarse measure.

[0096] Metrics for Non-Classification Models Metrics obtained from confusion matrices and other related accuracy measures are typically used for (binary) classification problems, but there are other types of models that can use different metrics. Some of these other metrics are mean squared error (MSE), root MSE, mean of average errors, mAP, and confidence scores, which are used in regression and object detection models.

[0097] Graphical Information Graphical information about the training process that took place, such as plots describing the loss (for both the training and validation sets) as a function of epochs, is useful for determining whether the model a) systematically improved its loss over a range of epochs and thus learned information, b) converged to a steady state, and c) is not overtraining (i.e., the training loss continues to improve while the validation loss worsens).

[0098] The distribution of scores at each epoch, displayed as a histogram or other plot style to visualize the distribution, can provide an indication of model performance. For example, if the distribution of predicted scores from a model attempting to solve a binary classification problem is bimodal and the modes are well separated, this is an indication of translatability. However, if the distribution is Gaussian-like, the majority of scores will be clustered around a decision threshold and likely will not perform better than random chance and therefore will not generalize well to unknown datasets, so the probability of correct classifications being higher than incorrect classifications may be weak.

[0099] The area under the curve (AUC) or receiver operating characteristic (ROC) curve is a common visualization tool for identifying a model's decision threshold, i.e., the threshold above which a prediction score is considered predictive of survival and below which it is considered predictive of non-survival (in the case of binary classification problems). It is constructed by plotting the TPR against the FPR. ROC curves are also useful for visually assessing whether the best threshold for a given model has significant predictive power compared to chance. However, in the case of highly imbalanced datasets, they may be considered unreliable.

[0100] Precision-recall curves are often recommended for highly imbalanced datasets to avoid the total number of true negatives in the calculation of recall or accuracy. For example, the precision-recall curve should remain roughly constant as the ratio of negative to positive results changes.

[0101] To further illustrate the method, we consider its application to the development of a binary embryo viability classification model for selecting embryos for transfer in IVF procedures. The dataset consists of 2D static optical microscope images of day 5 blastocyst embryos. Three case studies using different metrics are presented here. Various top-performing models are obtained based on the initial accuracy rate and / or other metrics and compared with the top-performing models based on a reliability metric. The models are then applied to a blind test set reserved for these experimental comparisons to assess whether there are differences in the robustness / generalizability of the models when transferred to a new dataset. Whether the metric should be used alone, with other metrics, or not at all was also empirically investigated and tested.

[0102] We begin by comparing various metrics in terms of generalization and consistency over multiple epochs, and then focus on model selection in terms of log-loss, the preferred metric for this problem.

[0103] Additional secondary measures related to this issue include: · Average correct answer rate; · Sensitivity-specificity ratio; Log-loss for individual classes (e.g., non-survival and survival); epoch number; and Any unused primary metric of the reliability score

[0104] In the case of the sensitivity-to-specificity ratio, the range of models should be chosen such that this metric varies with the models being ensembled, in order to provide a robust ensemble that includes models with different biases towards different subpopulations of embryos.

[0105] In the case of the number of epochs, the intention is to avoid models during training that accidentally performed well according to the first-order metric without having enough time to fully utilize training methods that require many epochs (e.g., augmentation). Therefore, a minimum number of epochs is specified (i.e., the minimum epoch threshold) to prevent these cases from contributing to the ensemble.

[0106] The dataset for these embodiments consists of 3,987 images from seven distinct clinical areas encompassing a total of 11 sites. Viability was assessed based on detection of a fetal heartbeat on the first ultrasound scan after implantation (typically 6-8 weeks).

[0107] For brevity, the names of the clinic datasets are denoted as Clinic Data 1, Clinic Data 2, etc. Table 1 summarizes the class sizes (total number of non-survival or survival images) and total sizes of the seven clinic datasets, and shows that the class distributions differ significantly among the datasets. There are a total of 3,987 images for model training and evaluation purposes. [Table 1]

[0108] Metric comparison, generalization, and consistency. Model selection according to a particular metric, as measured on a validation set, can be evaluated by examining the consistency of the particular selection metric across validation and test sets, the generalization of the model with respect to mean accuracy (i.e., does the model accuracy generalize well for a given selection metric that may not be the mean accuracy), and the distribution of scores as displayed by a histogram.

[0109] In Table 2 below, results for mean accuracy values ​​are presented for several trained AI models, each selected from a large cohort of models with distinct model configurations that include different training parameters and use different primary selection metrics. AI models that use mean class accuracy and mean accuracy as primary metrics have been found to typically arrive at similar trained AI models and epochs. The mean accuracy on the validation set is high for this problem (67.6%), but drops significantly for the test set (58%), indicating that the model does not generalize well to the test (blind) dataset (a double-blind dataset, i.e., containing data from a separate data source where none of the data was used for training) when translated to the validation set, and these metrics are by no means the best to use in model selection.

[0110] In the case of log-loss as a selection metric (confidence-based metric), the average accuracy rate on the validation set is lower than that of the accuracy metric, but the average accuracy rate measured on the test set shows improvement. Further investigation into log-loss, the confidence metric below, will reveal that it is the most reliable metric for generalization and therefore model selection. Recall presents the opposite challenge: a model that performs poorly in terms of average accuracy on the validation set can perform significantly better in terms of average accuracy on the test set. This particular feature is unique to the embryo viability problem: recall (or classifying viable embryos) represents a dataset with less label noise, whereas a dataset of non-viable embryos contains significantly more label noise. While we focus here on the effectiveness of the selection metric, recall (effectively ignoring the accuracy rate of non-viable embryos) cannot be used alone as a selection metric, as it is susceptible to models that classify positive examples with 100% accuracy but have low accuracy on negative examples. Nevertheless, recall represents an important selection metric and is important to consider as the primary selection metric for this problem, while precision acts as a selection metric similar to other accuracy measures. [Table 2]

[0111] Using recall as the primary selection metric, the distribution of scores extracted from the classification model is examined in Figures 4A and 4B for the validation set and in Figures 4C and 4D for the test set. Figures 4A and 4C show histograms of scores ranging from 0.0 to 1.0, with a binary threshold of 0.5, for live embryos (indicated by the vertical dashed line). Correctly classified embryos are color-coded as bars with a thick forward diagonal line (true positives) 42, and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false negatives) 41. Figures 4B and 4D show equivalent histograms for non-live embryos, where correctly classified embryos are color-coded as bars with a thick forward diagonal line (true negatives) 44, and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false positives) 43.

[0112] Note that the test set contains a clinic distribution containing both blind and double-blind test examples (here, the double-blind data is sourced from clinics not represented in the training or validation sets, and therefore the data distribution is different). While performance with this model is biased toward live embryos in the validation set, this is an inherent characteristic of focusing on recall as a metric; comparing Figures 4A and 4B reveals that the distribution of scores on the test set is not well-defined. With a single Gaussian-like (unimodal) distribution around the 0.5 threshold, the model's high performance in terms of mean accuracy is more likely due to chance and is unlikely to generalize to new double-blind sets.

[0113] A similar comparison can be made between Figure 4B and Figure 4D, showing that the distributions in the validation set that are not well separated are unlikely to yield strong generalization, as they remain poorly separated in the test set.

[0114] Metrics for ensemble models with constituent AI models selected based on average accuracy. In this section, trained AI models are selected for inclusion in the ensemble based on average accuracy in shared validation as the primary metric. The best-performing models (based on average accuracy) were selected and the candidate ensemble was combined using a majority-average confidence voting strategy. A class-specific breakdown of model performance associated with these metrics is also considered.

[0115] A shared validation set of 252 images from which the ensemble model components were selected was considered, and this model was then applied to a blind test set of 527 images for comparison.

[0116] A histogram associated with the scores assigned to live embryos by the ensemble model on the shared validation set can be seen in Figure 5A, where correctly classified embryos are color-coded as bars with a thick forward diagonal line (true positives) 52 and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false negatives) 51. Figure 5B shows the equivalent histogram for non-live embryos, where correctly classified embryos are color-coded as bars with a thick forward diagonal line (true negatives) 54 and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false positives) 53. The model distributions are better separated than the single-model case above, due to the fact that ensemble models generally show improved performance on both accuracy measures and generalization compared to single models. This is because multiple models voting on a single image allows more room for model discrepancies to be counteracted or addressed by the broader range of attentional preferences of the constituent models. The details of the inter-model bias are related to the inter-model voting strategy, which together with the components themselves define the ensemble model.

[0117] Note in Figures 5C and 5D that the histograms associated with the validation set results show good separation (bimodal distribution) between correctly identified embryos 52, 54 and incorrectly identified embryos 51, 53, both of which exhibit high accuracy values ​​as measured by TPR and TNR, as described below in the Class Breakdown section. In the blind test set, the separation between correctly and incorrectly identified embryos persists, a sign of generalization. However, as shown in Figure 5D, when a large number of false positives are observed, the accuracy value drops, which reduces the specificity value. This is an inherent problem with noisy datasets; high noise in non-viable embryo datasets contributes to reduced generalization.

[0118] Note the importance of dataset quality (e.g., label quality or accuracy) in demonstrating generalization, as well as how the choice of metrics in selecting a model from a group of ensemble models and selecting ensemble model components are major contributors to the final generalizability or translatability of a model. A breakdown of metrics measuring model performance on validation and test sets is presented in the next section.

[0119] Class Breakdown Table 3 shows the metrics associated with the breakdown of results for two classes of examples, viable and non-viable, for all clinics represented in the combined validation set. While the accuracy measure is high for both classes and establishes a benchmark for the associated log-loss values, Table 4 shows a drop in accuracy for "Class 0" or non-viable embryos when applied to the blind test set, as expected due to label noise. However, accuracy for "Class 1" or viable embryos remains high. Note that although log-loss worsens due to a drop in non-viable accuracy, i.e., specificity, the distributions associated with Figure 5D are still well separated, and log-loss is a more reliable metric for AI generalization because it takes score distribution information into account.

[0120] Class-specific log-losses are also compared and combined, and model selection based on these metrics is found to be consistent with log-loss. [Table 3] [Table 4]

[0121] A metric for the ensemble models selected based on log-loss as the primary metric. Several key metrics are analyzed for the ensemble model, where the constituent model is selected based on the best-performing log-loss from a cohort of trained AI models. This particular ensemble model had a majority-maximum confidence voting strategy. A class-specific breakdown of model performance associated with these metrics is then also considered.

[0122] The same shared validation set of 252 images and blind test set of 527 images are used as in the previous section.

[0123] A histogram associated with the scores assigned by the model to surviving embryos in the shared validation set can be seen in Figure 6A, where correctly classified embryos are color-coded as bars with a thick forward diagonal line (true positives) 62 and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false negatives) 61. Figure 6B shows the equivalent histogram for non-surviving embryos, where correctly classified embryos are color-coded as bars with a thick forward diagonal line (true negatives) 64 and incorrectly classified embryos are color-coded as bars with a thin backward diagonal line (false positives) 63. The model distributions are very well separated, with high values ​​of TPR and TNR. This is due both to the fact that the constituent model is selected based on the metric log-loss, which takes distributional information into account and tends to favor models that exhibit a high degree of separation, and to the fact that the optimal voting strategy for this model is maximum confidence (which also tends to enhance the bimodal aspect of the distribution).

[0124] In the blind test set, the separation between correctly and incorrectly identified embryos persists, and the distribution associated with the less noisy live embryos in Figure 6C matches the validation set in Figure 6B. Again, in this scenario, the Class 0 accuracy or TNR / specificity values ​​decrease, as shown in Figure 6D. Metrics associated with both the validation and blind test sets are considered in the next section on class breakdown.

[0125] Class Breakdown Table 5 shows the metrics associated with the breakdown of results for the two classes of examples, viable and non-viable, for all clinics represented in the combined validation set. While both the accuracy metric and log-loss exceed the values ​​in the previous section on class breakdown (Tables 2 and 3), Table 6, when applied to the blind test set, shows a larger drop in accuracy for "class 0" or non-viable embryos due to label noise, and in terms of accuracy for "class 1" or viable embryos, it exceeds the models in the section on metrics for the ensemble model using constituent AI models selected based on average accuracy. Note, however, that log-loss remains a fair value, albeit worsened by the drop in accuracy for non-viables.

[0126] As above, class-specific log-losses are also compared and combined, and model selection based on these metrics is found to be consistent with log-loss. [Table 5] [Table 6]

[0127] As a further example, Figures 7A-7D show histograms obtained using the class-specific tangent score versus log-loss ratio as the primary metric. Figures 7A and 7C represent histograms of the ratio, ranging from 0.0 to 1.0, with a binary threshold of 0.5 for live embryos (indicated by the vertical dashed line). Correctly classified embryos are shown as bars with a thick horizontal line (true positives) 72, and incorrectly classified embryos are shown as black columns (false negatives) 71. Figures 7B and 7D show equivalent histograms for non-live embryos, where correctly classified embryos are shown as bars with a horizontal line (true negatives) 74, and incorrectly classified embryos are shown as bars with a thick backward diagonal line (false positives) 73. Again, these indicate well-separated model separation, further demonstrating the advantage of a confidence-based metric. These histograms also show that false negatives are minimized by using a logarithmic loss metric in classes (viable embryos) considered less noisy (fewer mislabeled examples), thereby ensuring that the model does not tolerate many false negative examples. A false negative in the case of embryo viability (misclassifying a viable embryo as non-viable) is considered a higher risk misclassification compared to a false positive (misclassifying a non-viable embryo as viable). For false positives, the tangent score metric tolerates a certain amount of noise / misclassified examples if it is offset by a similar number of correctly classified examples with the same confidence. Therefore, classes considered noisier (with many mislabeled examples, such as those that appear non-viable but are actually viable and misclassified due to a patient medical condition outside the embryo image) are less susceptible to misclassifying viable embryos due to noise. Therefore, model training achieves better results during validation and testing because its training phase is more robust to noise.

[0128] As mentioned above, most AI training methods focus on the overall accuracy rate or the variance of the overall accuracy rate to judge model performance. These can include the model's accuracy rate for individual classes (of classification categories), i.e., "class accuracy rate," and variations of accuracy rate, such as weighting accuracy rate by the total number of images in each category or class, i.e., "average accuracy rate." However, the problem with these metrics that focus on accuracy rate is that these quantities do not directly measure the translatability or generalizability of an AI model.

[0129] In contrast, the embodiments described herein can be used to create well-performing AI models guided by both accuracy and confidence (or confidence / score distribution) that the AI ​​model can correctly classify a particular image / data (for final reporting). In particular, the method incorporates one or more metrics that correctly measure this confidence as an intermediate step in selecting the best AI model from among many potential models prior to reporting.

[0130] In particular, the method proposes calculating multiple metrics for various models on the same validation set and using these results to select top performers and / or diverse model configurations in an ensemble model. After selection, the models are applied to a blind or double-blind test set, and their performance on the blind set is evaluated in terms of multiple metrics. Note that well-generalized models are expected to have a high accuracy metric on the blind test set, even if they were not selected using the accuracy metric. Selecting a model based on a different metric may indeed lead to improved performance not only in that metric but also in other metrics, such as accuracy, that are more commonly reported and understandable by those outside the AI ​​field.

[0131] Note that the final reported accuracy rate on the validation or test set may actually be lower for models that perform better than corresponding models overtrained on the distribution of data that the validation or test set was fed. However, when confidence in accurate classification is a consideration, selecting models for commercial use or for combination using ensemble modeling techniques reduces uncertainty and creates more robust models compared to selecting models based on a binary accuracy rate metric where image confidence / score lies on either side of an arbitrary 50% threshold (or more generally, where the correct score for one class is just above the confidence score for the other class). For example, achieving 100% accuracy in correctly classifying 1,000 (blind test) images with an AI score / confidence of 55% (given a 50% threshold for correct classification) may be less valuable than achieving 100% accuracy in correctly classifying 1,000 images with an AI score / confidence of 99.9%.

[0132] As outlined above for selecting AI model components to form the final model, the performance of each model at each training epoch is evaluated on their shared validation set using a primary metric, and then two or more (or all) of the trained AI models are selected for inclusion in the ensemble model based on the best stored primary metric. For example, in the above embodiment where the AI ​​model is a day 5 embryo viability binary classification-based model, the primary metric is log-loss. Apart from the primary metric, other metrics such as information about the training process performed, plots describing the loss per epoch, and the distribution of scores at each epoch are considered, but the primary metric is used as the first metric for sorting the performance of models for selection or as candidates for inclusion in the ensemble model.

[0133] Various embodiments for generating AI models based on a reliability metric have been described. These methods train multiple AI models on a common validation dataset over many epochs. To enable comparison of different AI models, the reliability metric of the best epoch (across all epochs) is saved. These AI models can then be used to select a final AI model, for example, using ensemble, distillation, or other selection methods. For ensemble models, a reliability-based voting strategy can be used. Experimental results show that reliability metrics that consider both the accuracy rate in correctly classifying data and the AI's confidence in correctly classifying data (i.e., a higher AI score for correct classification indicates confidence in the correct classification), such as log-loss and its related metrics (e.g., combined class log-loss, combined data source log-loss, combined class and data source log-loss, tangent score, bounded tangent score, class-specific tangent score to log-loss ratio, and sigmoid score), can result in more accurate and generalizable models that can be applied to diverse contexts, such as healthcare.

[0134] Models incorporating confidence metrics are more robust and more trustworthy because higher confidence in correct classification suggests that the AI ​​model has identified features or correlations more strongly across a wider dataset for each class and data source, making them less susceptible to variability or outliers in new, unknown data.

[0135] It has been demonstrated that models selected using a confidence metric may show a reduced accuracy rate in the validation dataset, but have an overall higher final accuracy rate when applied to a blind (unknown) test set.

[0136] The results presented herein therefore demonstrate that models selected using this methodology exhibit better generalizability and are less prone to overfitting compared to other models trained on the same dataset, thus representing superior models as a result of this selection procedure.

[0137] Embodiments of the methods can be used for healthcare applications (e.g., healthcare data), particularly healthcare datasets that include images captured from a wide range of devices, such as microscopes, cameras, X-rays, MRIs, etc. Models trained using the embodiments described herein can be deployed to assist in making various healthcare decisions, such as fertility and IVF decisions and disease diagnosis. However, it will be understood that these methods can also be used outside of a healthcare environment.

[0138] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0139] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software or instructions, middleware, platforms, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0140] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, including cloud-based systems, in a software module executed by a processor, or in a combination of the two. For hardware implementations, processing may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or other electronic units designed to perform the functions described herein, or a combination thereof. Various middleware and computing platforms may be used.

[0141] In some embodiments, the processor module includes one or more central processing units (CPUs) or graphical processing units (GPUs) configured to execute some of the method steps. Similarly, a computing device may include one or more CPUs and / or GPUs. The CPU may include an input / output interface, an arithmetic logic unit (ALU), and a control unit and program counter element that communicate with input and output devices via the input / output interface. The input / output interface may include a network interface and / or a communication module for communicating with an equivalent communication module in another device using a predefined communication protocol (e.g., Bluetooth, Zigbee, IEEE 802.15, IEEE 802.11, TCP / IP, UDP, etc.). The computing device may include a single CPU (core) or multiple CPUs (multiple cores), or multiple processors. The computing device is typically a cloud-based computing device using a GPU cluster, but may also be a parallel processor, vector processor, or distributed computing device. Memory is operably coupled to the processor(s) and may comprise RAM and ROM components and may be provided internal or external to the device or processor module. The memory may be used to store an operating system and additional software modules or instructions. The processor(s) may be configured to load and execute the software modules or instructions stored in the memory.

[0142] A software module, also known as a computer program, computer code, or instructions, may include several source code or object code segments or instructions and may reside in any computer-readable medium, such as RAM memory, flash memory, ROM memory, EPROM memory, registers, a hard disk, a removable disk, a CD-ROM, a DVD-ROM, a Blu-ray disc, or any other form of computer-readable medium. In some aspects, the computer-readable medium may include a non-transitory computer-readable medium (e.g., a tangible medium). Additionally, in other aspects, the computer-readable medium may include a transitory computer-readable medium (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media. In another aspect, the computer-readable medium may be integrated with the processor. The processor and the computer-readable medium may reside in an ASIC or related device. The software code may be stored in a memory unit, and the processor may be configured to execute them. The memory unit may be implemented inside or outside the processor, in which case it may be communicatively coupled to the processor via various means as is known in the art.

[0143] Furthermore, it should be understood that modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by a computing device. For example, such a device may be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via storage means (e.g., RAM, ROM, physical storage medium such as a compact disk (CD) or floppy disk, etc.) such that a computing device can acquire the various methods upon coupling or providing the storage means to the device. Furthermore, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.

[0144] The methods disclosed herein include one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0145] Throughout this specification and the claims that follow, unless the context requires otherwise, the words "comprise" and "include," and variations such as "comprising" and "including," will be understood to imply the inclusion of a stated integer or group of integers, but not the exclusion of any other integer or group of integers.

[0146] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgment of any form of suggestion that such prior art forms part of the common general knowledge.

[0147] It will be understood by those skilled in the art that the present disclosure is not limited in its use to the particular application or applications described. The present disclosure is also not limited to its preferred embodiments with respect to the particular elements and / or features described or depicted herein. It will be understood that the present disclosure is not limited to the disclosed embodiment or embodiments, but is capable of numerous rearrangements, modifications, and substitutions without departing from the scope described and defined by the following claims.

Claims

1. A computer-implemented method for generating an artificial intelligence (AI) model, the method comprising: training a plurality of artificial intelligence (AI) models with a computing device using a common validation data set over a plurality of epochs, wherein during the training of each model, at least one confidence metric is calculated at one or more epochs, and for each model, a best confidence metric over the plurality of epochs and an associated epoch number for the best confidence metric are stored; generating an AI model, selecting at least one of the plurality of trained AI models based on the stored best reliability metric; calculating a reliability metric for the selected at least one trained AI model applied to a blind test set; and If the best reliability metric exceeds an acceptance threshold, deploying the AI ​​model to the computing device or another computing device. A method comprising:

2. The method of claim 1 , wherein the at least one reliability metric is calculated at each epoch.

3. 3. The method of claim 1, wherein generating an AI model comprises generating an ensemble model using at least two of the plurality of trained AI models based on the stored best reliability metric, and the ensemble model uses a reliability-based voting strategy.

4. The step of generating the ensemble model includes: selecting at least two of the plurality of trained AI models based on the stored best reliability metric; generating a plurality of distinct candidate ensemble models, each candidate ensemble model combining results of the selected at least two trained AI models from the plurality of trained AI models according to a confidence-based voting strategy; calculating the reliability metric for each candidate ensemble model applied to a common ensemble validation dataset; selecting a candidate ensemble model from the plurality of distinct candidate ensemble models and calculating a reliability metric for the selected candidate ensemble model applied to a blind test set; The method of claim 3, comprising:

5. The method of claim 4 , wherein the common ensemble validation data set is the common validation data set.

6. 6. The method of claim 4 or 5, wherein the common ensemble validation data set is an intermediate test set that is not used in training the multiple artificial intelligence (AI) models.

7. 7. The method of claim 4, wherein the reliability-based voting strategy is selected from the group consisting of maximum reliability, average reliability, majority-average reliability, majority-maximum reliability, median reliability, or weighted average reliability.

8. 3. The method of claim 1 or 2, wherein generating an AI model comprises using a distillation method to train a student model using at least two of the plurality of trained AI models using at least one reliability metric.

9. 3. The method of claim 1, wherein selecting at least one of the plurality of trained AI models based on the stored best reliability metric comprises: selecting at least two of the plurality of trained AI models; comparing each of the at least two of the plurality of trained AI models using the reliability metric; and selecting a best trained AI model based on the comparison.

10. 10. The method of claim 1, wherein the at least one reliability metric comprises one or more of a log-loss, a combined class log-loss, a combined data source log-loss, and a combined class and data source log-loss.

11. 11. The method of any one of claims 1 to 10, wherein a plurality of evaluation metrics are calculated and selected from the group consisting of accuracy rate, mean class accuracy rate, sensitivity, specificity, confusion matrix, sensitivity to specificity ratio, precision, negative predictive value, mean accuracy rate, log-loss, combined class log-loss, combined data source log-loss, combined class and data source log-loss, tangent score, bounded tangent score, class-specific tangent score to log-loss ratio, sigmoid score, number of epochs, mean squared error (MSE), root MSE, mean of mean errors, mean average precision (mAP), confidence score, area under the curve (AUC) threshold, receiver operating characteristic (ROC) curve threshold, precision-recall curve.

12. 12. The method of claim 11, wherein the plurality of evaluation metrics includes a primary metric and at least one secondary metric, the primary metric being a confidence metric and the at least one secondary metric being used as a tiebreaker metric.

13. 13. The method of claim 1, wherein the plurality of AI models comprises a plurality of separate model configurations, each model configuration comprising a model type, a model architecture, and one or more pre-processing methods.

14. 14. The method of claim 13, wherein the one or more preprocessing methods include segmentation, and the plurality of AI models includes at least one AI model applied to unsegmented images and at least one AI model applied to segmented images.

15. The method of claim 13 , wherein the one or more preprocessing methods include one or more computer vision preprocessing methods.

16. The method of claim 1 , wherein the validation dataset is a healthcare dataset consisting of a plurality of healthcare images.

17. 17. A computing system comprising one or more processors, one or more memories, and a communications interface, wherein the one or more memories store instructions for configuring the one or more processors to computationally generate an artificial intelligence (AI) model according to a method of any one of claims 1 to 16.

18. 17. A computing system comprising one or more processors, one or more memories, and a communications interface, wherein the one or more memories are configured to store an AI model trained using the method of any one of claims 1 to 16, the one or more processors are configured to receive input data via the communications interface and process the input data using the stored AI model to generate model results, and the communications interface is configured to transmit the model results to a user interface or a data storage device.

Images