AI methods for cleaning data to train artificial intelligence (AI) models

Abstract

A computational method and system for cleaning AI training data is described, which cleans a dataset by splitting the training dataset into multiple training subsets. For each training subset, multiple artificial intelligence (AI) models are trained on two or more of the remaining multiple training subsets, and these trained AI models are used to obtain estimated labels for each sample in the training subset for each AI model. Samples in the training dataset that are consistently incorrectly predicted by the multiple AI models are then removed or relabeled, and then one or more AI models are trained using the cleansed training dataset to generate and deploy a final AI model. A variation of this method is used to label a new dataset, which is inserted into the training dataset, and then the training process itself is used to determine the classification of the new dataset using a voting strategy on the estimated labels.

Description

[Technical field]

[0001] This application claims priority to Australian Provisional Patent Application No. 2020901043, entitled "AI METHOD FOR CLEANING DATA TO TRAIN AN ARTIFICIAL INTELLIGENCE (AI) MODEL", filed on April 3, 2020, the contents of which are incorporated herein by reference in their entirety.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates to artificial intelligence. In a particular aspect, the present disclosure relates to a method for training an AI model and classifying data. [Background technology]

[0003] Advances in artificial intelligence (AI) have enabled the development of new products that are reshaping business and the future of many important industries, including healthcare. Underlying these changes is the rapid growth of machine learning and deep learning (DL) technologies. In the context of this specification, AI is used to refer to both machine learning and deep learning methods.

[0004] Both machine learning and deep learning are two subsets of artificial intelligence (AI). Machine learning is a technique or algorithm that allows a machine to self-learn a task (e.g., create a predictive model) without the need for human intervention or explicit programming. Supervised machine learning (or supervised learning) is a classification technique that learns patterns in labeled (training) data that can be used to classify new, unseen data, where the label or annotation of each data point is associated with a set of classes, to create a (predictive) AI model.

[0005] Using the determination of embryo viability in IVF as an example, images of embryos can be labeled as "viable" if the embryo led to a pregnancy (viable class) and "non-viable" if the embryo did not lead to a pregnancy (non-viable class). Supervised learning can be used to train on large datasets of labeled embryo images to learn patterns associated with viable and non-viable embryos. These patterns are incorporated into an AI model. The AI ​​model can then be used to classify new unseen images (via inference on the embryo images) to identify whether the embryo is likely to be viable (should be transferred to a patient in an IVF treatment) or non-viable (should not be transferred to a patient).

[0006] Deep learning is similar to machine learning in terms of learning objectives, but goes beyond statistical machine learning models to better mimic the functioning of the human nervous system. Deep learning models are usually composed of artificial "neural networks" that contain many intermediate layers between the input and the output, where each layer is considered a sub-model, each providing a different interpretation of the data. Machine learning generally only accepts structured data as input, whereas deep learning, on the other hand, does not necessarily require structured data as its input. For example, to recognize images of dogs and cats, traditional machine learning models require user-predefined features from those images. Such machine learning models learn from specific numerical features as input and can be used to identify features or objects from other unknown images. An original image is sent layer by layer through the deep learning network, and each layer learns to define specific (numerical) features of the input image.

[0007] To train a machine learning model (including a deep learning model), the following steps are typically taken:

[0008] a) Exploring data in the context of the problem domain and the desired AI solution or application. This involves identifying what problem is being solved, e.g. a classification or segmentation problem, and then precisely defining the problem to be solved, e.g. what specific subset of data will be used to train the model and the model will output results.

[0009] b) Cleaning the data and preparing it for AI and validation, including data quality techniques to remove label noise or bad data (the focus of this patent).

[0010] c) Extract features if required by the model.

[0011] d) Selecting model configurations, including model architecture and machine learning hyperparameters.

[0012] e) Splitting the data into a training data set, a validation data set and / or a test data set.

[0013] f) Training the model using machine learning and / or deep learning algorithms on the training dataset. Typically, during the training process, many models are generated by adjusting and tuning the machine learning configurations to optimize the model's performance (e.g., to increase the accuracy metric) and generalizability (robustness). Each training iteration is called an epoch, and at the end of each epoch, the accuracy is estimated and the model is updated.

[0014] g) Selecting the best "final" model or ensemble of models based on the model's performance on the validation dataset, which is then applied to an "unseen" test dataset to validate the performance of the final AI model.

[0015] To effectively train a model, the training data must contain the correct labels or annotations (the correct class labels / targets in terms of the classification problem). Machine learning or deep learning algorithms find patterns in the training data and map them to the targets. The trained model that results from this process is able to capture these patterns.

[0016] As AI-based technologies become more widespread, the demand for high-quality (i.e., accurate) AI predictive models is becoming more evident. However, the performance of an AI model is highly dependent on the quality of the data, and the impact of poor-quality data for model training is significant, resulting in a low-quality AI model or AI product, which will have poor decision-making results when actually used (i.e., to classify new data).

[0017] Low quality data can arise in several ways. In some cases, data is missing or incomplete, due to unavailability of information or due to human error. In other cases, data can be biased, for example, when the distribution of training data does not reflect the actual environment in which the machine learning model will be run. For example, in binary classification, this can happen when the number of samples of one class ("class 0") is much higher than the number of samples of the other class ("class 1"). A model trained on this dataset can be biased towards predicting class "0" simply because it has been trained with more class 0 examples.

[0018] Another cause of low data quality is when the data is inaccurate, i.e. there is label noise that makes some class labels inaccurate. This can be due to data entry errors, uncertainty or subjectivity in the data labeling process, or factors beyond the scope of understanding of the collected data, such as measurement, clinical or scientific practice. In some cases or problem domains, noisy data may only occur for a subset of classes. For example, some classes can be reliably labeled (correct classes), whereas other classes (noisy classes) contain higher levels of noise due to uncertainty or subjectivity in the labeling process. Exceptionally, inaccurate or erroneous data may be intentionally added with the aim of negatively affecting the quality of the trained AI, which is called an "adversarial attack".

[0019] In healthcare, collecting high quality data is difficult. Some examples include:

[0020] a) When assessing embryo images for viability in IVF, an embryo can be considered viable if it leads to a pregnancy and non-viable if it does not. The viable class in this case is considered a certain ground truth outcome since it led to a pregnancy. However, the ground truth in the non-viable class is uncertain and may be misclassified or mislabeled, since a completely viable embryo may not lead to a pregnancy due to other factors related to the patient or the IVF process, not related to the original embryo viability.

[0021] b) When evaluating chest x-rays for pneumonia, radiologists visually look for white spots in the lungs (called infiltrates) that identify infection. This evaluation is subjective and prone to error. Also, images may lack information (or features) necessary for an AI or an expert radiologist to make robust and proper inferences that may or may not be available to medical personnel with access to more extensive testing, and are not evaluated purely from a single image. Thus, these images lack important features necessary for AI training, affecting the quality of the trained AI.

[0022] c) When assessing cancer on radiological imaging or early glaucoma on retinal imaging, the presence of cancer or glaucoma may be certain because it has been identified and confirmed by relevant medical testing (e.g., biopsy). However, the absence of cancer or glaucoma may be uncertain because cancer or glaucoma may be present but not identified or detected.

[0023] In the absence of ground truth or facts, the labeling process may result in noisy data in all classes for the same reasons mentioned above.

[0024] Therefore, data cleansing, the process of identifying and addressing low-quality data (e.g., mislabeled or "noisy" data) to improve data quality, is a key component for generating predictive models with both high classification accuracy and generalizability. AI companies are attempting to remove low-quality data from training datasets or train more robust (noise-resistant) models. However, this remains an open problem in many areas, and many companies are investing heavily in research and development to find techniques to address or mitigate the risks of low-quality data.

[0025] Many approaches assume that data labels can be correctly identified / annotated by experts and that large datasets may contain discriminative, noisy labels. These approaches are considered to be confident learning approaches. Therefore, a confident learning approach is: a) estimating a joint probability distribution to characterize the class-conditional label noise; b) removing noisy examples or changing their class labels; c) training a model on the "cleaned" dataset; and Includes.

[0026] If a small dataset with clean labels is available, it can be exploited to improve the reliability of neural networks using knowledge distillation. Another strategy for robust (noise-resistant) models is to introduce synthetic noise into a given dataset to promote noise-resistant parameters by updating the (student) model to give consistent predictions with the teacher model that is not affected by the synthetic label noise. Another confident learning approach involves using a mentor model to filter the corrupted ground truth labels. However, this is only possible if a definitive ground truth is obtained. That is, this kind of method requires some degree of supervision to work. Another confident learning method based on self-learning (SL) trains an initial classifier with noisy labels in an initial iteration (first epoch). Subsequent iterations (or epochs) use the ranked "prototypes" to revise the labels and retrain the model with the revised labels. This process continues iteratively until convergence is obtained.

[0027] However, in many cases, especially in real-world applications, the ground truth cannot be identified accurately or reliably, and so confident learning methods often fail when applied to real-world problems.

[0028] For example, there may be multiple data owners, each of which provides a set of data samples / images that can be used for training, validation, and testing of a model. However, different data owners may have different data collection procedures, data labeling processes, adopted data labeling rules (e.g., when measurements were taken), and geographic locations, resulting in different collection and labeling errors for each data owner. Also, for each data owner, labeling errors may occur in all classes or only in a subset of classes, with the remaining subset of classes containing minimal label noise.

[0029] Also, it is not always possible to pinpoint the ground truth or accurately assess the ground truth in all classes. For example, embryologists are not always correct in assessing the viability of an embryo. Confident cases (subclasses with a particular ground truth outcome) are those associated with images where the embryo was viable, was transferred to a patient, and the patient became pregnant six weeks later. All other cases have low confidence (or high uncertainty) that the embryo associated with the image will indeed lead to a successful pregnancy.

[0030] Therefore, there is a need to provide a method for cleaning data, or at least provide a useful alternative to existing methods. Summary of the Invention [Means for solving the problem]

[0031] According to a first aspect, there is provided a computational method for cleaning a dataset for generating an artificial intelligence (AI) model, the method comprising: generating a cleansed training data set, Splitting a training data set into multiple (k) training subsets; For each training subset, multiple (n) artificial intelligence (AI) models are The remaining (k-1) Training subset TTrain two or more of them, and use the trained AI models, Trained obtaining an estimated label for each example in the training subset for each AI model; Trained removing or relabeling samples in the training dataset that were consistently incorrectly predicted by the plurality of AI models; and training one or more AI models using the cleansed training dataset to generate a final AI model; Deploying the final AI model; and Includes.

[0032] In one form, the plurality of artificial intelligence (AI) models includes a plurality of model architectures.

[0033] In one embodiment, for each training subset, a plurality of artificial intelligence (AI) models are The remaining (k-1) Training subset T Training in two or more of them is For each training subset, multiple artificial intelligence (AI) models are The remaining (k-1) Training subset T To train with all our strengths, Includes.

[0034] In one form, removing or relabeling samples in the training dataset comprises: Obtaining a count of the number of times each sample in the training dataset was correctly predicted, incorrectly predicted, or passed a threshold confidence level by the plurality of AI models; removing or relabeling samples in the training dataset that are consistently predicted incorrectly by comparing the predictions to a consistency threshold; Includes.

[0035] In one form, the consistency threshold is estimated from a distribution of counts.

[0036] In one form, the consistency threshold is determined using an optimization method to identify a threshold count that minimizes the cumulative distribution of counts.

[0037] In one form, determining the consistency threshold comprises: generating a histogram of the counts, each bin of the histogram containing the number of samples in the training dataset that have the same count, the number of bins being the number of training subsets multiplied by the number of AI models; generating a cumulative histogram from the histogram; calculating a weighted difference between each pair of adjacent bins in the cumulative histogram; setting the consistency threshold as the bin that minimizes the weighted difference; Includes.

[0038] In one form, the method includes, after generating the cleansed training set and prior to generating the final AI model, iteratively retraining the plurality of trained AI models with the cleansed dataset; generating an updated, cleansed training set until a predetermined level of performance is achieved or there are no further samples with counts below the consistency threshold; Further includes:

[0039] In one form, prior to generating the cleansed dataset, the training dataset is tested for positive predictive power, and the training dataset is cleaned only if the positive predictive power is within a predetermined range, and estimating the positive predictive power comprises: Splitting the training dataset into multiple validation subsets; For each validation subset, multiple artificial intelligence (AI) models are The remaining (k-1) Validation subset of T Training in at least two of them, Each sample in the validation data set is Trained obtaining a first count of the number of times the plurality of AI models predicted correctly, predicted incorrectly, or passed a threshold confidence; Randomly assigning a label or outcome to each sample; For each validation subset, multiple artificial intelligence (AI) models were The remaining (k-1) Validation subset of T Training in at least two of them, When randomly assigned labels are used, each sample in the validation dataset Trained obtaining a second count of the number of times the plurality of AI models predicted correctly, predicted incorrectly, or passed a threshold confidence; estimating the positive predictive power by comparing the first count and the second count; and Includes.

[0040] In one form, the method is repeated for each dataset in the plurality of datasets, and the step of generating a final AI model by training one or more AI models with the cleansed training dataset includes: generating an aggregated dataset using the plurality of cleaned datasets; and training one or more AI models using the aggregated dataset to generate a final AI model; Includes.

[0041] In one form, after generating the aggregated data set, the method further comprises cleaning the aggregated data set according to the method of the first aspect.

[0042] In one form, after cleaning the aggregated data set, the method further comprises: for each dataset for which the positive predictive power falls outside the predetermined range, adding the untrainable dataset to the aggregated dataset and cleaning the updated aggregated dataset according to the method of the first aspect; Further includes:

[0043] In one form, the method comprises: Identifying one or more noisy classes and one or more accurate classes; Further comprising: After training a plurality of artificial intelligence (AI) models, the method includes: selecting a set of models, where the models are selected if the metric for each correct class exceeds a first threshold and the metric in each noisy class is less than a second threshold; obtaining a count of the number of times each sample in the training data set was correctly predicted or passed a threshold confidence level for each of the selected models; The step of removing or relabeling samples in the training dataset whose counts are below a consistency threshold is performed separately for each noisy class and correct class, and the consistency threshold is a consistency threshold for each class. The first metric and the second metric can be an average correct answer rate or a confidence-based metric. Multiple metrics may be calculated for each class (e.g., correct answer rate, average correct answer rate and log loss), and an order is defined (e.g., a primary metric and a secondary tie-breaking metric).

[0044] In one embodiment, the method further comprises assessing label noise in the dataset, the step comprising: Splitting the dataset into a training set, a validation set and a test set; Randomizing class labels within the training set; training an AI model on the training set using randomized class labels and testing the AI ​​model on the validation set and test set; estimating a first metric for the validation set and a second metric for the test set; rejecting the data set if the first metric and the second metric are not within a predetermined range; The first metric and the second metric may be an average correct answer rate or a confidence-based metric.

[0045] In one form, the method further comprises assessing metastatic potential of the dataset, the step comprising: Splitting the dataset into a training set, a validation set and a test set; training an AI model with the training set and testing the AI ​​model with the validation set and the test set; estimating a first metric for the validation set and a second metric for the test set for each epoch within the plurality of epochs; estimating a correlation between the first metric and the second metric across the plurality of epochs; and The first metric and the second metric may be an average correct answer rate or a confidence-based metric.

[0046] According to a second aspect, there is provided a computational method for labelling a dataset for generating an artificial intelligence (AI) model, the method comprising: Splitting a labeled training dataset into multiple (k) training subsets, where there are C labels; For each training subset, multiple (n) artificial intelligence (AI) models are The remaining (k-1) Training subset T Training in at least two of them, obtaining a plurality of label estimates for each sample in an unlabeled dataset using the plurality of trained AI models; repeating the dividing, training and obtaining steps C times; assigning a label to each sample in the unlabeled dataset by using a voting strategy and combining multiple estimated labels for the samples; Includes.

[0047] In one form, the plurality of artificial intelligence (AI) models includes a plurality of model architectures.

[0048] In one embodiment, for each training subset, a plurality of artificial intelligence (AI) models are The remaining (k-1) Training subset T Training in two or more of them is For each training subset, multiple artificial intelligence (AI) models are The remaining (k-1) Training subset T All of us should train, Includes.

[0049] In one form, the method further comprises cleaning the labelled training data set according to the method of the first aspect.

[0050] In one form, the steps of dividing, training and obtaining and repeating the dividing and training steps C times include: generating C temporary datasets from the unlabeled dataset, each sample in the temporary dataset being assigned a temporary label from the C labels such that each of the multiple temporary datasets is a distinguishable dataset; Including, repeating the partitioning, training, and obtaining C times includes performing the partitioning, training, and obtaining steps for each of the temporary datasets such that, for each temporary dataset, the partitioning includes combining the temporary dataset with the labeled dataset and then partitioning into multiple (k) training subsets; The steps of training and obtaining include: for each training subset, training a plurality (n) of artificial intelligence (AI) models to the The remaining (k-1) The training subset of brain and training the AI ​​models on two or more of the above. Using the trained AI models, TrainedObtaining an estimated label for each sample in the training subset for each AI model.

[0051] In one embodiment, the assigning of a temporary label from the C labels is randomly assigned.

[0052] In one form, the assignment of a temporary label from the C labels is inferred by an AI model trained on the training data.

[0053] In one form, assigning temporary labels from the C labels is assigned in a random order from the set of C labels such that each label occurs once in the set of C temporary data sets.

[0054] In one form, the step of combining the temporary dataset with the labeled training dataset further includes splitting the temporary dataset into a plurality of subsets, combining each subset with the labeled training dataset, splitting into a plurality (k) of training subsets, and performing the training step.

[0055] In one embodiment, the size of each subset is less than 20% of the size of the training set.

[0056] In one form, C is 1 and the voting strategy is a majority inference strategy.

[0057] In one form, C is 1 and the voting strategy is a maximum confidence strategy.

[0058] In one form, C is greater than 1 and the voting strategy is a consensus-based strategy based on the number of times each label has been estimated by multiple models.

[0059] In one form, C is greater than 1 and the voting strategy counts the number of times each label is estimated for a sample and assigns the label with the highest count that is greater than a threshold amount of the second highest count.

[0060] C is greater than 1, and the voting strategy is configured to estimate labels that are reliably estimated by multiple models.

[0061] In one embodiment, the dataset is a healthcare dataset. In a further embodiment, the healthcare dataset comprises a plurality of healthcare images.

[0062] According to a third aspect, there is provided a computing system including one or more processors, one or more memories, and a communications interface, the one or more memories storing instructions for configuring the one or more processors to perform the method of the first or second aspect. According to a fourth aspect, there is provided a computing system including one or more processors, one or more memories, and a communications interface, the one or more memories configured to store an AI model trained using the method of any one of claims 1 to 32, the one or more processors configured to receive input data via the communications interface and process the input data using the stored AI model to generate model results, the communications interface configured to transmit the model results to a user interface or a data storage device. [Brief description of the drawings]

[0063] Embodiments of the present disclosure will be described with reference to the accompanying drawings. [Figure 1A] FIG. 1A is a schematic diagram showing possible combinations of predictions (P), ground truth (T) and measurements (M) for a binary classification model with binary outcomes of viability (V) and non-viability (NV), along with noise sources categorized in terms of positive or negative outcomes of predictions, truth and measurements, according to one embodiment. [Figure 1B]FIG. 1B is a schematic flow chart of a method for cleansing a dataset according to one embodiment. [Figure 1C] FIG. 1C is a schematic diagram of cleaning multiple data sets according to one embodiment. [Figure 1D] FIG. 1D is a schematic flow chart of a method for labeling a dataset according to one embodiment. [Figure 1E] FIG. 1E is a schematic architectural diagram of a cloud-based computing system configured to generate and use AI models according to one embodiment. [Figure 1F] FIG. 1F is a schematic flow chart of a model training process on a training server according to one embodiment. [Diagram 2] Figure 2 is an example of an image of a dog that would easily be confused with a cat by an otherwise highly accurate model. [Figure 3A] FIG. 3A is a plot of the accuracy of a trained model measured on a test set, Ttest, where uniform label noise in the training data is only (■) and in the test set is only (▲), and is equal in both sets (●), in accordance with one embodiment. [Figure 3B] FIG. 3B is a plot of the accuracy of the trained model measured on a test set, Ttest, where single-class noise is only (■) in the training data and only (▲) in the test set, and is equal in both sets (●), in accordance with one embodiment. [Figure 4A] FIG. 4A is a plot of cumulative histograms Hl at various accuracy levels, where l is the uniform noise level for the 30 / 30 case, according to one embodiment. [Figure 4B] FIG. 4B is a plot of cumulative histograms Hl at various accuracy levels, where l is the asymmetric noise level for the 35 / 05% case, according to one embodiment. [Figure 4C] FIG. 4C is a plot of cumulative histograms Hl at various accuracy levels, where l is the uniform noise level for the 50 / 50 case, according to one embodiment. [Figure 4D] FIG. 4D is a plot of cumulative histogram Hl at various accuracy levels, where l is the asymmetric noise level for the 50 / 05% case, according to one embodiment. [Diagram 5] FIG. 5 is a set of histogram plots showing accuracy (top) and cross-entropy or log loss (bottom) for various model architectures before UDC (left) and for the ResNet-50 architecture after UDC (right) for varying accuracy thresholds l, according to one embodiment. [Figure 6] FIG. 6 is a set of histogram plots showing accuracy (top) and cross-entropy or log loss (bottom) for various model architectures before and after UDC (left) for varying accuracy thresholds l, in accordance with one embodiment. [Figure 7] FIG. 7 is a histogram of the number of images per accuracy threshold for the test and training sets in normal and pneumonia labeled images according to one embodiment. [Figure 8] FIG. 8 is a plot of images split into clean and noisy labels, further subdivided into images from training and test sets, and again subdivided into normal and pneumonia classes showing agreement and mismatch between clean and noisy labels, according to one embodiment. [Figure 9] FIG. 9 is a plot of the calculation of kappa coefficients for multiple noisy labels and clean labels, according to one embodiment. [Figure 10] FIG. 10 is a histogram plot of match and mismatch levels for both clean-label and noisy-label images, according to one embodiment. [Figure 11A] FIG. 11A is a histogram plot of the accuracy rate before and after UDC (cleaned data) for varying accuracy thresholds l, according to one embodiment. [Figure 11B]FIG. 11B is a histogram plot of accuracy rates (left) for various models before and after (right) UDC for varying accuracy thresholds l, according to one embodiment. [Figure 12] FIG. 12 is a plot of test curves when an embodiment of an AI model is trained on uncleaned data, with the non-viable and viable classes in dotted and solid lines, respectively, and two average curves in dashed lines. [Figure 13] FIG. 13 is a plot of test curves when an embodiment of an AI model is trained on cleaned data, with the non-viable and viable classes in dotted and solid lines, respectively, and two average curves in dashed lines. [Figure 14] FIG. 14 is a plot of frequency versus number of incorrect predictions when UDL according to one embodiment is applied to a set of 200 chest x-ray images inserted into a larger training set of over 5000 images, showing that clean labels are more susceptible to being correctly labeled, whereas noisy labels are less susceptible.

[0064] In the following description, like reference numbers represent like or corresponding parts throughout the figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0065] Method embodiments for cleaning a dataset to address the problem of label noise are described below and collectively referred to as "Untrainable Data Cleansing" (UDC). These embodiments may cleanse a dataset by identifying misclassified or noisy data in a subset of classes or all classes. For classification problems, UDC method embodiments allow mislabeled data to be identified so that the data can be removed, relabeled, or otherwise addressed before the start of or during training of an AI model. UDC method embodiments can also be applied to non-classification problems (i.e., non-categorical data or results), such as regression, object detection / segmentation models, where the model can provide a confidence estimate of the result. For example, for a model that estimates a bounding box in an image, the method estimates whether the box is unacceptable, relatively good, good, very good, or some other confidence level (rather than accurate / inaccurate). If misclassified or noisy data is identified, a decision can be made on how to clean the data (e.g., change the label or remove the data). The cleaned data can then be used to train an AI model, and the trained data can be deployed to receive and analyze new data and generate model results (e.g., classification, regression, object bounding boxes, segmentation, etc.) Embodiments of the method can be used on a single dataset or multiple datasets, either from the same source or from multiple sources.

[0066] As outlined below, embodiments of the UDC method can be used to identify mislabeled or difficult / impossible to label (inconsistent or useless) data with high levels of accuracy and confidence, even for "hard" classification problems such as detecting pneumonia from pediatric x-rays. Further variations of the same AI training method can be used for AI inference to confidently identify (or "infer") unknown labels for previously unseen data. This training-based inference approach, which we denote Untrainable Data Labeling (UDL), can produce more accurate and robust inferences, especially for applications where accuracy is important but time or cost is not (e.g., detecting cancer in images). This is particularly true for healthcare / medical datasets, although it will be realized that the method has broader applications beyond healthcare applications. This training-based inference is the opposite of traditional AI inference, which is a model-based approach.

[0067] Before focusing on specific methods for handling noisy data, it is instructive to consider a domain-specific example to further illustrate how label noise fits into the model predictions (P), the actual (ground truth) outcomes (T), and the possible outcomes of the (possibly noisy) measurements (M) recorded as a surrogate for the ground truth. Figure 1A is a schematic diagram 130 summarizing the possible combinations of these three categories in the case of a binary classification problem of embryo viability at day 5 (e.g., to aid in the choice of whether to transfer an embryo as part of an IVF procedure) by an AI model 110. In this scenario, an image of the embryo is evaluated by the AI ​​model to estimate viability and therefore whether the embryo should be transferred or not. Around six weeks after implantation, an attempt is made to measure the fetal heartbeat, which if detected is indicated as a viable embryo, if not detected is indicated as a non-viable embryo. This is a surrogate for the ground truth, as it is possible that a viable embryo was transferred but subsequently miscarried for some other reason (e.g., maternal or external factors). For these three categories, the binary tree is divided into two parts: 3There are =8 combinations, each of which may be associated with goodness or usefulness of training 132 (i.e., whether the examples represent real cases without label noise or are noisy) and the likelihood that they will occur in the dataset. An exemplary (non-exhaustive) overview of possible noise sources 134 is also shown in Figure 1A. Agreement or disagreement between classification model predictions (P) and measurements (M) is indicated by shading, with medium risk indicated by light shading and dark black shading indicating the highest risk in this problem area.

[0068] The combinations of prediction, truth and measurement corresponding to true positive (TP), false positive (FP), false negative (FN) and true negative (TN) are also shown in FIG. 1A. In this case, the highest risk of label noise is associated with prediction and truth being positive and measurement being negative (labeled as not suitable for training and likely to occur: "BL"). We outline an embodiment of how to distinguish these examples from false positives, which are generally not easily distinguishable in the absence of absolute ground truth. The arrows in FIG. 1 indicate that during machine learning, when noisy examples predominate, as marked, the model is trained incorrectly, which can lead to a large amount of false negatives. Similarly, there are examples where prediction and measurement are negative and truth is positive (labeled as "BL"), which can lead to a large amount of false positives during training.

[0069] FIG. 1B is a flowchart of a computational method 100 for cleaning a dataset for generating an artificial intelligence (AI) model, according to one embodiment. A cleansed training dataset 101 is generated by splitting the training dataset into multiple training subsets 102. Then, for each training subset, multiple artificial intelligence (AI) models are trained on two or more, usually (but not necessarily) all, of the remaining multiple training subsets 104 (i.e., a K-cross-fold validation based approach). Each of the AI ​​models may use different model architectures to create diverse AI models (i.e., distinguishable model architectures) or the same architecture may be used but with different hyper-parameters. The multiple model architectures include various common architectures such as random forests, support vector machines, clustering, deep learning / convolutional neural networks including ResNet, DenseNet, or InceptionNet, as well as the same general architecture but with different internal configurations such as the number of layers and connections between layers, e.g., ResNet-18, ResNet-50, ResNet-101.

[0070] Next, remove or relabel samples in the training dataset that are consistently predicted incorrectly by multiple AI models 104. In some embodiments, this may include taking a count of the number of times each sample in the training dataset was correctly predicted by multiple models, or alternatively the number of times it was predicted incorrectly by multiple models. Alternatively, a count of the number of times each sample in the training dataset passed a threshold confidence level by multiple AI models may be taken. Next, remove or relabel samples in the training dataset that are consistently predicted incorrectly by comparing the count to a consistency threshold 105 (e.g., below the consistency threshold if counting correct predictions, or above the consistency threshold if counting incorrect predictions). The consistency threshold may be estimated from the distribution of the counts, and an optimization method identifies a threshold count that minimizes the cumulative distribution of the counts (e.g., by using a cumulative histogram and calculating a weighted difference between each pair of adjacent bins in the cumulative histogram). The choice of whether to remove low-confidence cases or perform label swapping may be determined based on the problem at hand.

[0071] In some embodiments, this cleaning process may be repeated by iteratively retraining the trained AI models with the cleansed dataset to generate an updated cleansed dataset 106. The iterations may be performed until a predetermined level of performance is obtained. This may be a predetermined number of epochs, after which convergence is assumed to be obtained (so the model after the last epoch is selected). In another embodiment, the predetermined level of performance may be based on changing the thresholds of one or more metrics, such as a percent correct and / or confidence-based evaluation metric. In the case of multiple metrics, this may be a change in the thresholds of each metric, or a primary metric may be defined and a secondary metric may be used as a tiebreaker, or two (or more) primary metrics may be defined and a tertiary (or more) metric may be used as a tiebreaker. In some embodiments, prior to cleaning the dataset (101), the positive predictive power of the dataset may be estimated (107) to estimate the amount of label noise present (i.e., data quality). As described below, this may be used to influence whether or how data cleansing is performed.

[0072] After obtaining the cleansed dataset, a final AI model is generated by training one or more AI models using the cleansed training dataset 108, and then the final AI model 110 is deployed for use on the actual dataset.

[0073] The method embodiments may be used for a single dataset or multiple datasets. There may be single or multiple data owners, data sources and sub-data sets. Each of the multiple data owners provides a set of data samples / images that can be used for training, validating and testing the model. Data owners may have different data collection procedures, data labeling processes and geographic locations, and collection and labeling errors may occur differently for each data owner. Also, for each data owner, labeling errors may occur for all classes or only for a subset of classes, so that the remaining subset of classes may contain minimal label noise.

[0074] FIG. 1C illustrates an embodiment of a method 120 for cleaning multiple datasets based on the method 100 for cleaning a single dataset illustrated in FIG. 1B. In this embodiment, there are four datasets 121, 122, 123, 124, which may be from the same source or from multiple data sources. Each dataset is first tested for predictive power 107. Then, datasets with low predictive power, such as dataset 3 123, are removed. Then, datasets with sufficient (i.e., positive) predictive power (e.g., above a certain threshold) are individually cleaned (101) using the method illustrated in FIG. 1B. Next, the cleaned datasets are assembled (125), and the assembled datasets are cleaned (126) using the method illustrated in FIG. 1B. This cleaned assembled dataset is used to generate (and then deployed (110)) an AI model 108. In another embodiment, the dataset with low predictive power (e.g., dataset 3 123) is combined (127) with the cleaned, combined dataset 126, and this updated, cleaned, combined dataset is cleaned (128). A final AI model may then be generated (108) and deployed (110).

[0075] As briefly mentioned above, the UDC method can be modified to infer unknown labels for previously unseen data. Figure 1D is a flow chart of a method 130 for labeling a dataset according to one embodiment (the UDL method). Figure 1D shows two variants of the UDL method: a standard UDL method and a fast UDL method that is less computationally intensive than the standard UDL (the variant indicated by the dashed lines in Figure 1D). First, the standard UDL is described, followed by the fast UDL variant.

[0076] UDL is a completely new approach to AI inference. Current approaches to AI inference use a model-based approach, where training data is used to train an AI model, which is then used to infer previously unseen data and classify them (i.e., identify labels or annotations). The AI ​​model is based on general patterns or statistically averaged distributions learned from the training data. If the previously unseen data is of a different distribution or an edge case, the chances of misclassification / labeling are high, negatively impacting accuracy and generalizability (scalability / robustness). UDL, on the other hand, is a training-based inference approach. Instead of training an AI model, the AI ​​training process itself is used to determine the classification of previously unseen data.

[0077] For both standard UDL and fast UDL, labeled training datasets are obtained, C-labeled 131 and unlabeled dataset 133. In some embodiments, the labeled training datasets may be cleaned using an embodiment of the UDC method illustrated and described in Figures 1B and 1C.

[0078] Next, a C temporary dataset is generated from the unlabeled dataset, and each sample in the temporary dataset is assigned a temporary label from the C labels such that each of the multiple temporary datasets is a separate dataset (134). That is, each unlabeled sample is assigned one label from the list of classes c∈{1..C}. These temporary labels can be random labels or labels based on a trained AI model (as in standard AI model-based inference approaches). That is, the training data is used to train an AI model or an ensemble AI model, and the AI ​​model is used to perform first-pass inference and set preliminary labels for the unknown data. In another embodiment, the temporary labels are assigned in a random order from the set of C labels such that each label occurs once in the set of C temporary datasets. That is, the following UDL method is repeated for all / multiple labels in the unknown dataset, assigning each sample / data point (e.g., image) one label from the list of classes c∈{1..C} in a random order, and testing each class label for each sample / data point.

[0079] For each temporary dataset, the temporary dataset and the labeled training dataset are combined (135). That is, the unknown data is inserted into the training data. Next, the embodiment of the UDC method shown in FIG. 1B is performed to identify / infer the actual / final labels of the unknown data, which includes splitting the labeled training dataset into multiple (k) training subsets, where there are C labels (137), and then, for each training subset, training multiple (n) artificial intelligence (AI) models on two or more, typically (but not necessarily), all of the remaining multiple training subsets (i.e., a K-cross-split validation based approach). Each of the AI ​​models may use a different model architecture. Then, the multiple (e.g., n×k) trained AI models are used to obtain multiple (e.g., n×k) label estimates for each sample in the unlabeled dataset (139). This process is repeated (i.e., C times) for each temporary dataset 140.

[0080] Next, we assign a label to each sample in the unlabeled dataset by using a voting strategy to combine the multiple predicted labels for the sample (142). For example, if we use a binary classification scheme, C UDL = 1, for example, if you want to classify an image as a cat or a dog, you run UDL once in the unknown dataset by setting the temporary label to either cat or dog, and UDC will determine if the label was the correct choice and infer the final label. Alternatively, in multi-class classification, UDL is run for all possible labels. For example, if there are three possible classes (e.g., labels for cat, dog, and bear), C UDL = 3, so we run UDL three times, with the temporary labels first being cat, then dog, then bear (in a shuffled order for each image). The total number of trained models is R UDL = n × k × C UDL It becomes.

[0081] In the case of a single class (C UDL= 1), labels can be assigned using majority inference labeling. The labels selected for each of the unknowns are then used to UDL In another embodiment, a maximum confidence strategy may be used. The label selected for each unknown data point is the label or classification with the maximum sum of the confidences of the labels. That is, the sum of the confidence scores of the labels c is S c =Σ r conf c or the sum of the confidence scores of the label c for all models, r ∈ {1..R UDL}. The selected label is C UDL =max(S c ), i.e., the label with the maximum sum of confidence scores.

[0082] Multi-class, multi-label (C UDL >1), the voting strategy is a consensus-based strategy based on the number of times each label was estimated by multiple models. That is, we divide the inference results of each UDC performed by class label c and compare the number of correct predictions for each UDC result. The class with the highest number of correct predictions is the label selected for the image. If the label is easily identified as one of the classes from C, the difference between the number of correct predictions for this class compared to the other classes is expected to be very high. When this difference approaches the maximum difference (n × k), the confidence of the selected label is c. The confidence can be estimated based on the difference between the label with the highest number of successful predictions (say label A) and the second best label (say label B). Thus, the confidence of a UDL whose labels are the difference is =num(label A) - num(label B). A larger difference indicates a higher confidence.

[0083] In the above method, unlabeled data is inserted into the training data and UDC techniques are used up to c times in total to determine which temporary labels, if any, are clearly correct (not mislabeled) or clearly incorrect (mislabeled). Ultimately, if the actual label for a new image is recognizable (the data in the image is not so noisy or inconsistent / useless that it contains no identifiable features), UDC can be used to reliably identify (or predict / infer) this label or classification. The labeled data can then be used, for example, to make decisions, identify noisy data, and generate more accurate and generalizable AI models that can then be deployed (143).

[0084] By inserting unknown data into the training data, the training process itself tries to find specific patterns, correlations and / or statistical distributions of the unknown data relative to the (clean) training data. Thus, the process becomes more targeted and individualized to the unknown data, because the specific unknown data is analyzed and associated within the context of other data with known outcomes as part of the training process, and the iterative training-based UDC process itself eventually identifies the most likely label for the specific data, potentially increasing both accuracy and generalizability. Even if the statistical distribution of the unknown data is different or an edge case compared to the training data, embedding the data in the training extracts correlations or patterns with the training data that best classify the unknown data. If an AI model cannot be trained to classify the unknown data with a temporary label, especially if the unknown data is included in the training set itself, it becomes clear that the label is incorrect and an alternative label is the correct prediction / inference or the image is very noisy and does not contain any identifiable features for the AI ​​to learn from. Thus, inference using UDL (with UDC) is more likely to produce accurate and generalizable (robust and scalable) AI inferences than traditional model-based AI inferences. However, implementing UDL training-based inference is time-consuming and computationally expensive.

[0085] In some embodiments, the temporary dataset is split into multiple subsets, each of which is then combined with the labeled training dataset. This is to ensure that the size of the new dataset is small enough not to introduce significant noise into the larger training dataset (i.e., if the temporary label is incorrect). The optimal dataset size to avoid poisoning the training process is one sample, but this can be more expensive, since each data point in the dataset needs to undergo a cost- and time-dependent UDC process to infer the label. In some embodiments, the temporary dataset is split so that the size of each subset is less than 10% or 20% of the size of the training set.

[0086] FIG. 1D also shows an alternative embodiment called Fast UDL, which is a more computationally efficient approximation of UDL. It may use a standard model-based approach for inference rather than a training-based approach, but like UDC and UDL, allows for inference of many AI models to identify labels for unknown datasets. Therefore, in this embodiment, we skip creating a temporary dataset (134) and combining with the training dataset (135), and instead perform the splitting (137) and training (138) steps only on the training data, to generate a clean training dataset with n×k×C labels. UDL We then proceed directly to creating a diverse set of AI models (creating n×k models) (dashed line 136), and then obtain multiple label estimates (138) by using each of the n×k models to infer labels in the unknown dataset. The partitioning (137), training (138), and obtaining (139) are repeated C times (141). Thus, each data point in the unknown dataset has n×k inferences / labels and confidence scores, which are used to assign a label to each sample in the unlabeled dataset (142), similar to standard UDL.

[0087] The method embodiment may be implemented in a cloud computing environment or a similar server farm or high performance computing environment. Figure 1E is a schematic architecture diagram of a cloud-based computing system 1 configured to generate and use an AI model, according to one embodiment. This is shown in the context of training the AI ​​with healthcare data including medical / healthcare images and associated patient medical records (including clinical data and / or diagnostic test results). Figure 1F is a schematic flow chart of a model training process on a training server, according to one embodiment.

[0088] The AI ​​model generation method is handled by the Model Monitor 21 tool. The Monitor 21 requests that the user 40 provide data (including data items and / or images) and metadata 14 to a data management platform including a data repository. Data preparation steps are performed, e.g., moving data items or images to specific folders, renaming any images, pre-processing such as objection detection, segmentation, alpha channel removal, padding, cropping / localization, normalization, scaling, etc. Feature descriptors are also calculated and augmented images are pre-generated. However, additional pre-processing including augmentation may also be performed during training (i.e., on the fly). Images may also undergo quality assessment to reject obviously poor quality images and allow for incorporation of alternative images. Data such as patient records or other clinical data are processed (prepared) and classification results such as viable and non-viable in binary classification, output class in multi-class classification, or outcome measures in the case of non-classification are extracted that are linked or associated with each image or data item so that they can be used to train and / or evaluate the AI ​​model. The prepared data can be loaded 16 into a template server 28 of a cloud provider (e.g. AWS) that has the latest version of the training algorithms. The template server is stored and multiple copies are made across a training server cluster 37 (which can be CPU, GPU, ASIC, FPGA or TPU (Tensor Processing Unit) based) that forms the training servers 35.

[0089] The model monitor web server 31 can request a training server 37 from multiple cloud-based training servers 35 for each job submitted by a user 40. Each training server 35 can execute pre-prepared code (from the template server 28) to train an AI model using libraries such as PyTorch, Tensorflow or equivalent, and can use computer vision libraries such as OpenCV. PyTorch and OpenCV are open source libraries of low-level commands for building CV machine learning models. The AI ​​models can be deep learning models or machine learning models, including CV-based machine learning models.

[0090] The training server 37 manages the training process. This may include, for example, splitting the data or images into training, validation and blind validation sets using a random allocation process. During the training validation cycle, the training server 37 may also randomize the set of images at the beginning of the cycle so that a different subset of images is analyzed or in a different order in each cycle. If preprocessing was not performed previously or was incomplete (e.g., during data management), additional preprocessing may be performed, including object detection, segmentation and generation of masked data sets, computation / estimation of CV feature descriptors and generation of data augmentation. Preprocessing may also include padding, normalization, etc. of images as needed. Similar processing may also be performed on non-image data. That is, the preprocessing step 102 may be performed before training, during training or in some combination (i.e., distributed preprocessing). The number of training servers 35 running may be managed from a browser interface. As training progresses, log information about the status of training is logged (62) to a distributed logging service such as cloudwatch 60. Metrics are calculated and information is also parsed from the logs and stored in a relational database 36. Models are periodically saved 51 to data storage (e.g., AWS Simple Storage Service (S3) or a similar cloud storage service) 50 so that they can be retrieved and loaded at a later date (e.g., for restart in case of an error or other outage). Users 40 can send email updates 44 on the status of the training server when a job is completed or if an error occurs.

[0091] Within each training cluster 37, a number of processes take place. When a cluster is started via the web server 31, a script runs automatically, reading prepared images and patient records and initiating the specific Pytorch / OpenCV training code 71 requested. Input parameters for model training 28 are provided by the user 40 via the browser interface 42 or via a configuration script. The training process 72 is then initiated for the requested model parameters, which can be a long-running and intensive task. Therefore, to ensure that progress is not lost while training is in progress, logs are periodically saved (62) to a log (e.g., AWS cloudwatch) service 60, and the current version of the model (in training) is saved (51) to a data (e.g., S3) storage service 51 for later retrieval and use. An embodiment of a schematic flow chart of the model training process on the training server is shown in FIG. 3B. With access to the set of trained AI models on the data storage service, a set of deep learning models (e.g., PyTorch) and / or target computer vision models (e.g., OpenCV) can be incorporated to generate a robust AI model (108) that can then be deployed to a distribution platform 80, for example by combining multiple models together using ensemble, distillation, or similar approaches. The distribution platform may be a cloud-based, server-based, or other computing system, and the same computing system used to train the AI ​​model may be used to deploy the AI ​​model.

[0092] A model is defined by its network weights, and deployment may include exporting these network weights and loading them into the distribution platform 80 to finally run the trained AI model 108 on new data. In some embodiments, this may involve exporting or saving a checkpoint file or model file using appropriate functions of the machine learning code / API. A checkpoint file may be a file generated by the machine learning code / library in a defined format that can be exported and then reloaded using standard functions provided as part of the machine learning code / API (e.g., ModelCheckpoint(), load_weights()). The file format may be directly sent or copied (e.g., ftp or similar protocol) or serialized and sent using JSON, YAML or similar data transfer protocols. In some embodiments, additional model metadata may be exported / saved and sent along with the network weights that may further characterize the model, such as model accuracy, number of epochs, or otherwise aid in building the model on another computing device (e.g., cloud platform, server, or user computing device). In some embodiments, the same computing system used to train the AI ​​model may be used to deploy the AI ​​model, so deployment may involve storing the trained AI model, for example in memory of a web server 31, or exporting the model weights to a distribution server for loading.

[0093] The delivery platform 80 is a computing system including one or more processors 82, one or more memories 84, and a communication interface 86. The memory 84 is configured to store trained AI models that may be received from the model monitor web server 31 via the communication interface 86 or loaded from an export of a model stored in an electronic storage device. The processor 82 is configured to receive input data via the communication interface (e.g., images for classification from a user 40) and process the input data with the stored AI model to generate model results (e.g., classification), and the communication interface 84 is configured to transmit the model results to a user interface 88 or export an electronic report to a data storage device. The processor is configured to receive the input data, process the input data with the stored trained AI model, and generate model results. The communication module 86 is configured to receive the input data and transmit or store the model results. The communication module may receive the input data in communication with a user interface 88, such as a web application, to display the model results, e.g., classification, object bounding boxes, segmentation boundaries, etc. The user interface 88 runs on the user computing device and is configured to allow the user 40 to drag and drop data or images into the user interface (or other local application) 88, which triggers the system to perform any pre-processing of the data or images and pass the data or images to the trained / validated AI model 108 to obtain classification or model results (e.g., object bounding boxes, segmentation boundaries, etc.), which can be immediately returned to the user in a report and / or displayed in the user interface 88.The user interface (or local application) 88 allows the user to save data such as images and patient information to a data store such as a database, generate various reports on the data, and generate audit reports on usage of the tool for their organization, groups or specific users, as well as billing and user counts (e.g., create users, delete users, reset passwords, change access levels, etc.). The delivery platform 80 may be cloud-based and may allow a product administrator to access the system to create new customer accounts and users, reset passwords, as well as access customer / user accounts (including data and screens).

[0094] In the case of multiple data owners, an AI / machine learning model may be trained using the entire training set as a combination of the individual sub-datasets. In these embodiments, the trained predictive model may be able to generate accurate results on the individual sub-datasets and on the overall test set, which is a combination of data / images from different data owners, among others. In practice, the data owners may be in different geographic locations. The sub-datasets from different owners may be stored together in a centralized location / server or may be distributed and maintained locally at each owner's location / server to meet data privacy regulations. Embodiments of the method may be used regardless of the location of the data or privacy regulations.

[0095] Embodiments of the method may be used for a range of data types, including input data types (numeric, graphical, textual, visual, and temporal data) and output data types (e.g., binary classification problems and multi-class (multi-label) classification problems). In particular, numerical, graphical, and textual structured data are common data types in general machine learning models, while graphical, visual, and temporal (audio, video) data are more common in deep learning. Output data types may include binary and multi-class data, and embodiments of the method may be used for multi-class (multi-label) classification problems in addition to binary classification problems.

[0096] Embodiments of the method may use a series of model types (e.g., classification, regression, object detection, etc.), each of which typically uses a different architecture, and within each type there may be a series of architectures that may be used. The selection of the AI ​​model type may be based on the type of input and the object (e.g., outcome) that one wishes to predict. Embodiments of the method are particularly suited (but not limited to) for supervision / classification models and healthcare datasets, such as classification of healthcare images and / or diagnostic test data (again, use is not limited to only healthcare datasets). The model may be trained using a centralized (training data stored in one geographic location) or distributed (training data stored separately in multiple geographic locations) data source, depending on the data location and data privacy issues discussed above. For distributed training, the selection of model architecture and model hyperparameters is the same as for centralized training, but the training mechanism must ensure that private data is kept private and local to each data owner's location.

[0097] Model outputs are categorical (e.g., classes / labels) for classification models and non-categorical for regression, object detection / segmentation models. Embodiments of the method may be used for any classification problem, and in addition to identifying false labels, the method may be used for more general regression, object detection and segmentation problems, where the method may provide a confidence estimate of the result. For example, for a model that estimates a bounding box, the method estimates whether the box is unacceptable, relatively good, good, very good, or some other confidence level (not accurate / inaccurate). These can be used to determine how to clean the data. Different types of labels may be sensitive to different types of noise in the image, depending on the use case of the model being trained.

[0098] The choice of AI model type (e.g., binary classification, multi-class classification, regression, object detection, etc.) typically depends on the specific problem for which the AI ​​is trained / used. The multiple trained AI models may use multiple model architectures to provide diverse models. The multiple model architectures may include the same general architecture, e.g., ResNet, but with different internal configurations, such as different number of layers and connections between layers, e.g., ResNet-18, ResNet-50, ResNet-101, in addition to various general architectures such as random forests, support vector machines, clustering, deep learning / convolutional neural networks including ResNet, DenseNet, or InceptionNet. Additional diversity can be generated by using the same model type / configuration with different combinations of model hyperparameters.

[0099] AI models may include deep learning and neural nets as well as machine learning models such as computer vision models. Computer vision models rely on identifying key features of an image and representing them in terms of descriptors. These descriptors may encode qualities such as pixel variation, gray levels, texture roughness, fixed corner points or the orientation of image gradients, as implemented in OpenCV or similar libraries. By selecting such features and searching in each image, a model can be built by finding which configurations of features are good indicators of a desired class (e.g., embryo viability). This procedure is best performed with machine learning processes such as random forests or support vector machines, which can separate images in terms of explanations from computer vision analysis.

[0100] Deep learning and neural networks "learn" features rather than relying on hand-designed feature descriptors like machine learning models. This allows them to learn "feature representations" that fit the desired task. These methods are well suited for image analysis because they can capture both small details and overall morphological shapes to arrive at an overall classification. Various deep learning models are available, each with different architectures (i.e., different number of layers and inter-layer connections), such as residual networks (e.g., ResNet-18, ResNet-50, ResNet-101), densely connected networks (e.g., DenseNet-121, DenseNet-161), and other variations (e.g., InceptionV4 and Inception-ResNetV2). Training involves accumulating various combinations of model parameters and hyperparameters, including input image resolution, optimizer choice, learning rate value and scheduling, momentum value, dropout, and weight initialization (pre-training). A loss function may be defined to evaluate the model's performance and is optimized during training of the deep learning model by varying the learning rate to drive the network's weight parameter update mechanism to minimize the objective / loss function. The AI ​​models may include multiple model architectures, including similar architectures with different hyperparameters.

[0101] Generally, machine learning algorithms require an objective / loss function, and some evaluation metric can be used to evaluate the accuracy of the model being trained (evaluated at the end of each epoch). A common loss function for binary classification problems is binary cross entropy (other loss functions may be used). The loss function basically measures the difference between the goal (actual output label) and the model's outcome (predicted output label). Other metrics that can be used to rank model epochs include:

[0102] Cross-entropy (log) loss CE: is a measure of the average number of bits required (or additional information required) to identify an event drawn from a set when the coding scheme used for the set is optimized over an estimated probability distribution q rather than the true distribution p. In a classification problem, we consider the one-hot coding (true) probability p over classes c∈{1..N}. j (c) distribution and the estimated probability distribution q for each element j∈{1..N} j (c) The cross-entropy loss is calculated by comparing the mean and mean values ​​of the elements. The result is averaged over all elements (or observations) and is given by:

[0103]

number

[0104] A=N T / N (2) Class-based accuracy rate A (c) : is the class(N T (c) ) correct prediction rate. The calculation is similar to the accuracy rate, but only considers images of one class (c) at a time.

[0105] A (c) =N T (c) / N (c) (3) Balance accuracy (or F1 score) A bal : is more suitable when the distribution of data classes is unbalanced. The average accuracy is calculated as the average of the class-based accuracy rates over all classes c∈{1..N}.

[0106]

number

[0107] Accuracy-based metrics include mean squared error (MSE), root MSE, mean of mean errors, mean average accuracy (mAP), typically used for regression and object detection model types, in addition to accuracy, mean class accuracy, sensitivity, specificity, confusion matrix, ratio of sensitivity to specificity, accuracy, negative predictive value, and mean accuracy, typically used for classification model types. Confidence-based metrics include log loss, composite class log loss, composite data source log loss, composite class and data source log loss. Other metrics include number of epochs, area under the curve (AUC) threshold, receiver operating characteristic (ROC) curve threshold, and accuracy-recall curves that indicate stability and transferability.

[0108] Evaluation metrics may vary depending on the type of problem. For binary classification problems, these may include overall accuracy, mean accuracy, log loss, sensitivity, specificity, F1 score, area under the curve (AUC) including receiver operating characteristic (ROC) curves and accuracy-recall curves. For regression and object detection models, they may include mean squared error (MSE), root MSE, mean of mean errors, mean average accuracy (mAP), confidence score and recall.

[0109] The method embodiment will now be described in more detail and some examples will be given.

[0110] In some embodiments, the predictive power of the dataset 107 is first performed to investigate the level of label noise in the dataset of each data source, thus assessing the data quality, especially the label noise. In the presence of high label noise (i.e., implying low quality data due to low predictive power), an embodiment of the UDC can be used to address / minimize the label noise and improve the data quality of the dataset. Alternatively, if it is part of a larger aggregate dataset from multiple sources, it can be removed entirely (if feasible).

[0111] In one embodiment, a basic test is performed to ensure that the model is working properly by splitting the dataset into a training, validation, and test set, and then randomizing the class labels in the training dataset. The model is then trained on the training set and tested on the validation and test sets. The average accuracy should be around 50% in both the validation and test sets, i.e., no predictive power, since the labels in the training dataset are randomized. In this case, both the data and the model are in the correct order, and further cleansing techniques can be performed if necessary. Otherwise, there may be a problem with the model's construction, training algorithm, or highly biased training data. In this embodiment, the average accuracy metric is used rather than the overall accuracy rate, because in some cases, a biased class distribution on the dataset may be associated with a very high overall accuracy rate, even though the average accuracy rate is around 50% (see the example in the Experimental Results section below). However, other metrics may be used, including confidence metrics such as log loss.

[0112] We then proceed to test the model's positive predictive power. For each data source, we split the data into training and test sets using the original dataset with the original (non-randomized) labels. The model is trained on the training set and tested on the test set. The average correct answer rate on the test set is considered (although other metrics such as log loss may be used). The closer the correct answer rate is to 100% (or some maximum benchmark correct answer rate for a particular problem domain), which indicates high predictive power, the lower the label noise in the data and the higher the data quality. The closer the correct answer rate is to 50% (or the correct answer rate calculated above when training on a randomly labeled dataset), which indicates no predictive power, the higher the label noise in the data and the lower the data quality. In one embodiment, the test for positive predictive power is performed by applying a k-cross-validation approach to the training set. That is, we split the training set into k groups and train multiple AI models for each group. We then obtain a first count of the number of times each sample in the validation dataset is predicted correctly, incorrectly, or exceeds a threshold confidence level by the multiple AI models. We then randomly assign a label or outcome to each sample and repeat the k-cross-validation approach: split the randomized training set into k groups and train multiple AI models per group. We then take a second count of the number of times each sample in the validation dataset was correctly predicted, incorrectly predicted, or exceeded a threshold confidence level by the multiple AI models. We can then estimate the positive predictive power by comparing the first and second counts. If the two counts are similar, the predictive power of the dataset is low. If the difference is large (i.e., greater than a threshold count difference), indicating that the unrandomized data correctly predicts the label based on the counted data, then the positive predictive power is high (or sufficient).

[0113] In addition, the transferability of the model can be tested. The transferability of the model to a validation dataset (optional) and a test dataset is investigated. Low-quality training data with high label noise can undermine the general habit of model training. The data is split into a training (optional) validation and test dataset. The model is trained using the training dataset, and metrics such as the average accuracy rate or log loss are calculated, taking into account the results of the validation data and the test dataset. The test and validation accuracy rates are obtained at the same training epoch. Good quality data can have a high correlation (or consistency of accuracy rates) between the validation and test datasets.

[0114] Datasets with moderate or high label noise are candidates for the UDC embodiments described herein to reduce label noise, mitigate low quality data, and ultimately improve the resulting trained AI model. In this case, the following steps are taken:

[0115] The implementation of the UDC method can be used to deal with noisy data appearing in a subset of classes or all classes in the dataset (we denote these classes as noisy classes). We assume that the remaining classes in the dataset have no or minimal noise (we denote these classes as "correct classes"). If noisy labels appear in all classes, the number of correct classes is 0, and the technique can still be used even if the level of label noise in any of the classes is lower than 50%. For simplicity, we call the predictions of samples (or data / data points in the dataset) that we want to remove (or relabel) from the training data "incorrect".

[0116] In one embodiment, input samples that are consistently predicted incorrectly by multiple models trained on the same training dataset using k-fold cross-validation, i.e., "untrainable", are removed or relabeled from the training dataset (see Algorithm 1 below), and it is desirable that metrics such as accuracy rates for the validation and / or test datasets of each model are biased towards the correct class, where there is at least one correct class (there is no class-specific bias if all classes are noisy classes). It has been proposed that the best chance for an AI model to learn patterns related to the classification (or label) of each sample is the training dataset itself used to train the AI ​​model. This cannot be guaranteed with validation and test datasets, since there may not be representative samples in the training dataset. The reason why incorrectly labeled samples cannot be trained on the training dataset is because they belong to an alternative class that is known (or believed to be) to be the correct class.

[0117] If an AI model correctly trains and classifies samples in the correct class (biasing the accuracy rate of the validation / test dataset to the correct class), the model is less likely to be able to correctly classify mislabeled samples in the noisy class (as they may appear to be samples in the correct class, resulting in incorrect predictions or classifications). There is one case where this argument does not hold, and that is when the AI ​​model is overtrained or overfitted to the dataset. However, this case can be easily detected by a drop in the accuracy rate of the validation and / or test dataset, so these models can be removed from the analysis. The steps in this embodiment are as follows:

[0118] Aggregated Dataset (outside 1) TIFF0007643692000003.tif5114 is from d different data owners (individual datasets (outside 2) Each dataset contains data from the following: (Outside 3) TIFF0007643692000005.tif5114 is split into training and validation sets (and an optional test set) using k-cross validation (KFXV). A set of n model architectures (outside 4) TIFF0007643692000006.tif5114 is trained on the training dataset (n x k models in total) using KFXV. Classes can be identified (or pre-determined) as correct classes or noisy classes based on the particular problem. If there is at least one correct class, the set of learned models is (outside 5) TIFF0007643692000007.tif6114 is selected, and for each model, the model with both the highest correct rate (first priority) and the highest average correct rate (second priority) for the correct class is selected, with a confidence metric such as cross-entropy loss used as a tiebreaker. However, other combinations or prioritizations of metrics, such as confidence-based metrics, may be used as the primary metric. Models with high correct rates for noisy classes should be avoided, as this implies that the AI ​​model has been trained to misclassify data. In the absence of a correct class, the model with the highest average correct rate (again using cross-entropy loss as a tiebreaker) is selected. That is, one or more thresholds for the correct class and a threshold for the noisy class can be defined. Also, while the above example uses the correct rate and the average correct rate, a single metric may be used, or other combinations or prioritizations of metrics may be used. The metric may be a confidence-based metric, such as log loss, which may be used as the primary metric.

[0119] (outside 6) For each dataset in TIFF0007643692000008.tif5114, the training dataset (outer 7) Throughout TIFF0007643692000009.tif5114 (outside 8) Each AI model in TIFF0007643692000010.tif6114 is run on each sample in the training dataset. (outer 9) The AI ​​model's classification (or prediction) for TIFF0007643692000011.tif6114 is compared to its assigned label / class to determine whether the sample was correctly or incorrectly classified.

[0120] The so-called consistency threshold (l opt )(l opt is calculated using Algorithm 2 below, and defines a cutoff threshold for the number of successful predictions, below which the image is considered mislabeled or untrainable, by listing all samples in the noisy class that are consistently predicted as incorrect by the multiple selected models using a heuristic guide to determine an optimal value or window of values ​​for ( S i , below which the image is considered mislabeled or untrainable). A second aid to identifying mislabeled data is that priority is given to samples where the model is "really wrong" (i.e., where the model gives a high incorrect AI score (e.g., when the model should have given the sample a score of 1 for that class, but gave it a score of 0 because it was confident that the sample was of another class)).

[0121] Ignore repeated incorrect predictions from examples in the correct class, because it is unclear whether the examples in the correct class are mislabeled or whether the AI ​​model is continually training to correctly classify mislabeled data, forcing correctly labeled data to always be incorrect.

[0122] We remove or relabel these samples and retrain these models on the "cleansed datasets" using the same network architecture and configuration. We verify that the retrained AI models have improved performance (e.g., accuracy and generalizability) on the same validation and test datasets (indicating an improvement in both data quality and the resulting trained AI models).

[0123] In the case of multiple datasets from multiple data sources (or data owners), perform the data cleansing of (a) above on each sub-dataset so that mislabeled samples can be removed or relabeled from each sub-dataset. Collect multiple sub-datasets for machine learning training. An optional step is to perform the data cleansing of (a) above again on the collected dataset to remove remaining mislabeled samples. Finally, train a machine learning model on the collected and cleansed dataset.

[0124] To express the above methodology in a more algorithmic way, below we show how it works, first in a centralized way and then in a decentralized case. The user can choose the appropriate cleaning function specific to the problem at hand.

[0125] The dataset can be tested for predictive power using Algorithm 1 outlined below (recommended before applying the UDC methods described in Algorithms 2 and 3). Given images xj and (noisy) target labels (Outside 10) Sample with TIFF0007643692000012.tif6114 (Outside 11) Each dataset includes TIFF0007643692000013.tif8114 (Outside 12) First train either a single model or multiple models on TIFF0007643692000014.tif4114. The models trained on this data score the same data but with random labels. (Outside 13) In cases similar to those trained with TIFF0007643692000015.tif7114, the label noise in the data becomes so high that the dataset becomes untrainable. Such datasets are candidates for UDC.

[0126] Algorithm 1, in a slightly different form, shows how to compile individual datasets from a single data source. (Outside 14) TIFF0007643692000016.tif4114 into training, validation and test sets and can be used to determine the model's transferability by comparing the results of the validation and test sets using a confidence metric such as CE loss or a accuracy score such as the mean accuracy. If the correlation or consistency of the results between the validation and test datasets is very low, the dataset (Outside 15) TIFF0007643692000017.tif5114 can be marked as containing low quality data. Then, to address the suspected high label noise, (Outside 16) You can apply the UDC method to TIFF0007643692000018.tif3114 individually. If the label noise is very high (e.g., about 50% of the labels for each class are incorrect) and the UDC method is infeasible, (Outside 17) Consider removing TIFF0007643692000019.tif4114 completely. In such a case, (Outside 18) TIFF0007643692000020.tif4114 may be an untrainable dataset.

[0127] (Dataset (Outside 19) The UDC algorithm for a single data source (having TIFF0007643692000021.tif3114) is shown in the pseudocode in Algorithms 2 and 3. This technique is based on k-fold cross-validation (KFXV), uses multiple model architectures, and identifies noisy labels by taking advantage of the fact that noisy labels are likely to be misclassified by multiple models. By using KFXV, it is ensured that all samples can pass through the same number of UDC algorithms. Algorithm 2 counts and returns the number of (Outer 20) successful predictions per TIFF0007643692000022.tif4114 (Outer 21) which is used as input to Algorithm 3.

[0128] Using Algorithm 3, the images are binned together with the same number of successful predictions (Outer 22) per TIFF0007643692000024.tif7114 to generate a histogram, where bin l contains the images for which the prediction by model l was successful (0 < l < n × k). Then, the cumulative histogram (Outer 23) is used with TIFF0007643692000025.tif6114 to calculate the percent difference operator (Outer 24) per TIFF0007643692000026.tif8114.

[0129] If the number of models is sufficiently large, this measure serves as a good discriminator between good labels, which are less likely to be misidentified and cluster in bins with higher values ​​of l, and bad labels, which are more likely to be misidentified and cluster in bins with lower values ​​of l. The denominator acts as a filter, biasing the measure towards larger bins and avoiding bins that contain very few images. We therefore use the heuristic measure of consistency threshold as a rough guide for distinguishing between good and bad labels. (Outside 25) TIFF0007643692000027.tif6114 is used.

[0130] This consistency threshold is (outside 26) All elements z in TIFF0007643692000028.tif6114 j These elements are then used to identify the images that are consistently predicted incorrectly. (outside 27) New cleansed dataset removed from TIFF0007643692000029.tif4114 (outside 28) TIFF0007643692000030.tif5114 is generated. The steps of Algorithms 2 and 3 can be repeated multiple times until a predefined performance threshold is met or model performance is optimized.

[0131] The UDC algorithm is extended for multiple data sources in Algorithm 4 (UDC-M), based on the same algorithm as for a single data source (k-cross validation), which must first take into account the predictive power of the various data sets. The algorithm is based on d individual data sets. (outside 29) Set consisting of TIFF0007643692000031.tif5114 (Outside 30) Take TIFF0007643692000032.tif4114 as input, where s∈{1..d}. Before applying the UDC methods from Algorithms 2 and 3, we test the predictive power of each dataset using Algorithm 1 to identify untrainable datasets. (Outside 31) TIFF0007643692000033.tif7114 is a candidate for UDC-M because it is within the remaining trainable dataset. (Outside 32) TIFF0007643692000034.tif6114 is as explained below (Outside 33) This is because it can be used to cleanse TIFF0007643692000035.tif6114.

[0132] Algorithm 1 Algorithm 1 - N elements (Outside 34) Given a dataset with TIFF0007643692000036.tif8114 (Outside 35) TIFF0007643692000037.tif3114 is a set of images x j and the (noisy) target label (Outside 36) The predictive power of these learned mappings can then be tested by learning mappings between the training data and the (random) target labels. The learned mappings are then tested and scored using the training data as test data. The performance of these learned mappings is then compared using the same procedure but with (random) target labels. (Outside 37) A similar case was done with TIFF0007643692000039.tif7114, dataset (Outside 38) TIFF0007643692000040.tif3114 is untrainable.

[0133] [Table 1]

[0134] Algorithm 2 Given a dataset (Outside 39) TIFF0007643692000042.tif4114 is a file with N elements (outside 40) Contains a set of TIFF0007643692000043.tif8114, each image x j is the corresponding (noisy) target label (outside 41) It is paired with TIFF0007643692000044.tif6114 (for example, in the case of a binary classification problem, (outside 42) TIFF0007643692000045.tif6114∈{0,1}), where j∈{1..N}. (outside 43) TIFF0007643692000046.tif4114 (excluding a subset used as a blind test set) is the training set (outside 44) k mutually exclusive validation data sets with TIFF0007643692000047.tif4114 (outside 45) TIFF0007643692000048.tif5114, where i∈{1..N} is the cross-validation phase index. (outside 46) Model architecture with TIFF0007643692000049.tif5114 (outside 47) TIFF0007643692000050.tif3114, where m∈{1,..,n} is the set of each dataset. (outside 48) The set of mappings learned using Algorithm A is trained on TIFF0007643692000051.tif5114. (outside 49) TIFF0007643692000052.tif8114 is generated and selected using the confidence metric described above. Each model is based on a different dataset. (outside 50) The dataset is trained on TIFF0007643692000053.tif5114, and dropout is used during training so that the phase index i is included in the learned mappings. Then, we apply these learned mappings to the dataset (outside 51) Testing the entire TIFF0007643692000054.tif3114 and predicting the results (outside 52) Generate TIFF0007643692000055.tif7114 and use this to calculate the success prediction count for each element. (outside 53) TIFF0007643692000056.tif12114, where the model prediction is equal to the noisy target label. (outside 54) TIFF0007643692000057.tif7114 is 1, or (outside 55) TIFF0007643692000058.tif7114, and the rest are 0. All elements (outside 56) Vector containing TIFF0007643692000059.tif6114 (outside 57) TIFF0007643692000060.tif3114 is returned.

[0135] [Table 2]

[0136] Algorithm 3 All elements (outside 58) Vector containing TIFF0007643692000062.tif4114 (outside 59) Given TIFF0007643692000063.tif3114, the elements classified into bin l (outside 60) Histogram counting the number of TIFF0007643692000064.tif4114 (outside 61) TIFF0007643692000065.tif4114 is generated, where l∈{0..L} represents the number of successful predictions, and L=n×k is the total number of models used in the UDC algorithm (Algorithm 2). That is, the histogram (outside 62) The action used to generate TIFF0007643692000066.tif7114 is: (outside 63) TIFF0007643692000067.tif5114 (outside 64) Calculate the total number of elements in TIFF0007643692000068.tif3114, (outside 65) TIFF0007643692000069.tif5114 is a set (outside 66) Returns the size of TIFF0007643692000070.tif3114. Then the cumulative histogram (outside 67) TIFF0007643692000071.tif5114 is calculated and the weighted difference operator (outside 68) TIFF0007643692000072.tif5114 is minimized to the optimal stringency threshold (outside 69) TIFF0007643692000073.tif6114 is determined. The algorithm then creates a list of good labels. (outside 70) returns TIFF0007643692000074.tif7114, zUDC is the threshold (71 outside) TIFF0007643692000075.tif7114 is not satisfied and is therefore identified as a bad label. (outside 72) A new cleansed dataset containing TIFF0007643692000076.tif7114 (outside 73) TIFF0007643692000077.tif4114 can be used to retrain a better performing model. This process can be repeated until a predetermined level of performance is achieved or no further improvement can be achieved with the UDC procedure.

[0137] [Table 3]

[0138] Algorithm 4 d individual data sets (outside 74) Set TIFF0007643692000079.tif5114 (outside 75) Given TIFF0007643692000080.tif3114, first use Algorithm 3 to find the prediction power. (outside 76) has TIFF0007643692000081.tif5114 (outside 77) Identify the set of data sources in TIFF0007643692000082.tif3114, where: (outside 78) TIFF0007643692000083.tif4114 is a set of datasets that have no predictive power and are considered untrainable by themselves. Next, we apply the UDC method to (outside 79) Trainable dataset in TIFF0007643692000084.tif4114 (outside 80) Remove the untrainable samples from each of TIFF0007643692000085.tif6114. These individually cleansed datasets are then combined into a trainable dataset. (Outside 81) The images are then aggregated into a larger dataset containing only TIFF0007643692000086.tif7114. The UDC method can then be optionally applied to the aggregated dataset to produce a final cleansed aggregated dataset. (Outside 82) Generate TIFF0007643692000087.tif6114. Finally, we use the untrainable dataset (Outside 83) Combine with TIFF0007643692000088.tif6114 for a final round of cleansing, (Outside 84) We can remove noisy samples from the otherwise untrainable dataset in TIFF0007643692000089.tif4114.

[0139] [Table 4]

[0140] Algorithm 5 Given a dataset (Outside 85) TIFF0007643692000091.tif3114 is the (labeled) training dataset (Outside 86) TIFF0007643692000092.tif4114 and a separate unlabeled dataset for labeling via the UDL method (Outside 87) Contains TIFF0007643692000093.tif13114. Effectively similar to the UDC method, the UDL method inserts data into the training process and uses training-based inference to confidently identify labels for unlabeled data.

[0141] [Table 5] A series of case studies are now described. The datasets used in the following experiments are image-based and are used within the context of binary classification. In these cases, the mean accuracy is used as the evaluation metric, but it should be understood that other metrics can be used, including confidence-based metrics such as log loss. The results are split into three types of datasets:

[0142] Cats vs. Dogs We establish certain useful relationships using the benchmark (Kaggle) dataset of 24916 cat and dog images. This is because the ground truth is recognizable to the human eye, has a high degree of confidence, and allows obtaining models with accuracy rates close to or above 99%. Synthetic noise is added to this dataset to test the merits of the UDC method under various noise and consistency threshold levels. The heuristic algorithm shown in Algorithm 3 serves as a useful guide for the selection of the consistency threshold. We show that the UDC method is resilient to extreme levels of noise (up to 50% label noise in one class, while the remaining classes are relatively clean) and also to significant symmetric noise (30% label noise) in both classes. However, when the label noise in both classes is 50%, the UDC method fails. In this case, model training is not possible because the model is pulled away from convergence by an equal number of true / false positives / negatives, and such data is uncleanable.

[0143] Pediatric chest x-ray: We use another benchmark (Kaggle) dataset of 5856 chest x-ray images (split into 5232 training and 624 test images) to classify images of children aged 1-5 years as "normal" or "pneumonia". This dataset exhibits different behavior in the training and test sets, with the accuracy rate of the training set dropping sharply when the test set is included. Therefore, we treat the test set as a separate "data source" and clean it using Algorithm 4. We show improved performance even when a small amount of UDC specific noise labels are removed. Moreover, we obtain a significant improvement on the test set when only the training set is cleaned, even if the test set remains a blind set.

[0144] Embryo (day 5) images : Images of embryos can be labeled as “non-viable” or “viable” (for “non-viable” and “viable” classes, respectively). Since identifying embryo viability involves complex factors for which the ground truth cannot be known (see Figure 1), a surrogate for the ground truth must be used. In this work, a “viable” embryo is one that was implanted in a patient and resulted in a pregnancy (heartbeat after 6 weeks). A “non-viable” embryo is one that was implanted in a patient and did not result in a pregnancy (no heartbeat after 6 weeks). Using problem domain knowledge, we know that a “viable” embryo is an accurate ground truth result because it resulted in a pregnancy regardless of the influence of other variables on the outcome, so there is negligible label noise in the viable class. However, a non-viable embryo may not have an accurate ground truth because it may not result in a pregnancy due to other factors (patient, medical, or IVF process factors). Therefore, there is a significant label noise potential in the non-viable class.

[0145] Case Study 1: Cats and Dogs The approach of first testing the effect and removal of synthetic noise on a dataset of cat and dog images is taken for a very important reason: this "easy problem" can serve as a baseline since the ground truth for each image can be manually verified, whereas for harder problems such as disease classification through medical images, this is often not possible. The aim is to use the insights derived from this "easy problem" that can be usefully transferred to the harder problems. In later sections, we use the insights discovered from this "easy problem" to improve the performance of difficult recognition tasks such as detection of pneumonia in chest x-ray images and classification of embryo viability from (day 5) embryo images.

[0146] Pretreatment To ensure a proper baseline for experiments, the dataset is preprocessed in two ways. First, images are manually filtered to remove noise in the image data (clear outliers such as images of houses, i.e., those that do not contain cats or dogs). Second, images are identified by a unique hash key and duplicates are removed from the entire dataset to avoid biased results. After these preprocessing steps, the size of the dataset is 24916 images in the training set and 12349 images in the test set.

[0147] Case Study 1A: Effect of synthetic label noise on model performance To characterize the effect of label noise on model performance, we add synthetic noise (inverted labels) in a systematic way as follows.

[0148] The (preprocessed) dataset is split into three subsets:

[0149] Training Set T train : Total 24916 images (12453 cats, 12463 dogs) Test set T test : Total 12349 images (6143 cats, 6206 dogs) Two types of label noise are introduced.

[0150] Uniform (flipping labels in the same proportions for both classes) Asymmetric (flipping the labels in only one class - if the class distributions between classes are similar, one of the classes can be chosen as the noisy class) Inversion level n i (the proportion of labels flipped for each class i) varies from 0% to 70%.

[0151] In the training set (Outside 88) Only TIFF0007643692000095.tif5114 In the test set (Outside 89) Only TIFF0007643692000096.tif5114 In both the training and test sets (Outside 90) TIFF0007643692000097.tif5114To compare the results, we used the cleansed training set (i.e., n train T = 0 train A baseline accuracy rate is first determined by training an AI model on a cleansed test set (i.e., n test T = 0 test )), the average accuracy rate reaches about 99.2%, with the remaining 0.8% (about 200 images) being made up of difficult-to-classify images such as those shown in Figure 2. The difficult-to-classify images are image 200 of a dog 292 that is easily confused with a cat even by a very accurate model.

[0152] FIG. 3A illustrates a test set T in which uniform label noise is present only in the training data (■) 302 and only in the test set (▲) 303, and is present equally in both sets (●) 304, according to one embodiment. test3B is a plot of the average accuracy of the trained model measured against a test set T where the single-class noise in the training data is only (■) (solid line 311 for cats, dashed line 314 for dogs) and only (▲) in the test set (solid line 313 for cats, dashed line 315 for dogs), and equally (●) in both sets (solid line 313 for cats, dashed line 316 for dogs), in accordance with one embodiment. test Plot of the mean accuracy of the trained model measured against σ. In the case of uniform noise (a) the mean accuracy is the same as the class accuracy (solid line for cats, dashed line for dogs), so in this case the class accuracy is not shown, whereas in the case of symmetric noise (b), the class accuracy shows more useful information and the effect of noise in the training set leads to an opposite behavior to the effect of noise only in the test set.

[0153] The results in Figures 3A and 3B show that train and ntest We show how the generalization error changes as n changes, whether the noise is in the training set only, the test set only (to see if the introduction of synthetic noise leads to the expected linear behavior shown below), or both sets, and whether the noise is uniformly distributed among the classes (Figure 3A) or in only one class (Figure 3B). Here, we show the percentage of images of cats (the correct class), n cat In the case of asymmetric label noise experiments (Figure 3B), it is interesting to note how the class-based accuracy rate depends on the location of the label noise. For example, when the label noise is only in the training set, it confuses the model's concept of cat, so some cats are incorrectly classified as dogs, whereas when the label noise is only in the test set, the "dog" class contains images of cats, so the model is naturally wrong.

[0154] Case Study 1B: Identifying and Removing Label Noise Using UDC To test the UDC algorithm, we added synthetic label noise to a training dataset of 24916 images (no test set was used in this experiment), which was split 80 / 20 into training and validation sets with the following parameters used for each study: (outside 91) TIFF0007643692000098.tif7114 contains the fractional levels of inverted labels for the cat and dog classes, respectively, where 0≦n(%)≦100.

[0155] [Table 6]

[0156] Representative cumulative histograms created using Algorithm 2 are shown in Figures 4A-4D, and the results are shown in Table 2. In Table 2, the inverted and non-inverted labels are explicitly shown. The cumulative histograms H at various stringency levels l for uniform and asymmetric noise levels are i 4A to 4D. FIG. 4A shows the H for a uniform noise level of 30 / 30. i 4B shows the H for asymmetric noise levels of 35 / 05%. i 4C shows the H for uniform noise level in the 50 / 50 case. i 4D shows the H for asymmetric noise levels in the 50 / 05% case according to one embodiment. iThe vertical and diagonal filled columns show the noisy and accurate labels, respectively, while the yellow line shows the percentage error (logarithmic scale and inverted to show maximization instead of minimization). The idea behind searching for a good threshold is to minimize the number of non-inverted labels while maximizing the number of inverted labels (rear vertical column). While the distribution of non-inverted labels (front diagonal filled columns) is similar for the two asymmetric cases and a similar strictness threshold can be used, in the (30, 30) case the distribution of non-inverted labels is wider, leading to a lower strictness threshold being chosen in this case. The plot for the (50, 50) case clearly shows that optimizing the threshold using the heuristic of Algorithm 3 is not possible or at least is unreliable or performs very poorly.

[0157] [Table 7]

[0158] Table 2 shows the improvement rate for several experimental cases after applying the UDC method only once, and all cases have an improvement of more than 20%. (3) ,n (4)}, compared with {n (1) ,n (2)} obtains a higher average accuracy rate after one round of UDC. This is expected because in the asymmetric case, one class remains as the true and correct class, allowing the UDC method to be more robust in identifying mislabeled samples. The amount of improvement is higher in the uniform case because the asymmetric case reaches a very high accuracy rate (>98%) after only one round of UDC, while the uniform case only obtains a 94.7% accuracy rate. This indicates that some noise remains in the uniform case after one round of UDC. After applying one more round of UDC in the uniform case, an even better accuracy rate (99.7%) was obtained than the baseline accuracy rate (99.2%). This is suspected to occur because UDC helps to remove "hard to identify" images that can be predicted incorrectly by many models, thus exceeding the baseline accuracy rate. As shown in Table 2, n (4) For =(70, 70), n (3) = (30, 30). (5) For ,=(50, 50), the method discards all samples from the,opposite noisy class by simply learning to treat one class entirely as,incorrect and the other as correct.,Thus, as might be expected, Figure 4D shows that UDC fails when the,noise level for both classes is 50%.

[0159] Case Study 2: Chest X-ray In this section, the UDC method is tested on a relatively "hard problem": binary classification of pediatric chest x-rays. In the results below, the "normal" class is the negative class with label 0, and the "pneumonia" class is the positive class with label 1. The dataset is split into training and test sets, which appear to have different levels of noise. The results show that the UDC algorithms (Algorithms 1-3) can be used to identify and remove bad samples, improving model performance (using both confidence and accuracy metrics) on previously unseen datasets suspected of having significant levels of (asymmetric) label noise.

[0160] Pretreatment The dataset in this case study is manually filtered in a similar way as in case study 1. No clear outlier images were identified, but some duplicates were found and removed to avoid biasing the results. After preprocessing, the size of the dataset is 5856 images, with 5232 and 624 images used in testing.

[0161] Case Study 2A: Enhancing model performance on a blind test set In this study, instead of synthetically adding label noise to the dataset, the performance of trained models before and after UDC on a blind test set is compared. The metrics used to define performance are cross-entropy loss (CE) and average accuracy (A bal ).

[0162] Figure 5 shows the average accuracy (top) and cross-entropy or log loss (bottom) for various model architectures before UDC (left) and for the ResNet-50 architecture after UDC (right) for varying strictness thresholds for the test set. The shading of the bar represents the performance of the model on the test set when the selected epoch (or model) is the one that resulted in the lowest log loss measured on the test set (hatched) and the validation ("val") set (black). The discrepancy between these two values ​​indicates the generalizability of the model. That is, a model that performs well on one but not the other is not expected to generalize well. We show that this discrepancy improves with UDC.

[0163] Case Study 2B: Test set treated as additional data source Case Study 2A shows that UDC improves model performance even on the blind test set, which is a measure of the power of the UDC method. In this section, we investigate the effect of treating the test set as a separate data source. To this end, the test set is included (or "injected") into the training set, and the effect on the resulting model performance is noted.

[0164] Figure 6 is a set of histogram plots showing accuracy (top) and cross-entropy or log loss (bottom) for various model architectures before (left) and after (right) UDC for varying accuracy thresholds l for the validation set. The color of the bar represents the model's performance on the validation set, which is chosen as the epoch (or model) that minimizes the log loss on the validation set with (shaded) and without (black) the test set included in the training set. We see that performance drops significantly when the test set is included, indicating that the level of label noise in this test set is significant.

[0165] 7 is a histogram of the number of images per accuracy threshold for the test and training sets for normal and pneumonia labeled images according to one embodiment. FIG. 7 highlights two important effects of label noise in the sets: 1) Although representing only 12% of the aggregated dataset, the test set increases the number of noisy labels identified by 100% compared to the number in the training set alone, highlighting the knock-on effect that label noise has on the model's performance; and 2) this shows that false negatives added to the training set "confuse" the model, counterintuitively increasing the number of false positives.

[0166] Figure 6 shows a significant drop in performance on the aggregated dataset compared to the training set. Figure 7 shows suspected asymmetric label noise in the test set, where high label noise in the "normal" class (in the test set) causes more errors in the opposite "pneumonia" class (in the training set), similar to the phenomenon highlighted in Figures 1A and 3B.

[0167] Case Study 2C: Clean-vs-Many Noisy Label Annotation by Radiologists A radiologist evaluated 200 x-ray images, 100 of which were identified as noisy by UDC and 100 were judged to be correctly labeled "clean." The radiologist provided only the images, not the image labels or UDC labels (noisy or clean). The images were evaluated in random order, and the radiologist's assessment of the labels and confidence (certainty) in the labels for each image were recorded.

[0168] Results show that the level of agreement between radiologists' labels and the original labels was significantly higher for clean images compared to noisy images. Similarly, radiologists' confidence in the labels of clean images was higher compared to noisy images. This indicates that for noisy images, the image alone may provide insufficient information to reliably (or easily) assess pneumonia by either radiologists or AI.

[0169] Dataset and methodology A public dataset of pediatric chest x-ray images (associated pneumonia / normal labels) was obtained from Kaggle (5232 images in the training set and 624 images in the test set). In the AI ​​training process, the training set is used to train or create the AI, and the test set is used as a separate dataset to test how well the AI ​​performed in classifying new, unknown datasets (i.e., data that was not used in the AI ​​training process). The UDC method was applied to all 5856 images in the dataset, and approximately 200 images were identified as noisy.

[0170] The above results suggest that images identified by the UDC method as having noisy labels are suspected to have inconsistencies that make their annotation (or labeling) more difficult. Therefore, the level of agreement in pneumonia / normal assessment between different radiologists is expected to be lower for noisy-labeled images than for clean-labeled images, which are easily identified by the AI ​​model and where a relatively high level of agreement between radiologists is expected. The following two hypotheses are formulated and can be directly tested using the (Cohen's) Kappa coefficient test:

[0171] (Outside 92) TIFF0007643692000101.tif6114:The level of agreement between radiologists for noisy labels is different from chance.

[0172] (Outside 93) TIFF0007643692000102.tif6114:The level of agreement between radiologists for polynoisy labels is no different from chance.

[0173] (outside 94) TIFF0007643692000103.tif6114:The level of agreement between radiologists for clean labeling is not much different from chance.

[0174] (Outside 95) TIFF0007643692000104.tif6114:The level of agreement between radiologists for clean labeling is greater than chance.

[0175] An experimental dataset was created by splitting the data into clean and noisy labels as follows, and the two subsets were used in a clinical study to test the above hypotheses and validate the UDC method.

[0176] 200 elements (outside 96) Dataset with TIFF0007643692000105.tif5114 (Outside 97) TIFF0007643692000106.tif3114 is image xj and its (noisy) annotated labels. (Outside 98) I have TIFF0007643692000107.tif5114. This dataset is split into two equal subsets of 100 images each.

[0177] (outside 99) Labels identified as clean by TIFF0007643692000108.tif4114-UDC include: ·Usually 48 Pneumonia 52 (bacterial 39 / viral 13) (outside 100) Labels identified as noisy by TIFF0007643692000109.tif4114-UDC, including: ·Usually 51 Pneumonia 49 (bacterial 14 / viral 35) Dataset (Outside 101) TIFF0007643692000110.tif3114 is a new dataset that has been randomized and given to expert radiologists who are asked to label the images. (Outside 102) We create a TIFF0007643692000111.tif4114 to indicate the level of confidence or certainty of those labels (low, medium and high). This randomization is done to address fatigue bias and biases associated with ordering of the images.

[0178] The level of agreement between the expert radiologists and the original labels was calculated using the Cohen's Kappa coefficient test for the data set. (Outside 103) TIFF0007643692000112.tif4114 and (Outside 104) TIFF0007643692000113.tif5114 is compared.

[0179] Results - Level of match The results of the experiment are shown in Fig. 8, which is a plot of images split into clean and noisy labeled ones, further subdivided into images from the training and test sets, and again subdivided into normal and pneumonia classes. The images are surrounded by solid and dashed lines showing the agreement and disagreement of the assessment between the original and expert radiologists, respectively. The agreement rates are not highly biased between classes or dataset sources, suggesting that the label type (clean vs. noisy) is the most important source of variation.

[0180] Applying the Cohen's Kappa test to the results gives the agreement level for noisy (k≈0.05) and clean (k≈0.65) labels. Figure 9 is a plot of the calculation of Cohen's Kappa coefficient for noisy and clean labels according to one embodiment, showing the null hypothesis (Outside 105) TIFF0007643692000114.tif6114 and (Outside 106) We provide visual evidence that both TIFF0007643692000115.tif6114 are rejected with very high confidence (>99%) and effect size (>0.85). We therefore state that the labels identified as noisy have a level of agreement that is no different from chance. (Outside 107) We note that the labels identified as clean by TIFF0007643692000116.tif5114 and UDC have a level of agreement greater than chance and significantly higher than those of noisy labels. (Outside 108) Both alternative hypotheses in TIFF0007643692000117.tif6114 are accepted.

[0181] Analysis - Confidence With regard to the results displayed above, it is interesting to see yet another level of granularity: the confidence (low, medium, high) with which the labels were rated by expert radiologists. These confidence levels were determined from the radiologists' notes; ratings that included comments indicating serious radiological problems or other confounding variables were labeled with low confidence, ratings that included comments such as "probable", "possible", "not ruled out" were treated as medium confidence, and finally ratings made with relative certainty were labeled with high confidence.

[0182] FIG. 10 is a histogram plot of the levels of agreement and disagreement for both clean-labeled and noisy-labeled images according to one embodiment. FIG. 10 shows that 18 disagreements for the clean labels were low or medium confidence, again suggesting that clean labels are indeed more easily or consistently classified and that both UDCs and expert radiologists are generally confident that these labels reflect the ground truth. Of the 47 disagreements for the noisy labels, 14 are also high confidence, indicating that disagreements for the noisy labels are not only more frequent but also more assertive. FIG. 10 shows a breakdown of the confidence of the expert radiologists' assessments, showing that for the clean labels, even for the few images where the radiologists did not agree with the labels provided in the original dataset, the assessments were confounded by certain variables that reduced the confidence. This is in stark contrast to the noisy labels, where both agreements and disagreements have similar distributions of assessment confidence.

[0183] It is important to state that chest x-rays can have many confounding variables, and it is very common that some x-rays are rather uninformative for a radiologist to confidently assess. It is also important to put the number of noisy labels into context. For the study conducted here, while it was important to keep the number of labels balanced, with both pneumonia and normal classes as well as clean and noisy categories, there were orders of magnitude more clean labels (more than 5700) than noisy labels (less than 200), suggesting that the majority of the entire dataset is very consistent, with the exception of the noisy labels.

[0184] Finally, the type of pneumonia (bacterial, viral, other) was not explored at this level of detail, and is not relevant and beyond the scope of the study. Importantly, this shows that an embodiment of the UDC method can identify image-label pairs that are likely to be difficult to assess and therefore may require more attention. This type of method could be very useful as a screening tool for radiologists, helping radiology clinics triage and focus on suspicious (noisy) images over those that are likely to be easier to assess.

[0185] Improving AI performance with UDC clean datasets In this study, we compared the performance of an AI when trained with the original (uncleaned) X-ray dataset (without UDC) and the UDC-cleaned X-ray dataset (after UDC). The results are shown in FIG. 11A, which is a histogram plot of the average accuracy rate before and after UDC (cleaned data) for varying accuracy thresholds l according to one embodiment. FIG. 11A shows that training the AI ​​on the UDC-cleaned dataset improves both the accuracy rate of the AI ​​and the generalizability (and therefore scalability and robustness) of the AI.

[0186] FIG. 11A shows the results of two accuracy rates for the test dataset. The bars filled with diagonal lines represent the theoretical maximum accuracy rate possible for the test dataset with the AI. This is obtained by testing all AI models trained on the test dataset to find the maximum accuracy rate that can be achieved. On the other hand, the solid black bars are the actual accuracy rates of the AI ​​obtained using a standard method of training and selecting AI models. The standard method trains many AI models using the training dataset (with different architectures and parameters) and selects the best AI based on the performance of the AI ​​model on the validation dataset. Only when the AI ​​is selected, it becomes the final AI that is applied to the test dataset to evaluate the performance of the AI. This process ensures that the AI ​​is not selected or "cherry-picked" to maximize the accuracy rate of the test dataset, which actually occurs when the AI ​​needs to be applied independently to other unknown data blindly. In addition, the difference in accuracy between the diagonal bar (theoretical maximum accuracy rate of the AI) and the solid black bar (actual accuracy rate of the AI) is an indicator of the generalizability of the AI, i.e., whether the AI ​​can work reliably on other unknown data (X-ray images).

[0187] A very important feature highlighted in Figure 11A is the deviation between selecting the AI ​​model during training that performs best on the test dataset (shaded bars) (which can be seen as "cherry-picking" the best results) and selecting the model that performs best on the validation set (solid bars) (a method actually used to select models that are expected to generalize better on new data). This reduction in deviation for different levels of UDC applied to the training set indicates not only that the overall accuracy rate improved for the "best-validated" AI model, but also that UDC makes the AI ​​training process more robust (i.e., generalizability and its scalability). This is evidence that models trained on clean data can perform significantly better (>10%) on blind datasets than models trained on unclean datasets.

[0188] FIG. 11A also shows the accuracy rate given different UDC thresholds. The threshold relates to how aggressively UDC labels data as "bad" (noisy or dirty). A higher threshold removes more potentially bad data from the dataset, potentially resulting in a cleaner dataset. However, setting the threshold too high may result in clean data being erroneously identified as bad data and removed from the clean dataset. The results in FIG. 11A show that increasing the UDC threshold from 8 to 9 improves the accuracy rate of the AI, indicating that more bad data is removed from the clean dataset used to train the AI. However, FIG. 11A shows diminishing returns as the threshold becomes even larger.

[0189] The Dangers of Using Unclean Test Datasets to Report AI Performance In the final part of this example, we use UDC to explore whether the test dataset is clean or if it contains bad data. This is essential because the test dataset is used by AI practitioners to evaluate and report the performance (e.g. accuracy) of the AI ​​so that it can evaluate x-ray images for pneumonia. Too much bad data means that the accuracy results of the AI ​​are not a true representation of the AI's performance.

[0190] The UDC results show that there is a significant level of bad data in the test dataset. To test this, we insert the test dataset into the training dataset used to train the AI ​​and identify the maximum accuracy rate that can be obtained in the validation dataset.

[0191] FIG. 11B is a set of histogram plots showing the average accuracy rate for various model architectures before UDC (left) and after UDC (right) for varying accuracy thresholds l, according to one embodiment. The color of the bars represents the performance of the model on the validation set, with (solid black lines) and without (diagonal lines) the test set included in the training set. FIG. 11B shows that the performance of the AI ​​trained with the aggregated dataset (training dataset + test dataset) dropped significantly compared to the AI ​​trained with only the training set. This suggests a significant level of bad data in the test dataset. This suggests an upper limit to the accuracy rate achievable even for a good (generalizable) model. This is an important point since there are literatures reporting high accuracy rates (~92%) on this chest x-ray dataset. Unless new AI algorithms or AI targeting using medical domain knowledge are used, the high accuracy rate is likely a case of "cherry-picking" the best results rather than training a generalizable model.

[0192] That is, using standard AI training approaches, it is very difficult (or impossible) to get a model to perform well on both validation (or training) and test datasets simultaneously without additional new techniques to extract this information from noisy images. Our results show that by minimizing the removal of bad data from training, more generalizable performance on the test dataset reaches ~87% (see Figure 11A), and by removing a few (<<100) images from the test dataset, accuracy rates above 95% can be achieved on the UDC-cleaned test dataset.

[0193] Case Study 3: Embryo In this case study, the UDC-M algorithm is tested on a “hard problem” that also includes data from multiple sources: images of human embryos taken with an optical microscope, 5 days after IVF, and matched labels from clinical pregnancy data, are vulnerable to label noise in the manner illustrated in Figure 1A. Recall that the reasonable supporting evidence is that embryos measured as “non-viable” (no fetal heartbeat detected) on ultrasound 6 weeks after implantation are more likely to contain label noise, driven by, for example, patient factors, compared to embryos measured as “viable” (fetal heartbeat detected) on ultrasound, which is bad for training.

[0194] Supporting evidence from a demographic cross-section of the dataset compiled across multiple clinic sources can be obtained by examining the number of false positives (FPs) in both embryologist rankings and results in blind or double-blind sets of the trained AI model.

[0195] The following summary shows that higher FP counts naturally occur in younger age groups, for example, when comparing i) patients under 35 years old, ii) patients of all ages, and iii) patients over 35 years old. The interpretation of this fact is that patients at younger ages are more likely to require IVF due to disease or patient factors compared to those with lower embryo viability. Embryologist and AI FP counts have been shown in previous studies to be confident about viability, as indicated by high cross-entropy loss from the model. This confidence in viable embryos can be naturally explained by the predominance of noise in the non-viable class, for example due to patient factors.

[0196] i) Patients under 35 years of age 38.7% of all embryos are non-viable. Of those with available patient records, 63.6% of patients had various patient factors reported in the clinical database. 78.5% of non-viable blind / double-blind embryos are predicted by trained AI to be false positives. Despite conflicting labels from 6-week clinical pregnancy measurements, 83.7% of non-viable blind / double-blind embryos were false positives predicted by embryologists, demonstrating consistency with the AI ​​in that embryos appeared viable.

[0197] ii) Patients of all ages 42.2% of embryos were non-viable. For images with patient records, 57.4% of patients had various patient factors reported in the clinical database. 71.2% of non-viable blind / double-blind embryos are predicted by AI to be false positives 83.5% of non-viable blind / double-blind embryos were false positives as predicted by embryologists, showing consistency with the AI ​​in that embryos appeared viable.

[0198] iii) Patients aged 35 years or older 42.2% of embryos were non-viable. For images with patient records, 57.4% of patients had various patient factors reported in the clinical database. 71.2% of non-viable blind / double-blind embryos are predicted by AI to be false positives 83.5% of non-viable blind / double-blind embryos were false positives as predicted by embryologists, showing consistency with the AI ​​in that embryos appeared viable.

[0199] These results indicate that as patient age decreases, there is a systematic increase in reporting of patient factors, using a crude surrogate measure of the sum of false positives. This is a significant effect that holds for scores obtained in blinded / double-blind sets of multiple clinics from the trained AI model, and also (to a lesser extent) for embryologists. This consistency of false positives between the AI ​​and embryologists suggests that both the AI ​​and embryologists consistently considered certain embryos to be viable despite their nonviable label, and therefore that nonviable embryos may in fact have been viable and were misrecorded due to label noise. To confirm this evidence, we run the UDC-M algorithm on multiple datasets as follows:

[0200] Clinical Embryo Viability Data There were seven independent clinics as data owners that contributed datasets. Data from each owner is referred to as clinic data, and the combination of all data sources is referred to as the aggregated dataset (or simply the dataset). Each clinic data can be split into training, validation and test sets for training and evaluation purposes, and the split datasets are uniquely named to distinguish them from the rest of the sets. The aggregated dataset can also be split into training, validation and test sets for model training and evaluation purposes. In this case, the aggregated data can be referred to as the training set, or simply the training set.

[0201] For simplicity, we denote the names of the clinic datasets as Clinic Data 1, Clinic Data 2, etc. Table 3 summarizes the class sizes and total sizes of the seven clinic datasets, and shows that the class distributions vary significantly among the datasets. There are a total of 3987 images for training and evaluating the models. [Table 8]

[0202] Case study 3A: Single clinic predictive and metastatic testing In this study, we selected the largest (most representative) clinic data, clinic data 6, for class label randomization and transferability testing. The following steps were performed: Randomly split the Clinic Data6 dataset into training, validation and test sets. Randomly assign class labels to all images in the training set, while leaving the validation and test sets alone. Train a deep learning model on the original training set of clinic data6 and investigate the validation results and how the best validation results are transformed into test results. We report the results in four different metrics: overall accuracy rate, average accuracy rate, accuracy rate for the "unviable" class, and accuracy rate for the "unviable" class.

[0203] Table 4 shows the prediction results of the deep learning models trained and evaluated on Clinic Data 6 (trained on either the random class label training set or the original training set of Clinic Data 6). Note that Clinic Data 6 has a skewed class distribution where the size of the "unviable" class is more than twice that of the "viable" class. The first two rows of this table show the best validation results (the second row is considered as the case of randomized training class labels), and the last two rows show the best test results. Observations include:

[0204] When the labels of the training images are randomized, the average accuracy rate for both the validation and test datasets is about 53%, which is close to the 50% accuracy rate expected from a randomized dataset (i.e., a 50:50 chance or a coin toss would make each sample in the dataset correct). However, the overall accuracy rate is a bit higher, about 60% for both the validation and test datasets. The rationale for this is that there are more images in the "non-viable" class than in the "viable" class, and therefore the model was better trained on the images in the "non-viable" class. Therefore, the accuracy rate for the "non-viable" class is much higher than that of the "viable" class in both the validation and test sets, resulting in a higher overall accuracy rate. Here, we can point out that the predictive model is performing well and the distributions of the training, validation, and test sets are similar.

[0205] The predictive ability of this clinic data6 is confirmed, with average accuracy rates of ~76% and ~70% for the validation and test sets, respectively. However, it is possible to improve the accuracy rate by cleansing the dataset and removing mislabeled or noisy data. The transferability test is shown in Table 4, where the two middle rows are the corresponding test results transferred from the best validation results (e.g., running the training model using the same epoch number on both the validation and test sets). The transfer results for the test set (average accuracy rate of ~68%) are similar to the best test results. This means that transferability can be observed with this clinic data. After case study 3A, it can be confirmed that the training model is working properly, and the clinic data is a candidate for further data cleansing using untrainable data cleansing techniques. In the following case study 3B, we conduct predictive power tests on the remaining clinics in the original dataset.

[0206] [Table 9]

[0207] Case study 3B: Predictive power testing of remaining clinics. In this experiment, we repeat the predictive power test for each of the remaining clinics (clinic data). For simplicity, we randomly split each clinic data into a training and validation set. Since we are not performing transferability testing, we do not need to create a test set. The predictive power is expressed via the average correct answer rate on the validation set. Several deep learning configurations are used to learn on each training set and tested separately on each test set. The evaluation metrics for the report include the overall correct answer rate, the average correct answer rate, the correct answer rate of the "unviable" class, and the correct answer rate of the "viable" class, and the average correct answer rate was considered to be the most important (primary) metric for ranking the predictive power of each dataset. In this case, the class-based correct answer rate is used to check whether the correct answer rates are balanced across different classes. However, other metrics such as a confidence-based metric could have been used.

[0208] Table 5 shows the results of evaluating the predictive power of the seven clinic data sets. Clinic data 3 and 4 exhibited the lowest predictive power, while clinic data 1 and 7 exhibited the highest self-prediction ability. As explained in the previous section, a correct answer rate close to 50% is considered to have very low predictive power, which is likely due to high label noise in the data set. These data sets are candidates for data cleansing. The individual predictive power reports (Table 5) may indicate how much data should be removed from each clinic data set. That is, the lower the predictive power, the higher the number of mislabeled data that may need to be removed from the data set.

[0209] [Table 10]

[0210] Case Study 3C: UDC Applied to Aggregated Data Sets In this experiment, all clinic datasets were combined and randomly split into training, validation and test sets. Different deep learning configurations were used for training on each training set. The following steps were performed: -The best model was selected based on both the accuracy rate of viable classes and the average accuracy rate on the validation dataset. Among multiple trained models with different configurations (various network architectures and hyperparameter settings), the best 5 models were selected. However, other metrics could have been used, such as a confidence-based metric (e.g., log loss). -The five selected models were run on the aggregated training set and five output files were generated containing the per-image (or per-sample) accuracy results. The output consisted of the predicted scores, predicted class labels and actual class labels for all images in the training set. - Accumulate 5 output files, including only images from the noisy class (as these are the only images assumed to be potentially mislabeled), and generate one output file for each (non-viable) image in the dataset containing (1) the number of models producing incorrect results (up to 5) and (2) the average incorrect prediction score of these models, which indicates how incorrectly these models predict. - A short list of images was created that contained images that were misclassified (unviable) by multiple models with high inaccurate prediction scores, e.g., 4 or 5 models. These images were considered mislabeled data and were candidates for removal or relabeling. -To cleanse the dataset, the “mislabeled” images in the list were removed from the aggregate training set.

[0211] The following experiments were used to compare validation and testing results of models trained on the original training set and the cleaned (removed list of mislabeled images as described above) training set. The metric used to evaluate the results was the average accuracy rate, but other metrics such as confidence-based metrics (e.g., log loss) could also be used. Multiple model types and hyperparameter settings were used to make the results more representative. There are multiple options for the deep learning architecture. Common approaches include DenseNet, ResNet, and Inception(-ResNet)net. Several settings are used in Table 6. Setting 1 uses the same seed value, DenseNet-121 architecture and training set-based regularization approach, while other hyperparameters were changed for each model run. Similarly, setting 2 uses a uniform regularization method rather than training set-based regularization. Setting 3 fixes the network architecture as ResNet.

[0212] From Table 6, we can see that the cleaned training set improves accuracy over the noisy baseline. Overall, we get a 1%-2% improvement on the validation and test sets, respectively. Note that these are preliminary results. With a more precise process of selecting failed images, we can expect to see a significant improvement in accuracy.

[0213] [Table 11]

[0214] Case Study 3C: UDC Applied to a Training Set of Individual Clinic Data If data owners want to keep their data private and secure and do not allow their data to be moved and aggregated in a centralized location, untrainable data cleansing techniques can be deployed locally to each individual data owner's dataset (i.e., on their local servers). Note that this approach is also applicable in cases where there are no data restrictions or privacy concerns.

[0215] In this experiment, the clinic datasets were treated separately. Each was randomly split into training, validation and test sets. A different deep learning configuration was trained on each clinic training dataset. The following was done: -Multiple models were trained and the best model was selected for each clinic data set using the accuracy results of the validation set. For this embryo prediction problem, a model with high average accuracy and high misclassification rate for non-viable embryo images was selected because we wanted to capture as many noisy labeled images as possible from the non-viable class. -Run these models on their associated training datasets and generate a results file for each image containing the prediction score, predicted class label, and target. -Create a short list of misclassified (multi-noise, i.e. only in non-viable) images for each file. The prediction scores can be used for threshold filter purposes. -These images are removed from the training dataset, then all datasets are aggregated and the best model is retrained on the new cleaned aggregated dataset.

[0216] [Table 12]

[0217] It can be seen from Table 7 that when the model is trained and tested on the cleaned dataset, the improvement is significant across four different metrics.

[0218] The importance of improving data quality and using untrainable data cleansing techniques can be seen in the training graph of a single deep learning training run over multiple epochs.

[0219] Figure 12 is a plot of the test curves for an embodiment of an AI model trained on uncleaned data, shown as dotted 1201 and solid 1202 lines for the non-viable and viable classes, respectively, and two average curves 1203 in dashed lines. Figure 13 is a plot of the test curves for an embodiment of an AI model trained on cleaned data, shown as dotted 1301 and solid 1302 lines for the non-viable and viable classes, respectively, and two average curves 1303 in dashed lines.

[0220] Figures 12 and 13 show the accuracy rates and their averages for the test dataset for non-viable and viable classes for a single training run over multiple epochs for the original and cleansed datasets. Considering training on the noisy (low quality) original dataset (Figure 12), we see that the training is unstable and the most accurate class keeps switching between the viable and non-viable classes. This is seen in the strong "sawtooth" pattern that occurs for the accuracy rates for both classes across epochs. Note that even if the noise occurs mainly in one class, in a binary classification problem such as this one, the difficulty of identifying correct examples in one class affects the model's ability to identify correct examples in the other class. As a result, there are many data points that cannot be easily classified because their labels are inconsistent with the majority of the other examples on which the model was trained. Thus, small changes in the model weights have a large impact on these marginal examples.

[0221] Thus, for a noisy dataset (of unviable classes), as training progresses the model will switch between (1) learning to correctly classify the "correctly labeled" viable classes, resulting in a drop in accuracy for the noisy, mislabeled unviable classes, and (2) learning to correctly classify the mislabeled unviable classes, resulting in a drop in accuracy for the "correctly labeled" viable classes. Given that the correct viable images and the mislabeled unviable images are in fact likely to be of the same class and therefore have the same classification patterns / characteristics, it is understandable that training will become unstable when the model decides to classify these images into one class or the other, resulting in inaccuracies for all of the images in the alternative class, and switching accuracy between classes.

[0222] Considering training on the cleansed dataset (Figure 13), we see that the training is much more stable. For binary classification, the class that gets the higher accuracy rate does not switch between the two classes between epochs, and has a higher overall average accuracy rate (at each epoch and across epochs). Although there is still a certain number of noisy examples in the training, validation and test sets that are difficult to remove with 100% certainty that the correct images have been removed, the improved cleaned dataset starts to reveal which classes are easier to classify, with improved stability. In this case, since the viable classes consistently get a higher accuracy rate after a single cleansing pass is performed, they can be considered as likely to be the cleaner classes overall, and further cleansing can be focused on the non-viable classes.

[0223] This indicates that the untrainable data cleansing technique can indeed remove intermediate labels and noisy data from the dataset, ultimately improving the data quality and thus the performance of the AI ​​model.

[0224] Case Study 4: UDL for Chest X-ray To further test the above mentioned UDL algorithm, we present here the results of an experimental design based on the chest X-ray dataset, where the 200 images selected for expert annotation are further considered here. Using the same 200 images from the above mentioned experimental results, ignoring their original labels, the multi-label UDL algorithm is carried out as follows: The 200 image dataset is inserted into a larger (~5000 images) training set to form two separate datasets: A random label was assigned to each image (e.g., "normal" for a particular image) and Another with each of these labels inverted (i.e., the image is now labeled "pneumonia")

[0225] NOTE: In this experiment, the rest of the test set was ignored, so it can be expected that this will slightly affect the results compared to the original UDC results.

[0226] UDC was performed on both datasets and the results are shown in FIG. 14, which shows a plot of frequency versus number of incorrect predictions.

[0227] Clean Label: While 1401 with the correct (original) label is correctly predicted by all AI models, The one with the wrong (flipped) label 1402 is predicted incorrectly by multiple AI models (a single clean label with a flipped label will not be predicted correctly by all models) Noisy Label: The one with the correct (original) label, 1403, was predicted correctly by multiple AI models slightly more than the one with the correct label, The one with the incorrect (flipped) label 1404 is predicted slightly less accurately than the one above by multiple AI models.

[0228] A fairly large proportion (50%) of the noisy labels are “correctly predicted” by all AI models with their original labels, which is perhaps not too unexpected since AI is trained to learn even from noisy data. The main takeaway is that the difference in the number of incorrect predictions for noisy images does not change much when their labels are inverted (average difference of 2.7 for noisy labels compared to 8.0 for clean labels). This suggests that these images are only predicted correctly by overfitting, and their original labels are not as certain as those identified as having clean labels. In addition, analysis of the level of agreement between the “correctly predicted” noisy labels showed no significant level of agreement with either the expert radiologist or the original annotations, with only 35 of 61 of these images matching the original labels and X of 61 matching the expert radiologist. In conclusion, these results show that UDL techniques can be used to confidently label unknown datasets and are also useful for identifying images with noisy characteristics.

[0229] Various embodiments of the UDC method have been described. In particular, embodiments of the UDC method have been shown to address misclassification or noisy data in a subset of classes or all classes of a dataset. The UDC method uses an approach based on k-cross-validation, where a dataset is divided into multiple training subsets (i.e., k groups) and for each subset (k groups), multiple AI models with different model architectures are trained (e.g., generating n×k AI models). The estimated labels can be compared to known labels, and samples that are consistently incorrectly predicted by the AI ​​model are then identified as bad data (or bad labels), and these samples can be relabeled or removed. In the case of medical data that is removed (e.g., due to low quality information), it can be flagged for detailed analysis by experts, including recollecting the data (e.g., taking another x-ray). Embodiments of the method can be used with datasets from a single source or multiple sources, and even for binary classification, multi-class classification, as well as regression and object detection problems. The method may therefore be used with medical data, in particular medical datasets including images captured from a range of devices such as microscopes, cameras, x-rays, MRIs etc. However, it will be appreciated that the method may also be used outside of a medical environment.

[0230] Furthermore, UDL methods extend the UDC approach to infer unknown labels for previously unknown data, rather than training an AI model. Rather than training an AI model, the AI ​​training process itself is used to determine the classification of previously unknown data. In these embodiments, multiple copies of the unlabeled data are formed (one for each total number of classes) and each sample is assigned a temporary label. These temporary labels can be random or based on a trained AI model (as per standard AI model-based inference approaches). This new data is then inserted into a set of (clean) training data, and UDC techniques are used up to C times to identify which of the temporary labels are definitely correct (not mislabeled) or definitely incorrect (mislabeled). Finally, if the actual label of a new image can be understood (the data in the image is so clean that it does not contain recognizable features), then UDC can be used to reliably identify (or predictively infer) this label or classification. By inserting this unknown data into the training data, the training process itself tries to find certain patterns, correlations and / or statistical distributions in the unknown data in relation to the (clean) training data. As such, the process becomes more targeted and individualized to unknown data because the particular unknown data is analyzed and correlated within the context of other data with known outcomes as part of the training process, and the iterative training-based UDC process itself ultimately identifies the most likely labels for the particular data, potentially increasing both accuracy and generalizability.

[0231] We also presented a series of case studies applying the UDC methodology. The first case study was a simple one where we trained an AI to identify cats and dogs, and then intentionally "injected" bad data into the dataset by randomly flipping the labels of a certain ratio of images. In this study, we found that images with flipped (incorrect) labels were easily identified as mislabeled data (dirty labels). Because of this issue, there were relatively few cases of multiple noisy labels, e.g., when the images were of low quality and indistinguishable, e.g., an image of a dog with cat-like features, the image of the cat was out of focus or not high enough resolution to be recognizable as a cat, or only non-specific parts of the cat were visible in the image.

[0232] The second case study was a more challenging classification problem: identifying pneumonia from chest x-rays (Second Case Study), which is susceptible to subtle and hidden confounding variables. This study further found that UDC was able to identify bad data, and that the primary source of bad data was noisy labels, where the image itself alone does not contain enough information to reliably identify the label. This means that images are more likely to be mislabeled, and in extreme cases, the image does not contain enough information for any evaluation (AI or human) to identify the label.

[0233] The results were validated using expert radiologists. The radiologists assessed 200 x-ray images, 100 of which were identified as noisy by the UDC and 100 of which were judged to be correctly labeled and "clean". The radiologists were only provided with the images, and not with the image labels or the UDC labels (noisy or clean). The images were assessed in a random order, and the radiologists' assessment of the labels and the confidence (certainty) of the labels for each image were recorded. The results showed that the level of agreement between the radiologists' labels and the original labels was significantly higher for clean images compared to noisy images. Similarly, the radiologists' confidence in the labels for clean images was higher compared to noisy images. This indicates that for noisy images, the images alone provide insufficient information to reliably (or easily) assess pneumonia by either the radiologist or the AI.

[0234] Moreover, it was further shown that the performance of the AI ​​to detect pneumonia improved in accuracy and generalizability when the noisy labels were removed from the dataset to create a "clean" dataset for AI training. This suggests that training an AI with noisy data that likely does not contain enough information to classify (label) the data or is mislabeled can confuse the AI ​​and cause it to learn false features associated with pneumonia or normal x-rays. This highlights the challenges of the AI ​​healthcare industry, where datasets are assumed to be complete or clean, and are not due to a range of factors including poor quality data, subjectivity, uncertainty, human error, or mislabeling as a result of intentional (adversarial) attacks. Medical datasets that contain dirty or noisy labels can negatively impact both the accuracy and scalability (generalizability) of the AI, and ultimately the clinic or patients who rely on reliable AI evaluation. Therefore, by using embodiments of UDC, datasets can be cleaned and noisy data can be removed to improve the performance of AI models.

[0235] Embodiments of the UDC method have also been shown to identify high levels of noise in a test dataset of X-ray images. The test dataset is a separate blind "unknown" dataset not used in the AI ​​training process where the performance of the final trained AI is tested. This test dataset is used by AI practitioners to report the accuracy rate of an AI detecting pneumonia from X-ray images. Noise in the test dataset means that the accuracy rate of an AI reported on this dataset may not be truly representative of the accuracy rate of the AI. Some research groups have reported high levels of noise for this dataset. They report very high accuracy rates for their AIs. Therefore, it remains an open question whether their AIs are actually able to extract the correct information from the noisy images and label them better (potentially enabling AIs to be targeted with new AI algorithms or by incorporating additional medical domain knowledge) or whether the high accuracy rates are due to luck or "cherry-picking" an AI that can obtain a better accuracy rate for this particular dataset. Nevertheless, accuracy rates using uncleaned test datasets can mislead researchers or clinicians about the true performance and reliability of the AI, potentially resulting in real-world consequences for clinicians and patients who may rely on the AI ​​if used in practice.

[0236] The above case study demonstrated that even in a "hard" classification problem such as detecting pneumonia from pediatric x-rays, embodiments of the UDC method can be used to effectively identify bad data (or bad labels). Removal of the bad data was used to create a clean training dataset for AI training, resulting in improved AI performance for identifying pneumonia in x-rays. In addition, UDC found that bad data was present in the test dataset used by AI practitioners to test and report AI performance analyzing x-ray images for pneumonia. This means that the reported AI performance may not be the true performance of the AI ​​and could potentially be (unintentionally) misleading. Finally, the bad data identified by UDC for this particular problem was in the multi-noise category that both the AI ​​and radiologists found difficult to confidently label using the available x-ray images. This suggests that there is a higher chance of mislabeling or misdiagnosis as these images alone provide limited information to confidently make a definitive diagnosis.

[0237] Finally, the third case study demonstrated the application of the UDC method to a challenging problem, namely the estimation of embryo viability for IVF involving data from multiple sources (multiple clinics). The UDC method is able to identify and remove intermediate labelled noisy data from the dataset, ultimately improving data quality and thus the performance of the AI ​​model.

[0238] The results herein demonstrate that embodiments of the UDC method have the potential to have a significant impact on the industrial application of AI to challenging problems, such as AI healthcare. Medical data is often "dirty" in nature, and embodiments of the UDC method can be used to effectively clean and train higher performing AI, ensuring that it is accurate, scalable (generalizable), and more reliable for use by clinicians and patients worldwide.

[0239] Further embodiments of the UDC have the added advantage of being able to analyze medical data and identify which images are likely to have a lot of noise (i.e., are difficult to reliably evaluate), to the extent that they can be used as a potential triage tool to direct clinicians to cases that require additional detailed clinical evaluation.

[0240] Embodiments of the UDC method can be used to help AI practitioners clean reference test datasets, which are datasets used to test and report the effectiveness of their AI. Testing and reporting on uncleaned datasets can be misleading regarding the true effectiveness of the AI. The clean dataset after UDC processing allows for a true and realistic representation and reporting of the accuracy rate, scalability, and reliability of the AI, protecting clinicians or patients who must rely on it.

[0241] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips may be referenced throughout the above description and may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0242] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software or instructions, middleware, platforms, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the invention.

[0243] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two, including in a cloud-based system. In the case of a hardware implementation, the processing may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors or other electronic units designed to perform the functions described herein, or combinations thereof. Various middleware and computing platforms may be used.

[0244] In some embodiments, the processor module includes one or more central processing units (CPUs) or graphic processing units (GPUs) configured to perform some of the steps of the method. Similarly, a computing device may include one or more CPUs and / or GPUs. The CPU may include an input / output interface, an arithmetic logic unit (ALU), and a control unit and program counter element that communicate with an input / output device via the input / output interface. The input / output interface may include a network interface and / or a communication module for communicating with a comparable communication module in another device using a predefined communication protocol (e.g., Bluetooth, Zigbee, IEEE802.15, IEEE802.11, TCP / IP, UDP, etc.). The computing device may include a single CPU (core) or multiple CPUs (multi-core) or multiple processors. The computing device is typically a cloud-based computing device using a GPU cluster, but may also be a parallel processor, vector processor, or distributed computing device. The memory is operably coupled to the processor and includes RAM and ROM components and may be provided internal or external to the device or processor module. The memory may be used to store an operating system and additional software modules or instructions. The processor may be configured to load and execute software modules or instructions stored in the memory.

[0245] A software module, also known as a computer program, computer code or instructions, may include a number of source code or object code segments or instructions and may reside on any computer readable medium, such as RAM memory, flash memory, ROM memory, EPROM memory, registers, hard disk, removable disk, CD-ROM, DVD-ROM, Blu-ray disk, or any other form of computer readable medium. In some aspects, the computer readable medium may include a non-transitory computer readable medium (e.g., tangible medium). Additionally, in other aspects, the computer readable medium may include a transitory computer readable medium (e.g., signal). Combinations of the above should also be included within the scope of computer readable media. In another aspect, the computer readable medium may be integral to the processor. The processor and the computer readable medium may reside in an ASIC or related device. The software codes may be stored in a memory unit and the processor may be configured to execute them. The memory unit may be implemented inside or outside the processor, and if implemented externally, may be communicatively coupled to the processor via various means as known in the art.

[0246] It should also be appreciated that modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by a computing device. For example, such a device may be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via a storage means (e.g., a physical storage medium such as RAM, ROM, a compact disk (CD) or a floppy disk), such that a computing device may obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.

[0247] The methods disclosed herein include one or more steps or actions for achieving the described method. The method steps and / or actions may be interchanged without departing from the scope of the claims. That is, unless a specific order of steps or actions is specifically specified, the order and / or use of specific steps and / or actions may be changed without departing from the scope of the claims.

[0248] Throughout this specification and the claims which follow, unless the context otherwise requires, the terms "comprise" and "comprised" and variations such as "comprising" and "comprising" are understood to mean the inclusion of a stated integer or group of integers, but not the exclusion of other integers or groups of integers.

[0249] The reference to prior art in this specification is not, and should not be construed as, an admission that such prior art forms part of the common general knowledge.

[0250] Those skilled in the art will understand that the disclosure is not limited in its use to the particular applications described. The disclosure is not limited in its preferred embodiments with respect to the particular elements and / or features described or depicted herein. It will be understood that the disclosure is not limited to the disclosed embodiments, but is capable of numerous rearrangements, modifications, and substitutions without departing from the scope of the invention as defined and set forth in the following claims.

Claims

1. 1. A computational method for cleaning a dataset for generating an artificial intelligence (AI) model, the method comprising: generating a cleansed training data set, Splitting a training data set into multiple (k) training subsets; For each training subset, training a plurality (n) of artificial intelligence (AI) models on two or more of the remaining (K−1) training subsets and using the plurality of trained AI models to obtain an estimated label for each sample in the training subset for each trained AI model; removing or relabeling samples in each training subset that were consistently incorrectly predicted by the trained AI models; and training one or more AI models using the cleansed training dataset to generate a final AI model; Deploying the final AI model; and A method comprising:

2. The method of claim 1 , wherein the plurality of artificial intelligence (AI) models comprises a plurality of model architectures.

3. For each training subset, training a plurality of artificial intelligence (AI) models on two or more of the remaining (K−1) training subsets includes: For each training subset, training a plurality of artificial intelligence (AI) models on all of the remaining (K−1) training subsets; The method of claim 1 or 2, comprising:

4. Removing or relabeling the samples in each training subset may be done by: Obtaining a count of the number of times each sample in each training subset was correctly predicted, incorrectly predicted, or passed a threshold confidence level by the trained AI models; removing or relabeling samples in each training subset that are consistently predicted incorrectly by comparing the predictions to a consistency threshold; The method of any one of claims 1 to 3, comprising:

5. The method of claim 4 , wherein the consistency threshold is estimated from a distribution of counts.

6. The method of claim 5 , wherein the consistency threshold is determined using an optimization method to identify a threshold count that minimizes a cumulative distribution of counts.

7. Determining the consistency threshold may include: generating a histogram of the counts, each bin of the histogram containing the number of samples in the training dataset that have the same count, the number of bins being the number of training subsets multiplied by the number of AI models; generating a cumulative histogram from the histogram; calculating a weighted difference between each pair of adjacent bins in the cumulative histogram; setting the consistency threshold as the bin that minimizes the weighted difference; The method of claim 6, comprising:

8. After generating the cleansed training set and before generating a final AI model, iteratively retraining the plurality of trained AI models using the cleansed dataset; generating an updated, cleansed training set until a predetermined level of performance is achieved or there are no further samples with counts below the consistency threshold; The method of claim 4 , further comprising:

9. Prior to generating the cleansed dataset, the training dataset is tested for positive predictive power, and the training dataset is cleaned only if the positive predictive power is within a predetermined range, and estimating the positive predictive power includes: Splitting the training dataset into multiple validation subsets; For each validation subset, training a plurality of artificial intelligence (AI) models on two or more of the remaining (k−1) validation subsets; obtaining a first count of the number of times each sample in the validation subset was correctly predicted, incorrectly predicted, or passed a threshold confidence level by the trained AI models; randomly assigning a label or outcome to each sample in the validation subset; For each validation subset, training a plurality of artificial intelligence (AI) models on two or more of the remaining (k−1) validation subsets; obtaining a second count of the number of times each sample in the validation subset was correctly predicted, incorrectly predicted, or passed a threshold confidence level by the trained AI models when the randomly assigned labels were used; estimating the positive predictive power by comparing the first count and the second count; and The method of any one of claims 1 to 8, comprising:

10. The method is repeated for each dataset in the plurality of datasets to generate a plurality of cleansed datasets; Training one or more AI models using the cleansed training dataset to generate a final AI model, generating an aggregated dataset using the plurality of cleansed datasets; training one or more AI models using the aggregated dataset to generate a final AI model; 10. The method of claim 9, comprising:

11. The method of claim 10, wherein after generating the aggregated data set, the method further comprises cleaning the aggregated data set according to a method of any one of claims 1 to 9.

12. After cleaning the aggregated data set, the method further comprises: for each dataset for which the positive predictive power is outside the predetermined range, adding the respective dataset to the aggregated dataset and cleaning the updated aggregated dataset according to the method of any one of claims 1 to 8; The method of claim 11 further comprising:

13. Identifying one or more noisy classes and one or more accurate classes; Further comprising: After training a plurality of artificial intelligence (AI) models, the method includes: selecting a set of models, where the models are selected if the metric for each correct class exceeds a first threshold and the metric in each noisy class is less than a second threshold; obtaining a count of the number of times each sample in the training data set was correctly predicted or passed a threshold confidence level for each of the selected models; 13. The method of claim 1, wherein removing or relabeling samples in each training subset whose counts are below a consistency threshold is performed separately for the noisy class and the correct class, and the consistency threshold is a per-class consistency threshold.

14. The method further includes evaluating label noise in the dataset, the step comprising: Splitting the dataset into a training set, a validation set and a test set; Randomizing class labels within the training set; training an AI model on the training set using randomized class labels and testing the AI ​​model on the validation set and test set; estimating a first metric for the validation set and a second metric for the test set; rejecting the data set if the first metric and the second metric are not within a predetermined range; 14. The method of claim 1 , comprising:

15. and evaluating the metastatic potential of the dataset, the step comprising: Splitting the dataset into a training set, a validation set and a test set; training an AI model with the training set and testing the AI ​​model with the validation set and the test set; estimating a first metric for the validation set and a second metric for the test set for each epoch within the plurality of epochs; estimating a correlation between the first metric and the second metric across the plurality of epochs; and 15. The method of claim 1 , comprising:

16. 1. A computational method for labeling a dataset for generating an artificial intelligence (AI) model, the method comprising: Splitting a labeled training dataset into multiple (k) training subsets, where there are C labels; for each training subset, training a plurality (n) of artificial intelligence (AI) models on two or more of the remaining (k-1) training subsets; obtaining a plurality of label estimates for each sample in an unlabeled dataset using the plurality of trained AI models; repeating the dividing, training and obtaining steps C times; assigning a label to each sample in the unlabeled dataset by using a voting strategy and combining multiple estimated labels for the samples; A method comprising:

17. The method of claim 16 , wherein the plurality of artificial intelligence (AI) models comprises a plurality of model architectures.

18. For each training subset, training a plurality of artificial intelligence (AI) models on two or more of the remaining (k−1) training subsets includes: training a plurality of artificial intelligence (AI) models on all of the remaining (k−1) training subsets, for each training subset; 18. The method of claim 16 or 17, comprising:

19. 19. The method of any one of claims 16 to 18, further comprising cleaning the labelled training data set according to a method of any one of claims 1 to 15.

20. Repeating the dividing step, the training step, and the obtaining step C times comprises: generating C temporary datasets from the unlabeled dataset, each sample in the temporary dataset being assigned a temporary label from the C labels such that each of the multiple temporary datasets is a distinct dataset; Including, repeating the partitioning, training, and obtaining C times includes performing the partitioning, training, and obtaining steps for each of the temporary datasets such that, for each temporary dataset, the partitioning includes combining the temporary dataset with the labeled dataset and then partitioning into multiple (k) training subsets; 20. The method of any one of claims 16 to 19, wherein the training and obtaining steps comprise, for each training subset, training a plurality (n) of artificial intelligence (AI) models on two or more of the remaining (k-1) training subsets, and using the trained AI models to obtain an estimated label for each sample in the training subset for each trained AI model.

21. 21. The method of claim 20, wherein assigning a temporary label from the C labels is randomly assigned.

22. 22. The method of claim 20 or 21, wherein the assignment of a temporary label from the C labels is inferred by an AI model trained on the original, uncleaned training dataset.

23. 22. The method of claim 20 or 21, wherein assigning temporary labels from the C labels is assigned in a random order from the set of C labels such that each label occurs once in the set of C temporary data sets.

24. 24. The method of claim 20, wherein combining the temporary dataset with the labeled training dataset further comprises splitting the temporary dataset into a number of subsets, combining each subset with the labeled training dataset, splitting into a number (k) of training subsets, and performing the training step.

25. 25. The method of claim 24, wherein the size of each subset is less than 20% of the size of the training set.

26. 26. The method of claim 16, wherein C is 1 and the voting strategy is a majority inference strategy.

27. 26. The method of claim 16, wherein C is 1 and the voting strategy is a maximum confidence strategy.

28. 26. The method of claim 16, wherein C is greater than 1 and the voting strategy is a consensus-based strategy based on the number of times each label has been estimated by multiple models.

29. 30. The method of claim 28, wherein C is greater than 1, and the voting strategy counts the number of times each label is estimated for a sample and assigns the label with the highest count that is greater than a threshold amount of the second highest count.

30. 25. The method of claim 16, wherein C is greater than 1 and the voting strategy is configured to estimate a label that is reliably estimated by multiple models.

31. 31. The method of any one of claims 1 to 30, wherein the dataset is a healthcare dataset.

32. The method of claim 31 , wherein the healthcare dataset comprises a plurality of healthcare images.

33. 33. A computing system including one or more processors, one or more memories, and a communications interface, the one or more memories storing instructions for configuring the one or more processors to perform a method according to any one of claims 1 to 32.

34. 33. A computing system comprising one or more processors, one or more memories, and a communications interface, the one or more memories configured to store an AI model trained using the method of any one of claims 1 to 32, the one or more processors configured to receive input data via the communications interface and process the input data using the stored AI model to generate model results, and the communications interface configured to transmit the model results to a user interface or a data storage device.

Images