Machine learning model assessment

Abstract

Disclosed herein is a computer-implemented method which comprises obtaining testing data comprising a plurality of medical images and at least one likelihood value associated with each image. The likelihood values have been assigned by a machine learning model. Each likelihood value is indicative of a likelihood that a feature in the associated image is a malignant feature. The method further comprises determining a first classification data set for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold; comparing the first classification data set to a verified classification data set associated with the plurality of medical images; and determining, based on comparing the first classification data set to the verified classification data set, a performance score associated with the machine learning model.

Description

[0001] Machine learning model assessment

[0002] This disclosure relates to determining a performance score for a machine learning model.

[0003] Background

[0004] Population based screening programs have been shown to reduce cancer mortality, and are utilised widely to diagnose cancer before patients display symptoms. Medical images may have normal features or benign features as well as indeterminate, suspicious or highly suspicious features that could be malignant. Radiologists and radiographers interpret these medical images to identify suspicious features, and may recall patients for additional imaging or biopsy. However, there may also be false positive interpretations of images, that lead to additional, unnecessary imaging tests and biopsies for patients that do not have cancer.

[0005] Double reading can increase cancer detection rates and keep recall rates low, as it improves the sensitivity and specificity of screening. However, this approach is cost and labour intensive. It can also be difficult to achieve consistently due to reader shortages.

[0006] Artificial Intelligence (Al) may offer a solution to improve diagnostic accuracy and decrease workload. Some Al algorithms have been developed for image segmentation for diagnosis of a patient based on an image, for example, in interpretation of screening mammograms. Al can act as a clinical decision support tool to improve detection of suspicious lesions, a triaging tool to reduce workload by excluding true negative examinations, or even as a replacement for a human reader in a double reading workflow.

[0007] As new Al algorithms are created, and existing software products frequently undergo developments and upgrades, it may be difficult to know that these Al solutions are an adequate replacement for human screeners. Thus, strategies are needed for the quality assurance of new algorithm versions, in order to ensure patient safety and to prevent performance drift over time.

[0008] The present invention seeks to address these and other disadvantages encountered in the prior art by providing an improved system for evaluating the performance of a machine learning model. Summary

[0009] According to an aspect, there is provided a computer-implemented method comprising obtaining testing data comprising a plurality of medical images and at least one likelihood value associated with each image, the likelihood values having been assigned by a machine learning model, and wherein each likelihood value is indicative of a likelihood that a feature in the associated image is a malignant feature. The method further comprises determining a first classification data set for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold. The method further comprises comparing the first classification data set to a verified classification data set associated with the plurality of medical images. The method further comprises determining, based on comparing the first classification data set to the verified classification data set, a performance score associated with the machine learning model.

[0010] The method may further comprise outputting the performance score.

[0011] The first classification data set may comprise a label assigned to at least one feature of a medical image based on whether or not the at least one likelihood value is greater than a decision threshold. The label may be indicative of a clinical category.

[0012] The method of any preceding claim, wherein the verified classification data set comprises a verified label indicative that the at least one feature belongs to a clinical category.

[0013] The clinical category may comprise an indication that the at least one feature is a positive result or a negative result.

[0014] The clinical category may comprise an indication that the at least one feature is one of: benign, indeterminate, suspicious, or highly suspicious.

[0015] The first classification data set may comprise an indication that the at least one feature of a medical image is one of: benign, indeterminate, suspicious, or highly suspicious. The verified classification data set may comprise a determination that the at least one feature of a medical image is one of: normal, benign, indeterminate, suspicious, or highly suspicious.

[0016] The verified classification data set may further comprise, for at least one medical image of the plurality of medical images, a boundary surrounding at least one feature of the medical image. The first classification data set may further comprise, for at least one medical image of the plurality of medical images, location data for at least one feature of the medical image, wherein the location is assigned by the machine learning model.

[0017] Comparing the first classification data set to the verified classification data set may comprise comparing the location assigned to the least one feature to a boundary surrounding at least one feature.

[0018] Comparing the first classification data set to the verified classification data set may comprise determining that at least one feature of an image of the plurality of medical images is one of: a true positive; a false negative; a true negative; a false positive.

[0019] A true positive may be based on a determination that the label assigned to the at least one feature matches the verified label, and that the label assigned to the at least one feature is indicative of a positive result.

[0020] A false negative may be based on a determination that the label assigned to the at least one feature does not match the verified label, and that the label assigned to the at least one feature is indicative of a negative result.

[0021] A true negative may be based on a determination that the label assigned to the at least one feature matches the verified label, and that the label assigned to the at least one feature is indicative of a negative result.

[0022] A false positive may be based on a determination that the label assigned to the at least one feature does not match the verified label, and that the label assigned to the at least one feature is indicative of a positive result.

[0023] A true positive may be further based on a determination that the location assigned to the least one feature is within the boundary surrounding at least one feature.

[0024] Determining a performance score may comprise, based on determining that the first classification data set comprises a true positive and a false negative, calculating a sensitivity of the machine learning model. Determining a performance score may comprise, based on determining that the first classification data set comprises a true negative and a false positive, calculating a specificity of the machine learning model.

[0025] Determining a performance score may comprise, based on determining that the first classification data set comprises a true positive and a false positive, calculating a positive predictive value of the machine learning model at the decision threshold.

[0026] Determining a performance score may comprise, based on determining that the first classification data set comprises a true negative and a false negative, calculating a negative predictive value of the machine learning model at the decision threshold.

[0027] Determining a first classification data set for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold, comparing the first classification data set to a verified classification data set associated with the plurality of medical images, and determining, based on comparing the first classification data set to the verified classification data set, a performance score associated with the machine learning model may comprise a routine.

[0028] Determining a performance score may comprise performing the routine at a plurality of decision threshold values, wherein the performance score for each decision threshold value comprises a sensitivity and a specificity associated with the machine learning model at the decision threshold value. It may further comprise generating a receiver operating characteristic curve for the machine learning model based on the performance scores associated with the plurality of decision threshold values. It may further comprise calculating an area under the receiver operating characteristic curve.

[0029] The method may further comprise comparing the performance score associated with the machine learning model to a plurality of performance scores associated with humans for the plurality of medical images.

[0030] The performance score associated with the machine learning model may comprise a sensitivity and a specificity calculated for a given decision threshold value. The method may further comprise calculating a mean performance score for the plurality of performance scores associated with humans, wherein the mean performance score comprises a mean human sensitivity and a mean human specificity. The method may further comprise comparing at least one of: the sensitivity of the machine learning model for the given decision threshold with a mean human sensitivity; and the specificity of the machine learning model for the given decision threshold with a mean human sensitivity.

[0031] The method may further comprise performing the routine at a plurality of decision threshold values, wherein the performance score for each decision threshold value comprises a sensitivity and a specificity associated with the machine learning model at the decision threshold value. It may further comprise comparing the performance score for each decision threshold value to at least one of the mean human sensitivity or the mean human specificity. It may further comprise deter-mining a decision threshold value associated with at least one of: a sensitivity of the machine learning model that is equal to the mean human sensitivity, or a specificity of the machine learning model that is equal to the mean human specificity.

[0032] The method may further comprise outputting a percentile rank based on a percentage of the plurality of performance scores associated with humans that the performance score associated with the machine learning model is higher than.

[0033] The performance score may comprise at least one of: sensitivity, specificity, positive predictive value, negative predictive value, and / or area under the receiver operating characteristic curve.

[0034] The test set may comprise medical images comprising at least one of: architectural distortion features, asymmetrical features, microcalcification features, calcification features, ill-defined mass features, spiculate mass features, and / or well-defined mass features.

[0035] The method may further comprise calculating a second performance score for a subset of the testing data, wherein the performance score for the testing data is improved relative to the second performance score for the subset of the testing data. It may further comprise outputting an indication of an area of a weakness of the machine learning model based on common features of the subset of the testing data.

[0036] The method may further comprise outputting a training set comprising a second plurality of medical images for further training the machine learning model, wherein the second plurality of medical images are selected based on the area of weakness of the machine learning model.

[0037] The method may further performing the routine to calculate a second performance score using a second decision threshold, determining that the second performance score is improved relative to the first performance score, and outputting that the second decision threshold is an improvement relative to the first decision threshold. According to another aspect, there is a provided a system of evaluating a machine learning model for diagnosis, wherein the system comprises a processor and a memory. The system further comprises a receiving module stored in the memory and executed by the processor to receive, from a machine learning model, testing data comprising a plurality of medical images and at least one likelihood value associated with each image, wherein each likelihood value is indicative of a likelihood that the image contains a malignant feature, the likelihood values having been assigned by a machine learning model. The system further comprises a scoring module stored in the memory and executed by the processor to carry out a steps according to any of the methods described herein.

[0038] According to another aspect, there is provided computer readable medium containing instructions that, when executed by a processor, perform the steps according to any of the methods described herein.

[0039] Figures

[0040] Specific embodiments are now described, by way of example only, with reference to the drawings, in which:

[0041] Figure 1 depicts a flow diagram showing human reader inclusion and exclusion criteria.

[0042] Figure 2 depicts a method for outputting a performance score.

[0043] Figure 3 depicts an example of a medical image, specifically a right medio-lateral oblique mammogram showing an 8 mm ill-defined mass.

[0044] Figure 4 depicts an example of a medical image (a mammogram) showing findings by human readers (depicted in dark grey) and the Lunit INSIGHT MMG artificial intelligence algorithm (depicted by white cross).

[0045] Figure 5a depicts an example of a medical image, specifically a left medio-lateral oblique mammogram showing an asymmetrical density.

[0046] Figure 5b depicts a region of interest (white cross) marked on the medical image by the Al algorithm, demonstrating a true positive case. Figure 5c depicts the same medical image in which the Al algorithm has not marked the region of interest in the same breast , indicating a false negative case.

[0047] Figure 6 depicts a histogram showing the distribution of area under the receiver operating characteristic curve (AUC) values for human readers (mean depicted by dark grey line) and the Lunit INSIGHT MMG artificial intelligence (Al) algorithm (depicted by light grey solid line).

[0048] Figure 7 depicts the receiver operating characteristic curve for the Lunit INSIGHT MMG artificial intelligence (Al) algorithm plotted with individual results of the 552 human readers (blue dots).

[0049] Detailed Description

[0050] In overview, and without limitation, the application discloses a computer-implemented method suitable for evaluating a machine learning model. For example, machine learning models that can analyse medical images as part of diagnostic screening for cancer, such as the Lunit INSIGHT MMG artificial intelligence (Al) algorithm. The method comprises obtaining a plurality of medical images (which may be referred to as part of testing data), each of the images having been assigned a likelihood value by a machine learning model (the likelihood values also forming part of the testing data). The likelihood value is indicative of an estimated risk level, for example the value may be a rating or score that indicates a likelihood that a feature in the medical image is malignant. A score of zero may indicate that there are no features identified in the image or no features with a risk of malignancy, whereas a score of 100 may indicate that there is a feature that is at high risk of being malignant.

[0051] The method may also comprise comparing the likelihood value to decision threshold to determine a first classification data set for each image. This may comprise a classification that the image is one of: normal, benign, indeterminate, suspicious, or highly suspicious, or a classification that at least one feature in the image is one of: benign, indeterminate, suspicious, or highly suspicious. Alternatively or additionally, this may comprise a classification that the image indicates that a patient or subject should be recalled or not recalled. In other words, the classification may indicate a positive or negative decision as to what the outcome of the screening should be.

[0052] This first data set can be compared to a verified classification data set the plurality of medical images, the true or verified classification being indicative of a true risk level. Comparing the data sets may comprise comparing a first classification to a verified classification for each image of the plurality of medical images. The true risk level may be determined by a biopsy that has been performed after the medical image was obtained, or by consistent follow up screening to confirm that there were no features of interest in the original medical image. The true classification may also be referred to as a verified classification or a correct classification, or it may be referred to as a ground truth. Based on the comparing, a performance score is assigned to the machine learning model which can be output.

[0053] In some examples, the performance score may be indicative of or based on the sensitivity and / or the specificity of a detection rate of the machine learning model. A receiver operating characteristic curve may be generated based on the specificity and sensitivity at a plurality of decision thresholds, and an area under the curve may be calculated. An example of a receiver operating characteristic curve for a machine learning model is depicted by the dark grey line in Figure 7. The decision threshold used to determine the classification of each medical image may be adjusted or updated, thus changing the sensitivity and specificity of the outcome of the machine learning model. In some examples, a first decision threshold may have a low sensitivity compared to a human screener model. However, by determining a second performance score using a second decision threshold, the sensitivity may be brought in line with a human screener model.

[0054] In some examples, the performance score may be indicative of or based on positive predictive value and / or negative predictive value. Positive predictive value is the ratio of images or subjects correctly diagnosed as positive (e.g., the image was classified as having a malignant feature or that the patient should be recalled and that matched the verified classification data) to all those images or subjects who had a positive result (e.g., the image was classified as having a malignant feature or that the patient should be recalled, regardless of whether or not that matched the verified classification data). In other words, the positive predictive value provides an indication of how likely it is that an image that is classified as needing to be recalled actually needs to be recalled (based on, for example, biopsy of further screening to determine a ground truth). Negative predictive value is the ratio of images or subjects correctly diagnosed as negative to all those who had a negative result. In other words, the negative predictive value provides an indication of how likely it is that an image that is classified as not needing to be recalled actually did not need to be recalled as they were healthy (based on, for example, biopsy or further screening to determine a ground truth).

[0055] The machine learning model may be, for example, a neural network. In some examples, the machine learning model may be a deep learning neural network or a convolutional neural network. In some examples, the machine learning model may be a generative adversarial network (GAN), such as a cycleGAN. The machine learning model may perform image segmentation in order to identify features of a medical image. This may involve outputting a segment map outlining each feature or object in the image. The model may further annotate objects or features with a likelihood value, indicative of a likelihood that a particular feature is malignant. Location data assigned to a feature by the machine learning model may also form part of the testing data and the first classification data set used to evaluate the model.

[0056] In an example, the machine learning model being evaluated could be the Lunit INSIGHT MMG. This model generates: the location of breast cancer in the form of heatmaps and / or contour maps; an abnormality score reflecting the probability of the existence of breast cancer; and assessment of breast density.

[0057] The machine learning model may be used to diagnose cancer without human interpretation, which can reduce the workload of humans who would otherwise have to perform diagnostic screening. It can also be used as an additional double reader, in other words an additional level of screening which can increase cancer detection and improve its accuracy. This is particularly advantageous for complicated, hard-to-conclude cases. These may be classified as BI-RADS 3 or 4 cases. Radiologists may also compare their reading results with the abnormality score or classification of a machine learning model, and use that comparison to decide whether or not to order additional exams such as ultrasound and biopsy. It is therefore advantageous to know that a particular machine learning model and decision threshold for producing classifications of a medical image have the desired sensitivity and specificity, so that clinicians can trust the model in assisting with clinical decisions.

[0058] Due to the significant reduction in time and labour as a result of using a machine learning model for diagnosis, the methods of the present application are extremely valuable to evaluate the ability of the machine learning model to be used as a decision support tool to be utilised by healthcare professionals.

[0059] Turning to Figure 1 , Figure 1 depicts depicts a flow diagram showing human reader inclusion and exclusion criteria. NHSBSP = National Health Service Breast Screening Program, PERFORMS = PERsonal Performance in Mammographic Screening. In an example, each of the N human participants may be associated with test data comprising a plurality of medical images and a classification that they have assigned to each medical image. Optionally, features within each medical image may be identified. Human readers may also be referred to as human screeners of human participants. They may be radiologists, radiographers (technologists with Master’s level training), and clinicians who are not radiologists but work in the field of diagnosis (for example, clinicians who perform breast diagnosis from mammograms).

[0060] When compared to a true or verified classification, true positives, true negatives, false positives and false negatives may be identified. A sensitivity and specificity can therefore be calculated for the testing data for each human participant. An average sensitivity and specificity may be calculated for N human participants in the study. In the example of Figure 1 , there may be two sets of medical images forming the testing data (part 1 and part 2), and only the 552 participants who have provided a classification of both sets of medical images may be included.

[0061] In an example, the testing data may comprise two test sets each consisting of 60 challenging mammograms (with each mammogram comprising two breasts to be analysed). The medical images may be evaluated by a machine learning algorithm. The testing data further comprises at least one likelihood value associated with each image.

[0062] Optionally, the plurality of images may also be evaluated by human readers. In an example, a machine learning algorithm and a total of 552 human readers interpreted both test sets which included 161 / 240 normal, 70 / 240 malignant, and 9 / 240 benign breasts. Advantageously, comparing the performance score for a machine learning model at a decision threshold to a plurality of performance scores for humans can provide improved evaluation of a machine learning model.

[0063] Turning to Figure 2, Figure 2 depicts a method 200 according to the present disclosure.

[0064] At block 202, testing data comprising a plurality of medical images and at least one likelihood value associated with each image is obtained, the at least one likelihood value assigned by a machine learning model. Each likelihood value is indicative of a likelihood that the image contains a malignant feature.

[0065] Optionally, the likelihood value may be a continuous rating for each feature detected within each view of each breast. For example, the continuous rating may be between 0 and 100. This rating quantifies the machine learning algorithm’s level of suspicion that each feature detected within each breast view is malignant, with a higher rating indicating a higher level of suspicion or probability that the feature is malignant. Where no features are detected by the Al breast screening algorithm within either view of a given breast, the lowest rating possible is assigned as the breast-level rating (e.g., “0”). Optionally, there may be more than one likelihood value associated with a single medical image, as there may be a value for each feature of the medical image identified by the machine learning model. Optionally, a single likelihood value may be assigned to each medical image, the single likelihood value comprising the highest rating assigned to any feature within the medical image. In other words, a rating or value may be assigned to each feature, and only the highest forms part of the testing data. If no feature is marked in a medical image, the medical image may be considered or classified as normal.

[0066] In some implementations, the at least one likelihood value may be received from a machine learning model being run by a different processor to the processor performing the method described herein. The machine learning model may be, for example, a neural network such as a deep learning neural network, a convolutional neural network, a generative adversarial network (GAN) or a cycleGAN.

[0067] The plurality of medical images may be classified by a verified source belonging to a clinical category. For example, each image or each feature within each image may be classified as a positive or negative result. Optionally, a positive result may indicate that the subject of the image containing the features should be recalled, and a negative result may indicate that the subject of the image containing the features should not be recalled. Optionally, the clinical category may be one of: benign, indeterminate, suspicious, or highly suspicious. If no feature is marked in a medical image, the clinical category may be normal. Labels indicating the clinical category may be associated with features within a medical image or associated with the medical image itself.

[0068] The medical images may optionally be selected to comprise particular features or have particular characteristics. For example, they may comprise at least one of: architectural distortion features, asymmetrical features, microcalcification features, calcification features, ill- defined mass features, spiculate mass features, and / or well-defined mass features. In some implementations, the plurality of medical images may be selected such that a defined percentage (typically around 35%) correspond to challenging malignant cases such as cancers manifesting as masses, distortions, asymmetric densities, and microcalcifications.

[0069] Optionally, the testing data may further comprise a location assigned to the at least one feature. For example, cartesian coordinates may be provided. The testing data may therefore comprise a plurality of medical images, wherein at least one medical image of the plurality of medical images has at least one feature, each feature having a set of associated location coordinates and an associated likelihood value. In Figure 3, feature 310 is present, and in Figure 4, the location of the feature has been identified by the machine learning model and indicated by a cross 410. The centre of the cross may form part of the testing data. Alternatively, the machine learning model may provide location data forming a boundary around an identified feature.

[0070] At block 204, a first classification data set is determined for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold. The decision threshold may alternatively be referred to as a recall threshold.

[0071] In some implementations, the classification data may comprise a label assigned to at least one feature of the medical image. The label may be indicative of a clinical category, for example there may be two available classifications corresponding to a positive and negative result. The first classification data set may comprise a classification or label assigned to each image of the plurality of medical images. In other words, each image may be classified as above the threshold or below the decision threshold. The first classification may comprise an indication that the medical image should be flagged so that the patient can be recalled (for example, so that a biopsy or further imaging can be performed). This may be referred to as a positive result. If the classification comprises an indication that the medical image should not be flagged for recall, this may be referred to as a negative result. In a clinical environment, this classification may be an indication of the recommended outcome of screening the image. For example, a likelihood value (rated from 0 to 100) of 50 or greater may indicate that a patient associated with the medical image should be recalled for biopsy or further imaging, whereas a likelihood value of less than 50 may indicate that a patient shouldn’t be recalled. The decision threshold may fall anywhere between the upper and lower bounds of the likelihood value scale.

[0072] Optionally, the first classification may comprise an indication that the medical image is one of: normal, benign, indeterminate, suspicious, or highly suspicious. The highly suspicious classification may alternatively be referred to as a malignant classification, as it indicates a very high probability that there is a malignant feature in the image. Optionally, the classification may be a numerical score, for example 1 to 5. Each numerical score may correspond to a range of likelihood values. In some implementations, a first value indicates that the medical image is normal, and the subject should not be recalled; a second value indicates that the medical image is benign, and the subject should not be recalled; a third value indicates that the medical image is indeterminate and the subject should be recalled; a fourth value indicates that the medical image is suspicious and the subject should be recalled; a fifth value indicates that the medical image is highly suspicious and the subject should be recalled. In an example, the decision threshold may be 3, such that a classification of 3, 4, or 5 indicates that the patient should be recalled. The classification of 3 may correspond to a range of likelihood values. These classifications may be determined by comparing the likelihood value to a decision threshold, and to further classification thresholds. The comparing may comprise comparing at least one likelihood value to a decision threshold, wherein being above the threshold indicates that the subject of the medical image should be recalled, and being below the threshold indicates that the subject should not be recalled. If the likelihood value is less than the decision threshold, the likelihood value may be compared to a first classification threshold in order to determine whether the medical image should be classified as normal or benign. If the likelihood value is equal to or greater than the decision threshold, the likelihood value may be compared to a second classification threshold and a third classification threshold in order to determine whether the medical image should be classified as indeterminate, suspicious, or highly suspicious.

[0073] At block 206, for each image of the plurality of medical images, the first classification data set is compared to a verified classification data set associated with the image.

[0074] The verified classification data may optionally be based on biopsy. Additionally or alternatively, the verified classification may be based on further imaging data from the same subject, collected after the medical image included in the testing data. Biopsy or further imaging data may be used to verify that a medical image is normal, benign, indeterminate, suspicious, or highly suspicious. Biopsy or further imaging data may also be used to verify that features within the medical image are benign, indeterminate, suspicious, or highly suspicious. For example, further imaging data can verify that none of the features developed into benign or malignant tumours. In some implementations, the verified classification may be assigned to each medical image based on an expert or an expert panel. In some implementations, the verified classification data comprises a verified label indicative that at least one feature belongs to a clinical category. These clinical categories may be the same as those associated with the classification data set determined by comparing likelihood values to a decision threshold. Optionally, the verified classification data may further comprise boundary data for at least one feature of the image. Optionally, the verified classification data may further comprise information indicating the type of feature, for example masses, distortions, asymmetric densities, and microcalcification.

[0075] The verified classification may be in the same format as the first classification such that they can be compared. For example, if the verified classification comprises a score from 1 to 5, then the first classification also comprises a score from 1 to 5. Alternatively, the verified classification may be in a different format as the first classification. For example, the verified classification may be a binary classification of each medical image as either malignant or non-malignant (or indicating recall or not indicating recall). The binary classification can be compared to a first classification of normal, benign, indeterminate, suspicious, or highly suspicious (or a score of 1 to 5). It would be understood that an indeterminate, suspicious or highly suspicious classification from a machine learning model would indicate that the subject of the medical image should be recalled, and a normal or benign classification indicates that the subject does not need to be recalled.

[0076] Optionally, the method may further comprise determining that the verified classification is at least one of: a true positive; a false negative; a true negative; or false positive. A true positive result may be based on a determination that the first classification is the same as the verified classification, and it is indicative of a positive result i.e. , that the subject of the medical image should be recalled or there is a malignant feature. A true negative result may be based on a determination that that the first classification is the same as the verified classification, and it is indicative of a negative result i.e., that the subject of the medical image should not be recalled or there is not a malignant feature. A false negative result may be based on a determination that that the first classification is different to the verified classification, and it is indicative of a negative result. A false positive result may be based on a determination that that the first classification is different to the verified classification, and it is indicative of a positive result.

[0077] Optionally, wherein the testing data comprises location data, the method may further comprise comparing the location data to a verified boundary for the feature. The verified classification may comprise, for at least one medical image of the plurality of medical images, a boundary surrounding at least one feature of the medical image. The boundary may also be referred to as a hitbox. In some implementations, the boundary may be an outline of at least one feature. Alternatively, the boundary may be a geometric shape that contains the entirety of the at least one feature.

[0078] In Figure 4, the boundary is indicated by pentagon 420. In Figure 5, the boundary is depicted by shape 520. Optionally, if the location of the at least one feature assigned by the machine learning model is inside the boundary, then it may be determined that there is a positive match between the verified classification and the first classification. In some implementations, this determination may be based on both the location of the at least one feature being inside the boundary and a first classification score or category matching the verified classification. In other words, a true positive result may be based on a determination that the first classification is the same as the verified classification (i.e., the labels match), that there is a positive result, and that the location data assigned to the at least one feature is inside the verified boundary or hitbox. At block 208, a performance score associated with the machine learning model is determined based on the comparing at block 206.

[0079] The performance score may be based on at least one of: sensitivity, specificity, and I or an area under a receiver operating characteristic (ROC) curve. Sensitivity and specificity may be calculated for a particular decision threshold. Area under a ROC curve may be calculated by calculating the sensitivity and specificity at a plurality of decision thresholds in order to generate an ROC curve for the machine learning model. An example of a models performance being indicated by the area value under the ROC curve is shown in Figure 6. In Figure 6, the value of the area under the ROC for a machine learning model (the Lunit INSIGHT MMG artificial intelligence algorithm) is shown by the solid light grey line labelled “Al”. Figure 7 also depicts an ROC for an Al, shown as a solid staggered line that changes with sensitivity and (1 - specificity). The solid black line depicts a bell curve showing the normal distribution for a plurality of human readers.

[0080] Optionally, the performance score may comprise a percentile rank of the area under the ROC curve for the machine learning model compared to a normal distribution of a plurality of performance scores comprising areas under ROC curves for a plurality of human readers. The percentile rank may provide an indication of the performance relative to the average human reader.

[0081] Optionally, the performance score may comprises a value between 0.5 and 1.0 representing the AUC ROC score for the machine learning model. This value is indicative of the proportion of correct results (i.e., results that are the same as the verified classification) in a two- alternative forced-choice test (malignant or non-malignant), where the number of malignant and non-malignant cases balanced. In this implementation, the performance score may indicate the sensitivity and specificity of the machine learning model over a plurality of different decision thresholds.

[0082] In implementations in which the scale of likelihood value for which increasing suspicion is conveyed by a higher value, increasing the decision threshold will decrease sensitivity whilst increasing specificity. As such, the value of the decision threshold may impact the performance score assigned. The receiver operating characteristic (ROC) curve is a plot of sensitivity on the y axis against (1 - specificity) on the x axis, as depicted in Figure 7. The area under the ROC value therefore quantifies the ability of the machine learning model to discriminate between malignant and non-malignant cases. The performance score may comprise the sensitivity and / or the specificity of the machine learning model. The sensitivity may be calculated as a percentage based on the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN):

[0083] TP Sensitivity = - x 100 TP + FN

[0084] The specificity can also be calculated as a percentage based on the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN):

[0085] TN Specificity - x 100

[0086] K JTN + FP

[0087] Sensitivity may also be known as a true-positive rate or probability of detection. The falsepositive rate may also known as probability of false alarm and equals (1 - specificity).

[0088] Optionally, at block 210, the performance score may be output.

[0089] Outputting the performance score may comprise displaying the performance score. Optionally, the output may further comprise indicating an area of weakness of the machine learning model. For example, the sensitivity of the machine learning model for a particular type of image (e.g., microcalcifications) may be lower than its overall performance score. There may be common features of the plurality of medical images that allow this area to be identified. These can be features present in the image itself, the view of the image, or other information about the images such as the type of machine used for imaging or the manufacturer of the machine.

[0090] Optionally, the method may further comprise providing a training set of medical images. This training set may be used to further train the machine learning model in the area of weakness indicated (e.g., microcalcifications). For example, the training set may include medical images comprising a feature with associated verified classification data indicating that the feature is a microcalcification or a feature that may be mistaken for a microcalcification. The verified data classification set may comprise a label indicative of a clinical category. For example, a positive result (e.g., malignant) or a negative result (e.g., not malignant). The clinical category may also comprise an indication of a type of positive result (e.g., microcalcifications, masses, distortions, asymmetric densities). Optionally, the method may further comprise calculating a second performance score using a second decision threshold. This calculating may optionally comprise determining a second first classification for each image of the plurality of images by comparing, for each image, the at least one likelihood value to the second decision threshold. Increasing the decision threshold will decrease sensitivity whilst increasing specificity. Similarly, decreasing the decision threshold will increase sensitivity while decreasing specificity. In some implementations, the method may comprise providing pairs of sensitivity and specificity values at a plurality of different decision thresholds. A receiver operator characteristic curve may be output, demonstrating how the decision threshold affects sensitivity and specificity.

[0091] The method may optionally comprise determining that the second performance score is improved relative to the first performance score. This may comprise providing the change in one of: specificity, sensitivity, and I or area under the ROC curve. This change may be presented as an absolute change, or as a change in percentile rank, for example. The method may optionally further comprise outputting that the second decision threshold is an improvement relative to the first decision threshold. This determining may be based on maximising sensitivity and / or specificity. As depicted in Figure 7, the medium grey dot depicting the threshold matching average human specificity may be considered an improvement over the light grey dot depicting the developer’s recommended threshold as sensitivity is increased. Improvement of the performance score may be increased sensitivity and / or decreased specificity. It can be determined that a performance score for a subset of the images have a worse performance score i.e. , decreased sensitivity and / or specificity and / or PPV and / or NPV relative to the entire plurality of images. This may be used to identify an area of weakness.

[0092] Figure 3 depicts a right medio-lateral oblique mammogram. The Unadulterated mammogram shows an 8 mm ill-defined mass (pointed to by the arrowhead, feature 310). Following biopsy, this feature was determined to be a histological grade 2 ductal carcinoma of no special type.

[0093] Figure 4 depicts a mammogram shows findings by human readers (depicted in dark grey, for example features 430) and the Lunit INSIGHT MMG artificial intelligence algorithm (depicted by a white cross, feature 410). Each dark grey dot is the mark placed by an individual human reader on a perceived abnormality when the PERFORMS case was read. A region of interest (depicted by a pentagon, feature 420) has been annotated by the Personal Performance in Mammographic Screening scheme organizers and their expert radiology panel. This region of interest forms a boundary or hitbox. The Al has correctly marked the region of interest in the right breast for recall. Figure 5a depicts a left medio-lateral oblique mammogram. The unadulterated mammogram shows an asymmetrical density (feature 510 pointed at by the white arrow head). Following biopsy, this feature was determined to be a histological grade 2 ductal carcinoma.

[0094] Figure 5b depicts that the Al has correctly marked the region of interest in the left breast for recall (marked by the white cross, feature 520) when set at a recall threshold of >2.91 to match average human specificity. This image therefore demonstrates a true positive case.

[0095] Figure 5c depicts that the Al has not marked the region of interest in the same breast when set at a recall threshold of >3.06. This image therefore indicates a false negative case. Dark grey dots indicate findings identified by the human readers. This demonstrates how modifying the threshold for recall can impact the sensitivity of the Al model.

[0096] Figure 6 depicts a histogram showing the distribution of area under the receiver operating characteristic curve (AUG) values for human readers (mean depicted by dark grey line) and the Lunit INSIGHT MMG artificial intelligence (Al) algorithm (depicted by light grey solid line) interpreting mammograms at the breast level (n = 240) for recall or return to normal screening. The black, dashed lines represent the lower and upper bounds of the AUG 95% confidence interval (Cl) and the dot-and-dash lines represent the 25th and 75th percentiles. The interquartile range (IQR) is measured between the 25th and 75th percentiles.

[0097] The average AUC for the human readers shown in Figure 6 is 0.88 (95% confidence interval (Cl): 0.81 , 0.95), and the AUC for the machine learning model shown in Figure 6 is 0.95 (95% confidence interval (Cl): 0.90, 0.97). It can therefore be determined that the AUC for the machine learning model is at the 96.8thpercentile against the range of human readers, and there is no observable difference between the mean AUC for human readers and the mean AUC for the machine learning model. The Z value is +1.46 and the P value is 0.15.

[0098] In some implementations, the performance score based on an area under the ROC curve for a machine learning model can be compared to a plurality of performance scores based on areas under ROC curves. The plurality of performance scores may be based on a plurality of human readers, for example working radiographers who participate in regular testing. Additionally or alternatively, the plurality of performance scores may be based on other machine learning models, or previous versions of the same model. The method may further comprise calculating a Z-score for the machine learning model. This score provides an indication of where the machine learning model lies on a bell curve of normally distributed performance scores. For example, in Figure 6, the machine learning model being tested is in the 96.8thpercentile.

[0099] The performance score associated with the machine learning model may be compared to a plurality of performance scores associated with humans. Optionally, the performance scores may be for the same test set of a plurality of medical images. In some implementations, it may additionally or alternatively be compared to a performance score associated with a second machine learning model. The comparison may comprise calculating a mean performance score for the plurality of performance scores associated with human readers, such as a mean sensitivity and a mean specificity.

[0100] These mean values may be compared to the sensitivity and specificity of the machine learning model calculated for a given decision threshold. For example, the decision threshold may be selected such that the sensitivity of the machine learning model is equal to the mean human sensitivity. The specificity of the machine learning model can then be compared with a mean human specificity. Likewise, the decision threshold may be selected such that the specificity of the machine learning model is equal to the mean human specificity. The sensitivity of the machine learning model can then be compared with a mean human sensitivity. Optionally, the method may further comprise outputting a result. For example, an optimised decision threshold may be output. In some implementations, selecting the decision threshold may comprise performing the method 200 at a plurality of decision thresholds.

[0101] In some implementations, the plurality of performance scores associated with humans may be based on humans who have been tested using the same plurality of medical images as the machine learning model. In other words, the testing data comprises a plurality of medical images that have also been used to assess the performance of human readers. Advantageously, the comparison and ranking of the performance score associated with the machine learning model is more accurate if it is based on comparable scores from humans.

[0102] Figure 7 depicts the receiver operating characteristic curve for the Lunit INSIGHT MMG artificial intelligence (Al) algorithm plotted with individual results of the 552 human readers (shown as dots). The y-axis represents sensitivity (cancer detection rate i.e., true positive rate) and the x-axis represents specificity (false positive rate).

[0103] The cancer detection and false positive rates for the Al at a recall threshold score of >10.38 (an example of the developers recommend threshold), >3.055 (an example of a threshold matching average human sensitivity), and >2.905 (an example of a threshold matching average human specificity) are represented by a light grey, dark grey, and medium grey dots, respectively. In some implementations, an improvement may comprise one of: increasing sensitivity, decreasing the false positive rate, and / or proximity to average human sensitivity or specificity.

[0104] As depicted in Figure 7, the medium grey dot depicting the decision threshold matching average human specificity is different to the light grey dot depicting the developer’s recommended threshold, as the sensitivity is higher when matching human specificity. The dark grey dot depicting the decision threshold matching average human sensitivity is also different.

[0105] In an example, no difference was observed between the performance of 552 human readers and Al when interpreting test-sets from an external quality assurance assessment scheme (PERFORMS).

[0106] Key Results:

[0107] No difference in performance was observed between artificial intelligence (Al) and human reader detection of breast cancer in 120 exams from two PERFORMS test sets (AUG, 0.93 and 0.88, respectively; P = .15).

[0108] Using Al score recall thresholds that matched mean human reader performance (90% sensitivity and 76% specificity), the Al showed no difference in sensitivity (91%; P = .73) or specificity (77%; P = .85) compared to human readers.

[0109] The Personal Performance in Mammographic Screening (PERFORMS) scheme is used by the United Kingdom National Health Service Breast Screening Program (NHSBSP) to regularly assess reader performance. Whether this scheme can assess the performance of artificial intelligence (Al) algorithms developed for breast cancer detection is unknown.

[0110] A purpose of the method described herein is to compare the performance of human readers and a commercially available Al algorithm interpreting PERFORMS test sets.

[0111] In this retrospective study, two PERFORMS test sets each consisting of 60 challenging mammograms, were evaluated by human readers between May 2018 - March 2021 and evaluated by an Al algorithm in 2022. The Al considered each breast separately and assigned a suspicion of malignancy score to features detected. Performance may be assessed using the highest score and multiple recall score thresholds. Performance metrics may include sensitivity, specificity, and area under the receiver operating characteristic curve (AUC), and can be calculated for the Al and human readers.

[0112] In an example, a total of 552 human readers interpreted both PERFORMS test sets which included 161 / 240 normal, 70 / 240 malignant, and 9 / 240 benign breasts. No difference was observed at the breast level between the AUC for Al and human readers (0.93 and 0.88, respectively P = .15). Using the developer suggested recall score threshold, no difference was observed for Al verses human reader sensitivity (84% and 90%, respectively; P = .34), but the specificity of the Al was higher (89%) than that of the human readers (76%; P = .003). Using recall thresholds to match mean human reader performance (90% sensitivity and 76% specificity), the Al showed no difference in performance with a sensitivity of 91% (P =. 73) and specificity of 77% (P = .85).

[0113] No difference was observed in the diagnostic performance of human readers and an artificial intelligence algorithm interpreting mammograms from the Personal Performance in Mammographic Screening scheme.

[0114] Population based screening programs have been shown to reduce breast cancer mortality. Mammographic screening does not detect every breast cancer and so women present symptomatically between screening rounds with interval cancers. False positive interpretations also occur, with women without cancer recalled unnecessarily for additional imaging tests and biopsy. In an effort to improve the sensitivity and specificity of screening mammography many European countries undertake double reading as this has been shown to increase cancer detection rates by 6 -15% and keep recall rates low. This approach is labor intensive and can be difficult to achieve due to reader shortages. Artificial Intelligence (Al) may offer a solution to improve diagnostic accuracy and decrease workload.

[0115] Al algorithms have been developed and have gained regulatory approval for the interpretation of screening mammograms. Al could act as a clinical decision support tool to improve detection of suspicious lesions, a triaging tool to reduce workload by excluding true negative examinations, or even as a replacement for a human reader in a double reading workflow. Retrospective studies as well as a recent meta-analysis have suggested that Al can achieve or even exceed human reader cancer detection performance. Although not currently ready for deployment outside of clinical trials, it seems increasingly likely that Al will eventually play a part in the interpretation of screening mammography. Once deployed in everyday practice, the ability to provide on-going monitoring of the performance of these algorithms will be crucial to their long-term success. All software products undergo frequent developments and upgrades and the same is likely to be true for Al. Thus, strategies will need to be devised for the quality assurance of new algorithm versions to ensure patient safety and to prevent performance drift over time.

[0116] In the United Kingdom, the National Health Service Breast Screening Program (NHSBSP) measures the performance of all readers who interpret screening mammograms. Measuring individual reader performance has the potential to improve service quality by informing screening-service managers of underperforming individuals and highlighting areas that would benefit from more tailored training (15). This is achieved through a combination of individual reader real-life performance data and the use of an external performance assessment scheme, Personal Performance in Mammographic Screening (PERFORMS). PERFORMS is a requirement for all readers in the NHSBSP and has been used to test readers in this program for over 30 years. Each PERFORMS test consists of sixty challenging exams from the NHSBSP with abnormal, benign, and normal findings. Readers record their opinions for each case on a web-based reporting platform and, once the set is completed, are given individual feedback including a comparison with all other readers.

[0117] Performance metrics could be collected for Al using methods similar to those already used for human readers. As the PERFORMS scheme is undertaken twice per year by readers in the UK and some international readers, this scheme may be useful in assessing the performance of Al compared to a large number of human readers. The aim of this study was thus to use test sets from the PERFORMS external quality assurance scheme to assess the performance of a commercially available Al algorithm and compare it to the performance of human readers undertaking the same test sets.

[0118] Two consecutive PERFORMS test sets, each consisting of challenging mammograms from 60 patients, were assessed by readers in the UK as part of routine quality assurance for the NHSBSP. The first test set was undertaken between May 2018 - September 2018 and the second between August 2020 - March 2021. The same two sets were used to evaluate the performance of the Al in August 2022.

[0119] Any reader in the UK who participated in PERFORMS as part of the NHSBSP was considered for inclusion in this study. The NHSBSP routinely invites women between the ages of 50 and 70 years to undergo two view, two-dimensional digital mammography every three years. Readers included board certified radiologists, radiographers (technologists with Master’s level training), and breast clinicians (physicians who are not radiologists but work in the field of breast diagnosis). All readers who participate in PERFORMS are required to read a minimum of 5000 mammograms per year. Further details of readers in the NHSBSP, along with information about their qualifications and training, can be found elsewhere. Readers were excluded if they did not complete both consecutive PERFORMS test sets (see Figure 1).

[0120] The mammograms for the PERFORMS test sets were derived from a pool of anonymized 2D full field digital mammograms contributed by screening centers from across the NHSBSP. The mammograms in each set were chosen by the scheme organizers and their expert panel of three breast radiologists, each with at least 20 years of experience interpreting screening mammography and compliant with the NHSBSP reading standard of 5000 mammograms per year. The test sets consisted of normal, benign, and malignant cases. The PERFORMS sets are heavily enriched with challenging malignant cases (typically around 35%) and include cancers manifesting as masses, distortions, asymmetric densities, and microcalcifications. PERFORMS sets were uploaded to the picture archiving and communication system at each screening center for interpretation. Every reader recorded their findings, including the location of any abnormalities identified, on a password protected website. Further details of PERFORMS can be found elsewhere.

[0121] Human Reader Image Evaluation and Scoring

[0122] The final pathology for all malignant and benign cases was confirmed via biopsy and normal cases had at least 3 years of mammographic follow-up, providing the ground truth for reader performance analysis. When reviewing PERFORMS cases, readers scored each breast separately on a scale from 1 to 5 according to the UK Royal College of Radiologists classification system: 1 = normal, 2 = benign, 3 = indeterminate, 4 = suspicious, and 5 = malignant I highly suspicious. A score of 3, 4, or 5 indicated that the reader wished to recall the case for further investigation, while a score of 1 or 2 indicated that the reader thought recall was not required with an outcome of return to routine screening. For all PERFORMS cases, the reader score for each breast was compared to the ground truth.

[0123] The Al algorithm used was a commercially available product (Lunit INSIGHT MMG, version 1.1.7.1 ,). All images were analyzed using Lunit software installed on the author’s (YC) local server at the University of Nottingham. Lunit had no access to the cases before, during, or after the study. The Al acted as an independent reader of cases. The Lunit Al algorithm provided scores which rated suspicion of malignancy against each feature detected on a scale of 0 (low) to 100 (high). The highest rating given to a feature detected within each breast was taken as the overall score for that breast and compared to the ground truth. As each breast was scored separately, an abnormality requiring recall had to be localized to the correct breast by the Al (Figures 3 and 4). When no features of interest were detected, a breast-level score of zero was assigned.

[0124] All analyses were carried out at breast level rather than at lesion level or patient level. To compare the performance of the Al to the binary decision of the human readers (recall or return to normal screening), the Al output was dichotomized based on the manufacturer suggested recall threshold: >10, determined during product development to give a sensitivity of 90%. Other recall thresholds for the Al were tested to match average human reader sensitivity and specificity. A priori power analysis to detect a medium-sized effect (odds ratio 3.5 or 0.29) in terms of cancers missed (i.e. , 1 - sensitivity) by the Al and the average human reader was 80%.

[0125] Area under the receiver operating characteristic curve (AUC) analysis.

[0126] Receiver operating characteristic curves were generated for the Al and human readers. The AUC was calculated using the trapezoidal method. Comparisons between the AUC score for the Al , and the distribution of AUC scores for the population of 552 human readers, were based on conversion of the raw difference between the Al score and mean human score to a z-score, with its related p-value taken from the normal distribution.

[0127] Sensitivity and Specificity.

[0128] The performances of the Al algorithm and human readers were also compared using sensitivity (cancer detection rate) and specificity (true negative rate). The formula for the calculations can be found elsewhere (21).

[0129] Al and average human reader performance analysis.

[0130] Comparisons between human reader and Al performance (at the different recall thresholds) in terms of sensitivity on the 70 malignant breasts and specificity on the 170 cases breasts were conducted as odds ratios. For these analyses the mean human sensitivity was used to determine the number of true positives and false negatives; whilst mean human specificity was used to determine the number of false positives and true negatives. The 95% confidence intervals and P values for the odds ratios were calculated as described by Kirkwood and Sterne, 2003 (22). Each of the analyses described above were repeated to look at radiologists as a group, and at radiographers and breast clinicians as a non- radiologist group. Statistical calculations were performed using IBM SPSS Statistics (version 27.0) statistical software by author ID, who has 18 years of experience conducting statistical analyses. All statistical tests were 2-sided. For all analyses the a level for statistical significance was < .05.

[0131] Of the 815 NHSBSP readers who participated in the PERFORMS scheme during the study period, 263 were excluded as they did not complete both PERFORMS test sets (Figure 1). Therefore, 552 human readers were ultimately used for the comparison with Al, including 315 (57%) board certified radiologists and 237 non-radiologist readers made up of 206 (37%) radiographers, and 31 (6%) breast clinicians. Table 1 shows radiological and pathological features of each of the 120 PERFORMS cases. With each breast treated separately, there were 161 / 240 (67%) normal breasts, 70 / 240 (29%) malignant breasts, and 9 / 240 (4%) benign breasts. Masses were the most common malignant mammographic feature (64.3% [45 / 70]) followed by calcifications (12.9% [9 / 70]), asymmetries (11.4% [8 / 70]), and architectural distortions (11.4% [8 / 70]). The mean size of malignant lesions was 15.5 mm ± 9.2 [SD],

[0132] Table 1

[0133] Table 1 depicts radiological and pathological features of mammograms included in two Personal Performance in Mammographic Screening test sets, analyzed at the breast level. In this example, categorical data is presented as proportional for two sets of 60 images (with each image comprising two breasts). The percentages are shown in parentheses and continuous data is presented as the mean and standard deviation.

[0134] Performance of Human Readers and Al using Al developer’s Recommended Recall Score The average AUC was 0.88 (95% confidence interval [Cl]: 0.81 , 0.95) and the AUC for Al was 0.93 (95% Cl: 0.90, 0.97) (Figure 3). The AUC for the Al was at the96.8th percentile against the range of human readers, but we did not observe a difference between the mean AUC for readers and the AUC for the Al (z = +1.46, P = .15).

[0135] The mean sensitivity and specificity of the human readers was 90% (95%CI: 83%, 97%) and 76% (95%CI: 70%, 83%), respectively (Table 2a). In Figure 4, the sensitivity (cancer detection rate) and 1 - specificity (false positive rate) of each of the 552 individual human readers is plotted with the receiver operating characteristic curve of the Al model. Of the 552 human readers, 52% (287) fell above, 36% (198) fell below, and 12% (67) fell on the receiver operating characteristic curve of the Al model. Figure 4 also shows the performance of the Al algorithm at the recall threshold suggested by the developer (yellow dot). Their suggested threshold score of 10.0 could not be modelled as the Al did not assign a score of exactly 10 to any cases, given the nature of the empirical receiver operating characteristic curve a threshold of >10.38 was therefore established. We found no evidence of a difference between the Al sensitivity (84% [95% Cl: 76%, 93%]; 59 / 70 malignant cancers detected) and the sensitivity of human readers (90% [95% Cl: 83%, 97%]; 63 / 70 malignant cancers detected; P = .34). The Al specificity was 89% (95% Cl: 84%, 94%) which was higher than the of the average human reader specificity of 76% (95% Cl: 70%, 83%, odds ratio, 0.40, P = .003).

[0136] Table 2a

[0137] Table 2a shows performance metrics of artificial intelligence and the average (mean) human reader at different recall thresholds (also referred to as decision thresholds). TP = true positive, TN = true negative, FN = false negative, SE= standard error, Cl = confidence interval.

[0138] Performance of Al using Recall Scores Matched to Human Reader Sensitivity and Specificity Al sensitivity matched the average human reader when the threshold for recall score was set at >3.06 (Figure 4, red dot). At this threshold, Al detected 63 / 70 cancers and missed 7 cancers (which according to radiological feature type included 5 masses, 1 region of calcification and 1 region of asymmetry). This Al recall score yielded a specificity of 77% (95% Cl: 71%, 83%) which did not differ from the specificity of the human reader (76% [95% Cl: 70%, 83%]; P = .85) (Table 2a). Al specificity matched that of the average human reader at a recall score of >2.91 (Figure 4, green dot). This Al recall score yielded a sensitivity 91% (95% Cl: 85%, 98%), but we did not find evidence of difference between this and the average human reader sensitivity of 90% ([95% Cl: 83%, 97%]; P = .73) (Table 2a). The impact of modifying the Al threshold for recall on performance is illustrated in Figure 5.

[0139] Performance metrics of the Al compared to radiologists, and non-radiologists

[0140] The mean sensitivity of radiologists and non-radiologists was 91 % (95% Cl: 84%, 97%) and 89% (95% Cl: 81%, 96%) respectively (Table 2b). We found no evidence of a difference between the Al sensitivity and the sensitivity of human readers from either professional group; 63 / 70 cancer detected for radiologist readers and 62 / 70 cancer detected for non-radiologist readers, P=.26, P=.47 respectively. The mean specificity of radiologists and non-radiologists was 76% (95% Cl: 69%, 82%) and 77% (95% Cl: 71%, 83%) respectively. The average specificity of radiologists and non-radiologist readers was lower than the Al specificity of 89% at the developer’s recall threshold (odds ratio: 0.39, 0.42, P=.OO2, P=.OO4 respectively).

[0141] Table 2b

[0142] Table 2b shows performance metrics of the artificial intelligence model at the developer recommended threshold compared against those of the average (mean) human, by profession (radiologist, radiographer / breast clinician).

[0143] Discussion

[0144] The National Health Services Breast Screening Program routinely uses the quality assurance scheme Personal Performance in Mammographic Screening (PERFORMS) to assess reader performance, but whether this scheme could be used to assess the performance of artificial intelligence (Al) algorithms developed to read screening mammograms is unknown. Thus this study compared the performance of human readers and a commercially available Al algorithm acting as a stand-alone reader using two PERFORMS test sets. The area under the receiver operating characteristic curve (AUG) for Al was 0.93 which we did not find to differ from the AUG for human readers of 0.88 (P = .15). When the Al operated at the recall threshold suggested by the developer, the sensitivity of the Al was 84% (59 / 70 cancers detected) which was not found to be different from the mean human reader sensitivity of 90% (P = .34). However, the specificity of the Al was higher at 89% (151 / 170 true negatives) compared to the mean human reader specificity of 76% (P = .003). Using Al recall scores that matched the average human reader performance (90% sensitivity and 76% specificity), no difference was observed between human and Al performance in regard to sensitivity (91%; P = .73) or specificity (77%; P = .85). Many studies have used retrospective test sets enriched with malignant cases to assess Al cancer detection performance. A recent meta-analysis comparing the stand-alone detection performance of Al to human readers in 14 studies which included 185,252 mammograms and 39 readers demonstrated that the Al algorithms had a pooled sensitivity, specificity, and AUC of 75%, 91%, and 0.89, respectively, compared to the human reader pooled sensitivity, specificity, and AUC of 73%, 89%, and 0.85, respectively (7). The Al and human reader performance on the PERFORMS tests sets in our study are in line with these reported performance metrics. It is important when assessing Al as a stand-alone reader that recall thresholds are used which are relevant to the screening program where the product is deployed. Larsen et al compared the performance of Al to independent double reading in the Norwegian screening program by selecting a threshold that mirrored the average individual radiologists rate of positive interpretations and found that Al detected 80.1 % of screen detected breast cancers (9). We also matched the recall thresholds to UK readers based on their PERFORMS scores and observed no difference in performance between Al and human readers interpreting the PERFORMS sets.

[0145] Studies comparing the performance of Al in breast screening with the human reader typically use a larger set of mammograms (typically more than 10,000), but with only one or two human opinions per case (6,7). In our study, we use a smaller number of cases to assess Al performance, but the comparison is made to a large number of human readers. Salim et al used a test set of 8805 mammograms enriched with 739 cancers to assess the performance of three commercial Al algorithms and found that the best performing Al, which was the same product tested in our study, had an overall AUC of 0.956 which was similar to the AUC of 0.93 achieved on the 120 case PERFORMS set (23). In a study by McKinney et al Al performance was compared to 6 radiologists who read a cancer enriched test-set of 465 cases, the Al achieved an AUC of 0.871 which was greater than the mean human reader AUC of 0.750 (p=.0002) (12). In contrast we observed no difference in performance between Al and human readers in the 120 PERFORMS exams. The strength of using PERFORMS tests is that Al performance can be immediately compared to a large cohort of human readers from different professional groups who are all readers in the same screening programme and have read the same cases. In this study the 552 readers represent 68% of readers in the NHSBSP and so this should provide a robust performance comparison between human readers and Al.

[0146] Test-sets are used by some breast screening programs to identify reader performance problems. Significant positive correlations have been demonstrated between human reader cancer detection rates and recall rates in real-life and their sensitivity and specificity scores in test-set based assessment, indicating that these schemes are an accurate reflection of real- life performance. Performance monitoring is likely to play a part in the assessment of Al algorithms deployed into clinical practice due to product upgrades and concerns that Al performance can drift overtime (26). There are no other studies to date that have compared human reader performance in routine quality assurance test-sets to Al and so this study may provide a model for assessing Al performance in a real-world setting.

[0147] Our study has several limitations. First, the PERFORMS sets consist of a small number of mammograms, that are not representative of typical screening populations as they are enriched with challenging malignant cases and may not include a radically or ethnically diverse set of patients. Thus, the Al performance results reported here may not be generalizable. Second, the readers in this study were from the UK, where screening intervals are 3 years, readers include non-radiologists and 2D screening mammography remains the standard of care, rather than digital breast tomosynthesis. Consequently, our findings may not be applicable to other screening programs.

[0148] In conclusion, no difference was observed in the diagnostic performance of human readers and a commercially available Artificial intelligence (Al) algorithm for the interpretation of screening mammograms from two test sets used in the UK-based individual performance assessment scheme Personal Performance in Mammographic Screening (PERFORMS). The use of external quality assessment schemes like PERFORMS may provide a model for regularly assessing the performance of Al in a similar way to the monitoring of human readers, but further work is needed to ensure this assessment model could work for other Al algorithms, screening populations and readers.

[0149] The test data comprises medical images which may be used to compare an Al algorithm’s breast screening performance with that of human breast screeners.

[0150] The plurality of medical images may comprise different views of the same organ or region of the body. For example, breast screening cases may be submitted for potential inclusion in PERFORMS as full-field digital mammograms of the mediolateral oblique (MLO) view and the craniocaudal (CC) view of each breast, by breast screening units from the UK NHS Breast Screening Program (NHS BSP). The cases are of known outcome i.e. , there is a verified classification for each medical image (for example, biopsy proven malignant or benign, or normal cases with a normal three-year follow-up). Cases may be selected for inclusion in the PERFORMS scheme by a panel of expert radiologists. Case selection criteria may include that the mammogram must be of good technical quality, and that they should be challenging for the human screener in respect of interpretation. The locations of any malignant and benign features may be marked within each view of each breast, by the expert radiologists, alongside a description of the feature, its classification, and a rating for suspicion of malignancy. The verified classification may comprise a score or rating (for example, a score of 1 to 5 corresponding to normal, benign, indeterminate, suspicious, malignant) and a boundary or hitbox for the at least one feature.

[0151] A set of cases may be selected from the pool of PERFORMS cases for the purposes of comparing an Al algorithm’s breast screening performance with that of human breast screeners. For instance, 120 cases from one PERFORMS annual quality assurance testing cycle in the NHS BSP may be selected - each case may comprise two breasts, and each breast may be treated as a separate entity for the purposes of analyses (so 120 cases are analysed as 240 breasts).

[0152] Within a standard PERFORMS set of 240 breasts, typically about a sixth of the breasts have malignant features, with a range of features present across the case set (e.g., architectural distortions, asymmetries, calcifications, ill-defined masses, spiculate masses, well-defined masses). Test sets could also be constructed to contain different ratios of malignant and non- malignant breasts, or to focus on specific types of malignant feature (e.g., spiculate masses, architectural distortions), or to focus on breasts with certain characteristics (e.g., low density, high density, those most challenging to human readers), or specific breast views (MLO or CC), or specific mammography manufacturers / vendors. Advantageously, the methods described herein may allow weaknesses of a machine learning model to be identified, for example a weakness in identifying specific types of malignant features in medical images, organ characteristics, specific views of medical images, or particular imaging modalities or manufacturers I vendors.

[0153] Deriving the model of human performance at breast-level, based on the test-set Each breast is associated with the radiological opinions of all NHS BSP screeners who took part in PERFORMS at that time, typically over 600 breast screeners from the NHS BSP.

[0154] Each human screener has given an opinion on each feature detected within each view of each breast regarding their level of suspicion that the feature is malignant, using a five-point scale:

[0155] 1. normal, return to screen 2. benign, return to screen

[0156] 3. indeterminate, recall

[0157] 4. suspicious, recall

[0158] 5. highly suspicious, recall

[0159] Both breasts of the same case were available at the time of giving a breast-level opinion. The highest score given to a feature within a breast, across both views, is taken as the breast-level score. Where no features are detected within a given breast for either view, the overall breastlevel score is “1. normal, return to screen.”

[0160] Breast-level scores of 1 or 2 on this scale indicate that the human screener would not recall that breast for further investigation, whilst breast-level scores of 3 to 5 on this scale indicate that the human screener was sufficiently suspicious of malignancy to recall the breast for further investigation.

[0161] For each human screener, one of four outcomes is possible for each breast o true positive (TP), a breast with malignant features that is recalled o false negative (FN), a breast with malignant features that is returned to screen o true negative (TN), a breast with no malignant features that is returned to screen o false positive (FP), a breast with no malignant features that is recalled

[0162] For each human screener, sensitivity and specificity are calculated as percentages, based on the number of breasts in each category (TP, FN, TN, FP):

[0163] TP

[0164] Sensitivity = - x 100 TP + FN

[0165] TN

[0166] Specificity - x 100

[0167] K JTN + FP

[0168] Thus, for each human screener, sensitivity may be the percentage of breasts with malignant features recalled, whilst specificity is percentage of breasts without malignant features returned to screen.

[0169] The human screener model: o the sensitivity of the human screener model is derived by calculating mean sensitivity across human screeners (by calculating the sum of sensitivity scores across human screeners and dividing this by the number of human screeners), o whilst the specificity of the human screener model is derived by calculating mean specificity across human screeners (by calculating the sum of specificity scores across human screeners and dividing this by the number of human screeners).

[0170] For each human screener, the area under the curve of the receiver operator characteristic (AUC ROC) is calculated. The trapezoid method is used to calculate the AUC ROC, based on breast level-scores and using breast-level pathology as the ground truth. Breasts with malignant features are considered to be the “signal present” condition whilst breasts without malignant features are considered to be the “signal absent” condition. The AUC ROC gives a single score for each human screener that quantifies their ability to discriminate between malignant and non-malignant cases. The AUC ROC score is expressed as a value between 0.5 and 1.0, and is equivalent to the proportion correct in a two-alternative forced-choice (malignant or non-malignant) with the number of malignant and non-malignant cases balanced.

[0171] The human screener model can be based on all human screeners who read the case set, or on a subset of these human screeners (e.g., the best performing ten percent, best performing quartile, worst performing ten percent, worst performing quartile, radiologists only, advanced practitioner radiographers only etc.).

[0172] Comparing the performance of an Al breast screening algorithm with that of the human screener model, based on breast-level performance

[0173] For each breast in the case set, an Al breast screening algorithm should produce a continuous rating (e.g., between 0 and 100) for each feature detected within each view of each breast. This rating quantifies the algorithm’s level of suspicion that each feature detected within each breast view is malignant, with a higher rating indicating a higher level of suspicion. The highest rating assigned to a feature within either view of the breast is taken as the breast-level rating. Where no features are detected by the Al breast screening algorithm within either view of a given breast, the lowest rating possible is assigned as the breast-level rating (e.g., “0”). The receiver operator characteristic (ROC) is calculated for the Al breast screening algorithm, with the AUC ROC derived using the trapezoid method, based on breast levelscores. For human screeners, the decision threshold to recall is set at a suspicion rating of 3 or greater on the rating scale of 1 to 5. However, for the Al breast screening algorithm, a decision threshold could be set at any point along its rating scale, with each particular decision threshold producing a different pair of sensitivity and specificity values. On a rating scale for which increasing suspicion is conveyed by a higher rating, increasing the decision threshold will decrease sensitivity whilst increasing specificity. The Al breast screening algorithm’s ROC curve provides the pairs of sensitivity and specificity values at every possible decision threshold.

[0174] Comparing the human screener model and the Al breast screening algorithm may comprise at least one of:

[0175] 1) adjust the breast screening algorithm’s decision threshold so that the algorithm’s specificity matches that of the human screener model in order to compare the sensitivity of Al breast screening algorithm with the sensitivity of the human screener model (referred to below as Al algorithm decision threshold 1)

[0176] 2) adjust the breast screening algorithm’s decision threshold so that the algorithm’s sensitivity matches that of the human screener model in order to compare the specificity of Al breast screening algorithm with the specificity of the human screener model (referred to below as Al algorithm decision threshold 2)

[0177] Using the sensitivity and specificity scores of the human screener model (derived as described above), the number of breasts in each outcome category is calculated for the human screener model (with fractional numbers of breasts allowed): o number of TP breasts = number of breasts with malignant features multiplied by human screener model sensitivity; o number of FN breasts = number of breasts with malignant features minus number of TP breasts; o number of TN breasts = number of breasts without malignant features multiplied by human screener model specificity; o number of FP breasts = number of breasts without malignant features minus number of TN breasts.

[0178] Using Al algorithm decision threshold 1, which matched the Al algorithm’s specificity to the human screener model’s specificity in order to compare sensitivity values, the numbers of breasts in each of the TP and FN outcome categories are obtained on the basis of the continuous score assigned to each breast by the Al algorithm: o number of TP breasts = number of breasts with malignant features that received an Al rating score at or above decision threshold 1 o number of FN breasts = number of breasts with malignant features minus number of TP breasts

[0179] Using Al algorithm decision threshold 2, which matched the Al algorithm’s sensitivity to the human screener model’s sensitivity in order to compare specificity values, the numbers of breasts in each of the TN and FP outcome categories are obtained on the basis of the continuous score assigned to each breast by the Al algorithm: o number of TN breasts = number of breasts without malignant features that received an Al rating score below decision threshold 2 o number of FP breasts = number of breasts without malignant features minus number of TN breasts

[0180] The sensitivity of the human screener model and the sensitivity of the Al breast screening algorithm are then compared using an odds ratio (odds of the human screener model recalling a malignant breast divided by the odds of the Al breast screening algorithm recalling a malignant breast).

[0181] Similarly, the specificity of the human screener model and the specificity of the Al breast screening algorithm are compared using an odds ratio (odds of the human screener model returning to screen a non-malignant breast divided by the odds of the Al breast screening algorithm returning to screen a non-malignant breast).

[0182] TN (human) TN (Al) FP (human) FP(AI)

[0183] 95% confidence intervals are calculated for each odds ratio as described in Kirkwood BR, Sterne JAC (2003) Essential medical statists, 2nd ed. Oxford: Blackwell Science, with a value significantly greater than “1” indicating that the human screener model performed better than the Al breast screening algorithm on the given measure, and a value significantly lower than “1” indicating that the human screener model performed worse than the Al breast screening algorithm on the given measure.

[0184] Other decision thresholds may also be tested for the Al breast screening algorithm: o the algorithm manufacturer’s may have a recommended decision threshold; o the breast screening algorithm’s ROC curve can be used to work out the decision threshold that produces the maximum difference between hit rate (sensitivity) and false alarm rate (one minus specificity); also known as the maximum Kolmogorov-Smirnov statistic, this is the decision threshold at which the Al algorithm produces maximal discrimination between the signal present and signal absent conditions (although it is not necessarily the most clinically appropriate threshold); o any other decision threshold, for example, based on a clinically acceptable level of sensitivity or specificity.

[0185] For instance, using the algorithm manufacturer’s recommended decision threshold, the number of breasts in each outcome category can be obtained on the basis of the continuous score assigned to each breast by the Al algorithm: o number of TP breasts = number of breasts with malignant features that received an Al rating score at or above the manufacturer’s recommended decision threshold; o number of FN breasts = number of breasts with malignant features minus number of TP breasts; o number of TN breasts = number of breasts without malignant features that received an Al rating score below manufacturer’s the recommended decision threshold; o number of FP breasts = number of breasts without malignant features minus number of TN breasts;

[0186] These values can then be used to compare the human screener model’s sensitivity with the Al breast screening algorithm’s sensitivity at the manufacturer recommended threshold, and the human screener model’s specificity with the Al breast screening algorithm’s specificity at the manufacturer recommended threshold, using the odds ratio method described above. .

[0187] Comparing the performance of an Al breast screening algorithm with that of the human screener model, based on feature-level, location-based performance Feature-level comparisons of performance proceed in the same manner as breast-level comparisons of performance (described above), except that each malignant feature in each view of each breast (CC and MLO), as demarcated with a hitbox by the expert radiologist, is treated as a separate entity and entered into the marking scheme as if it were a separate breast. For breast views with no malignant features, each view of each breast is marked as if it were a separate non-malignant breast. Please note, in breast views from breasts with malignant features, the area outside of any malignant features’ hitboxes is ignored for the purposes of marking, as this area may contain malignancies that were not detected by human screeners at the time; if one view of a given breast contains a malignant hitbox, but the other view does not, marks in the view without a malignant hitbox are still ignored.

[0188] In order for a human screener or the Al breast screening algorithm to score a TP, the location of a malignant feature must be marked within the hit box drawn by the expert radiologist in a given breast view (only one mark per hitbox is counted - that with the highest rating). For the human screener the feature-level suspicion rating must also be 3 or higher, otherwise an FN score will be assigned. For the Al breast screening algorithm, the assignment of a TP or an FN score for marks within the hitbox of a malignant feature will depend on the decision threshold adopted. Any malignant feature not receiving a mark within its hitbox is assigned a rating of “1. normal” for a human screener or the lowest score possible for an Al breast screening algorithm, essentially scoring it as an FN. Any marks in the area outside the hitboxes in breast views from a breast with malignant features are ignored.

[0189] Breast views from breasts with no malignant features are marked as at breast level. The highest rating assigned to any human screener or Al breast screening algorithm applied feature mark within the breast view is taken as the breast view rating for that human screener or the Al algorithm. If no features are marked on such views, the view is assigned a rating of “1. normal” for a human screener or the lowest score possible for an Al breast screening algorithm, essentially scoring it as an TN.

[0190] For each human screener, one of four outcomes is possible for each breast location: o true positive (TP), a feature marked within a malignant hitbox, within a given breast view, rated greater than 2 (again noting that only one mark per hitbox is counted - that with the highest rating) o false negative (FN), a malignant hitbox, within a given breast view, that is either not marked, or that is marked but not rated greater than 2 o true negative (TN), a breast view from a breast with no malignant features that is either not marked with a feature or has not received a mark with a rating greater than 2 o false positive (FP), a breast view from a breast with no malignant features that is marked with at least one feature that has a rating greater than 2

[0191] For each human screener, sensitivity and specificity are calculated as percentages, based on the number of breast locations in each category (TP, FN, TN, FP):

[0192] TP Sensitivity = - x 100 TP + FN

[0193] TN

[0194] Specificity - x 100

[0195] K JTN + FP

[0196] Thus, for each human screener, o sensitivity is the percentage of breast locations with malignant features that received ratings greater than 2, o whilst specificity is percentage of breast locations without malignant features (i.e. , breast views from breasts without malignant features) that either received no feature marks or received no ratings greater than 2.

[0197] The human screener model at feature-level, for location-based performance: o the sensitivity of the human screener model is derived by calculating mean sensitivity (location-based) across human screeners (by calculating the sum of sensitivity scores across human screeners and dividing this by the number of human screeners), o whilst the specificity of the human screener model is derived by calculating mean specificity (location-based) across human screeners (by calculating the sum of specificity scores across human screeners and dividing this by the number of human screeners).

[0198] The AUC FROC for each human screener and the Al breast screening algorithm may also be calculated at the feature-level to quantify location-based performance in discriminating between the presence and absence of malignancy. The “signal present” condition comprises each malignant feature, as demarcated with a hitbox by the expert radiologist, in each view of a breast where malignancies are visible. Meanwhile the “signal absent” condition comprises each view of each breast with no malignant features.

[0199] The human screener model and the Al breast screening algorithm may then compared in the same way as for breast-level analyses, but using the location-based human screener model’s sensitivity and specificity scores to work out the appropriate decision threshold’s on the Al breast screening algorithm’s location-based FROC. The decision threshold that matches the Al breast screening algorithm’s specificity to human specificity (both location-based) is then used to work out which malignant breast locations that the Al algorithm would recall (TPs) or not (FNs). And the decision threshold that matches the Al breast screening algorithm’s sensitivity to human sensitivity (both location-based) is used to work out which non-malignant breast locations the Al algorithm would recall (FPs) or not (TNs). The human screener model’s location-based sensitivity is then compared to the Al breast screening algorithm’s location-based sensitivity using an odds ratio, as described for the breast-level analyses. Similarly, the human screener model’s location-based specificity is compared to the Al breast screening algorithm’s location-based specificity using an odds ratio, as described for the breast-level analyses.

[0200] Comparing the overall performance of an Al breast screening algorithm in discriminating between malignant and non-malignant cases with the range of human screener performances

[0201] The AUG ROC for the Al breast screening algorithm can be compared with the range of human screener AUC ROCs on the same breasts of the given case set, using several methods. These comparisons can be made on the basis of breast-level AUC ROC scores or the feature-level, location-based AUC FROC scores.

[0202] The percentile rank, calculated as the percentage of human screener AUC ROC scores that the Al breast screening algorithm’s AUC ROC is higher than, may provide a direct description of the Al breast screening algorithm’s performance relative to the range of human performance on the given breasts, or at feature level.

[0203] It may be determined if the Al breast screening algorithm’s AUC ROC is an outlier relative to the range of performance exhibited by human screeners using Tukey’s fences: o the 25thpercentile and 75thpercentiles of the human screener AUC ROC scores are calculated, and the interquartile range (IQR) of these scores is calculated (the 75thpercentile minus the 25thpercentile); o a lower-inner fence for extremely poor performing Al breast screening algorithm outliers is defined as 1.5 times the IQR below the 25thpercentile of human screener scores, whilst a lower-outer fence for the most extremely poor performing Al breast screening algorithm outliers is defined as 3.0 times the IQR below the 25thpercentile of human screener scores; and / or o an upper-inner fence for extremely high performing Al breast screening algorithm outliers is defined as 1.5 times the IQR above the 75thpercentile of human screener scores, whilst an upper-outer fence for the most extremely high performing Al breast screening algorithm outliers is defined as 3.0 times the IQR above the 75thpercentile of human screener scores.

[0204] Thus, in some examples, with respect to an Al breast screening algorithm’s AUG ROC score: o a score lower than the lower-outer fence indicates most extremely poor performance relative to human screeners; o a score lower than the lower-inner fence (but higher than or equal to the lower-outer fence) indicates extremely poor performance relative to human screeners; o a score lower than the 25thpercentile (but higher than or equal to the lower- inner fence) indicates performance amongst the lowest quartile of human screeners, but not extremely poor performance; o a score higher than or equal to the 25thpercentile, but lower than or equal to the 75thpercentile indicates performance amongst the middle 50% of human screeners; o a score higher than the 75thpercentile (but lower than or equal to the upper- inner fence) indicates performance amongst the highest quartile of human screeners, but not extremely high performance; o a score higher than the upper-inner fence (but lower than or equal to the upper-outer fence) indicates extremely high performance relative to human screeners; and / or o a score higher than the upper-outer fence indicates very extremely high performance relative to human screeners.

[0205] If the human screener AUG ROC scores are normally distributed, the Al breast screening algorithm’s AUC ROC can also be compared to the range of human screener AUC ROC scores by converting it to a Z-score. A Z-score lower than -1.96 or higher than +1.96 would indicate that the Al breast screening algorithm’s AUC ROC was significantly different to the mean human screener AUC ROC at the 95% confidence level. Variants

[0206] It will be understood that the above description of specific embodiments is by way of example only and is not intended to limit the scope of the present disclosure. Many modifications of the described embodiments, some of which are now described, are envisaged and intended to be within the scope of the present disclosure.

[0207] While the above method has been described with reference to breast cancer diagnostic screening and mammographic medical images, in other implementations the machine learning model may be suitable for other types of diagnosis, including diagnosis of different types of cancer and other diseases.

[0208] It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure has been described with reference to specific example implementations, it will be recognized that the disclosure is not limited to the implementations described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

Claims1. A computer-implemented method comprising: obtaining testing data comprising a plurality of medical images and at least one likelihood value associated with each image, the likelihood values having been assigned by a machine learning model, and wherein each likelihood value is indicative of a likelihood that a feature in the associated image is a malignant feature; determining a first classification data set for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold; comparing the first classification data set to a verified classification data set associated with the plurality of medical images; determining, based on comparing the first classification data set to the verified classification data set, a performance score associated with the machine learning model.

2. The method of claim 1 , further comprising outputting the performance score.

3. The method of any preceding claim, wherein the first classification data set comprises a label assigned to at least one feature of a medical image based on whether or not the at least one likelihood value is greater than a decision threshold, wherein the label is indicative of a clinical category.

4. The method of any preceding claim, wherein the verified classification data set comprises a verified label indicative that the at least one feature belongs to a clinical category.

5. The method of claim 3 or 4, wherein the clinical category comprises an indication that the at least one feature is a positive result or a negative result.

6. The method of any of claims 3 to 5, wherein the clinical category comprises an indication that the at least one feature is one of: benign, indeterminate, suspicious, or highly suspicious.

7. The method of any preceding claim, wherein the verified classification data set further comprises, for at least one medical image of the plurality of medical images, a boundary surrounding at least one feature of the medical image.

8. The method of claim 7, wherein the first classification data set further comprises, for at least one medical image of the plurality of medical images, location data for at least one feature of the medical image, wherein the location is assigned by the machine learning model.

9. The method of claim 6, wherein comparing the first classification data set to the verified classification data set further comprises comparing the location assigned to the least one feature to a boundary surrounding at least one feature.

10. The method of any of claims 4 to 9, wherein comparing the first classification data set to the verified classification data set further comprises determining that at least one feature of an image of the plurality of medical images is one of: a true positive, wherein a true positive is based on a determination that the label assigned to the at least one feature matches the verified label, and that the label assigned to the at least one feature is indicative of a positive result; a false negative, wherein a false negative is based on a determination that the label assigned to the at least one feature does not match the verified label, and that the label assigned to the at least one feature is indicative of a negative result; a true negative, wherein a true negative is based on a determination that the label assigned to the at least one feature matches the verified label, and that the label assigned to the at least one feature is indicative of a negative result; or a false positive, wherein a false positive is based on a determination that the label assigned to the at least one feature does not match the verified label, and that the label assigned to the at least one feature is indicative of a positive result.

11. The method of claim 10, wherein determining that at least one feature of an image of the plurality of medical images is a true positive is further based on a determination that the location assigned to the least one feature is within the boundary surrounding at least one feature.

12. The method of claim 10 or 11 , wherein determining a performance score comprises at least one of: based on determining that the first classification data set comprises a true positive and a false negative, calculating a sensitivity of the machine learning model at the decision threshold; and based on the determining that the first classification data set comprises a true negative and a false positive, calculating a specificity of the machine learning model at the decision threshold;based on determining that the first classification data set comprises a true positive and a false positive, calculating a positive predictive value of the machine learning model at the decision threshold; based on determining that the first classification data set comprises a true negative and a false negative, calculating a negative predictive value of the machine learning model at the decision threshold.

13. The method of claim 12, wherein determining a first classification data set for the plurality of images by comparing, for each image, the at least one likelihood value to a decision threshold, comparing the first classification data set to a verified classification data set associated with the plurality of medical images, and determining, based on comparing the first classification data set to the verified classification data set, a performance score associated with the machine learning model comprises a routine, and wherein determining a performance score further comprises: performing the routine at a plurality of decision threshold values, wherein the performance score for each decision threshold value comprises a sensitivity and a specificity associated with the machine learning model at the decision threshold value; generating a receiver operating characteristic curve for the machine learning model based on the performance scores associated with the plurality of decision threshold values; calculating an area under the receiver operating characteristic curve.

14. The method of any preceding claim, further comprising comparing the performance score associated with the machine learning model to a plurality of performance scores associated with humans for the plurality of medical images.

15. The method of claim 14, wherein the performance score associated with the machine learning model comprises a sensitivity and a specificity calculated for a given decision threshold value, and wherein the method further comprises: calculating a mean performance score for the plurality of performance scores associated with humans, wherein the mean performance score comprises a mean human sensitivity and a mean human specificity; comparing at least one of: the sensitivity of the machine learning model for the given decision threshold value with a mean human sensitivity; and the specificity of the machine learning model for the given decision threshold value with a mean human sensitivity.

16. The method of claim 15, wherein the method further comprises: performing the routine at a plurality of decision threshold values, wherein the performance score for each decision threshold value comprises a sensitivity and a specificity associated with the machine learning model at the decision threshold value; comparing the performance score for each decision threshold value to at least one of the mean human sensitivity or the mean human specificity; determining a decision threshold value associated with at least one of: a sensitivity of the machine learning model that is equal to the mean human sensitivity, or a specificity of the machine learning model that is equal to the mean human specificity.

17. The method of any of claims 14 to 16, further comprising outputting a percentile rank based on a percentage of the plurality of performance scores associated with humans that the performance score associated with the machine learning model is higher than.

18. The method of any preceding claim, wherein the performance score comprises at least one of: sensitivity, specificity, positive predictive value, negative predictive value, and / or area under the receiver operating characteristic curve.

19. The method of any preceding claim, wherein the test set comprises medical images comprising at least one of: architectural distortion features, asymmetrical densities, microcalcification features, calcification features, ill-defined mass features, spiculate mass features, and / or well-defined mass features.

20. The method of any preceding claim, further comprising: calculating a second performance score for a subset of the testing data, wherein the performance score for the testing data is improved relative to the second performance score for the subset of the testing data; outputting an indication of an area of a weakness of the machine learning model based on common features of the subset of the testing data.

21. The method of claim 20, further comprising outputting a training set comprising a second plurality of medical images for further training the machine learning model, wherein the second plurality of medical images are selected based on the area of weakness of the machine learning model.

22. The method of any preceding claim, wherein the method further comprises:performing the routine to calculate a second performance score using a second decision threshold; determining that the second performance score is improved relative to the first performance score; outputting that the second decision threshold is an improvement relative to the first decision threshold.

23. A system of evaluating a machine learning model for diagnosis, wherein the system comprises: a processor; a memory; a receiving module stored in the memory and executed by the processor to receive, from a machine learning model, testing data comprising a plurality of medical images and at least one likelihood value associated with each image, wherein each likelihood value is indicative of a likelihood that the image contains a malignant feature, the likelihood values having been assigned by the machine learning model; a scoring module stored in the memory and executed by the processor to carry out a method according to any of claims 1 to 22.

24. A computer readable medium containing instructions that, when executed by a processor, perform the method of any of claims 1 to 22.

Images