Method of classification

Abstract

Obtaining a first data set comprising first input data corresponding to a first class; Obtaining a second data set comprising second input data corresponding to the first class; training at least one classifier using the first data set; 1. A computer-implemented method comprising: inputting second input data from a second dataset into at least one classifier; and providing a classification model comprising a first classification and a second classification, wherein the first classification predicts a greater proportion of the second input data corresponding to the first class that will be in the first class than the second classification.

Description

[Technical field]

[0001] The present disclosure relates to methods of classification, systems for classification, and methods of training. [Background technology]

[0002] Classifier models have applications in a wide range of fields. For example, classifier models may be used in various medical applications. Classifier models may be used to provide medical diagnoses, for example, by taking physiological measurements as input. In another example, classifier models may be used to analyze medical image data, for example, to predict whether an input tissue image contains cancerous cells or to predict whether a particular biomarker is associated with the tissue image.

[0003] Such a classifier model may be trained using a training dataset. For example, the training dataset may comprise a large number of example inputs along with labeled classification predictions. A classification method that uses a classifier model modeled on the training dataset, however, will have a certain level of inaccuracy when deployed on new input data. The accuracy of a classification method may be measured by the number of correct predictions as a proportion of the number of total predictions made by the method. This may be assessed, for example, on a test dataset, where a correct prediction is known for each input in the dataset.

[0004] In some use cases, e.g., medical use cases, high accuracy is required. The accuracy of the classification method can be improved by improving the classifier model performance. Such improvement can be achieved, for example, by further training of the classifier model on a new training data set. However, in some cases, for example, when further training data cannot be easily obtained, further improvement to the classifier model performance is not possible. Summary of the Invention

[0005] Systems and methods according to non-limiting embodiments will now be described with reference to the accompanying figures. [Brief description of the drawings]

[0006] [Figure 1] 1 is a schematic diagram of a method of classification according to one embodiment. [Figure 2(a)] Schematic of the classifier model output for the dataset. [Figure 2(b)] Schematic diagram of a method according to a first example. [Diagram 3] Schematic diagram of a method according to a first example. [Figure 4] FIG. 1 illustrates an example of a convolutional neural network. [Diagram 5] Schematic diagram showing the training pipeline and deployment of the trained classification system according to the first example. [Figure 6] FIG. 1 is a schematic diagram illustrating a first predictive threshold. [Figure 7] FIG. 13 is a schematic diagram illustrating a second prediction threshold. [Figure 8(a)] Schematic diagram of a method for training a system for classification according to a first example. [Figure 8(b)] Schematic diagram of a method for training a system for classification according to a first example. [Figure 9] Schematic diagram of the transition from perfect sensitivity to perfect specificity. [Figure 10(a)] Schematic diagram of a method for training a system for classification by modification of the first example. [Figure 10(b)] FIG. 1 illustrates an example optimization process. [Figure 10(c)] FIG. 13 shows accuracy values ​​calculated for threshold combinations. [Figure 11(a)] FIG. 11 is a diagram showing a classification method according to the second example. [Figure 11(b)] Schematic diagram of the method including input data. [Figure 12(a)] Schematic diagram of a method for training a classification system according to a second example. [Figure 12(b)] Schematic diagram showing the training pipeline and deployment of the trained classification system according to the second example. [Figure 12(c)] Schematic diagram of how to train a classifier by modifying the second example. [Figure 13] Schematic diagram of a method of classification according to a third example. [Figure 14] Schematic diagram of an exemplary method of classification. [Figure 15] Schematic diagram of an exemplary classification system. [Figure 16] FIG. 11 is a diagram showing a classification method according to the fourth example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] According to a first aspect, Obtaining a first data set comprising first input data corresponding to a first class; Obtaining a second data set comprising second input data corresponding to the first class; training at least one classifier using the first data set; inputting second input data from a second dataset into at least one classifier and providing a classification model comprising a first classification and a second classification, wherein the first classification predicts a greater proportion of the second input data corresponding to the first class that will be in the first class than the second classification; A computer-implemented method is provided, comprising:

[0008] In one example, the second classification predicts a greater proportion of the second input data corresponding to the second class to be in the second class than the first classification.

[0009] In one example, the first class is a negative class and the second class is a positive class.

[0010] In one example, the first classification comprises applying a first classifier and using a first prediction threshold, and the second classification comprises applying the first classifier and using a second prediction threshold, the first prediction threshold being higher than the second prediction threshold.

[0011] In one example, the first classifier is a binary classifier. Determining the first predictive threshold may comprise determining a highest output value of the binary classifier for an input from the second data set corresponding to the first class and setting the first predictive threshold less than or equal to the highest output value, and determining the second predictive threshold comprises determining a lowest output value for an input from the second data set corresponding to the second class and setting the second predictive threshold greater than or equal to the lowest output value.

[0012] In one example, the at least one classifier comprises at least two classifiers, and providing the classification model comprises determining a first classifier of the at least two classifiers, where the first classifier predicts a largest proportion of second input data corresponding to a first class that will be in the first class of the at least two classifiers, and determining a second classifier of the at least two classifiers, where the second classifier predicts a largest proportion of second input data corresponding to a second class that will be in the second class of the at least two classifiers.

[0013] In one example, providing the classification model comprises determining a first predictive threshold and a second predictive threshold, where determining the first predictive threshold and the second predictive threshold comprises obtaining a plurality of possible combinations of values ​​for the first predictive threshold and values ​​for the second predictive threshold, determining an accuracy of the classification model using each of the possible combinations, and selecting a combination having an accuracy higher than a predetermined accuracy value.

[0014] In one example, the first data set comprises classification labels for the first input data and the second data set comprises classification labels for the second input data.

[0015] In one example, the first classifier and the second classifier correspond to different model architectures and / or correspond to parameters updated based on different input data and / or using different hyperparameters.

[0016] In one example, the first data set comprises first input data corresponding to a first population, and the second data set comprises second input data corresponding to a second population.

[0017] According to another aspect, Obtaining a first data set comprising first input data corresponding to a first class; Obtaining a second data set comprising second input data corresponding to the first class; training a classifier using a first data set; inputting second input data from a second dataset into a classifier and providing a classification model with a first classification, wherein providing the classification model with the first classification comprises determining a prediction threshold corresponding to the first class. A computer-implemented method is provided, comprising:

[0018] In one example, determining the predictive threshold comprises determining a highest output value of the classifier for inputs from the second data set that correspond to the first class, and setting the predictive threshold less than or equal to the highest output value.

[0019] In one example, determining the predictive threshold comprises determining a lowest output value for an input from the second data set that corresponds to the first class, and setting the predictive threshold greater than or equal to the lowest output value.

[0020] In one example, determining the predictive threshold comprises obtaining a plurality of possible values ​​for the predictive threshold, determining a score for the classification model using each of the possible values, and selecting the value having a score higher than a predetermined score. The score may be a measure of a proportion of data not corresponding to the first class that is classified as being in the first class. The plurality of possible values ​​may comprise a set of values ​​between a highest output value of the classifier for an input from the second data set corresponding to the first class and a lowest output value of the classifier for an input from the second data set corresponding to the first class.

[0021] According to another aspect, Obtaining input data; inputting input data into a classification model; and the classification model applying a first classification and a second classification, wherein the first classification is configured to correctly predict a greater proportion of the input data corresponding to the first class than the second classification. determining a classification prediction for the input data based on the output of the first classification and the second classification; A computer-implemented method is provided, comprising:

[0022] According to another aspect, Obtaining input data; inputting input data into a classification model; and the classification model applying a first classification and a second classification, where the first classification uses a first predictive threshold and the second classification uses a second predictive threshold, where the first predictive threshold is higher than the second predictive threshold; determining a classification prediction for the input data based on the output of the first classification and the second classification; A computer-implemented method is provided, comprising:

[0023] In one example, the first classification comprises applying a first classifier and using a first predictive threshold, and the second classification comprises applying the first classifier and using a second predictive threshold, where the first predictive threshold is higher than the second predictive threshold.

[0024] In one example, the first classification comprises applying a first classifier and the second classification comprises applying a second classifier.

[0025] In one example, the input data comprises an image of tissue, where determining the classification comprises determining information related to a medical diagnosis.

[0026] In one example, determining a classification prediction for the input data based on an output of the first classification and the second classification includes: if the output of the first classification and the output of the second classification are the same, outputting a classification prediction, which is the output of the first classification or the second classification; Determining that there is no classification prediction Equipped with.

[0027] According to another aspect, Obtaining input data; inputting input data into a classification model; and the classification model applying a first classification, wherein the first classification comprises applying a first classifier and using a prediction threshold corresponding to the first class. determining a classification prediction for the first class based on the output of the first classification; A computer-implemented method of classification is provided, comprising:

[0028] In one example, determining the classification prediction includes: if the output of the first classification is the first class, outputting a classification prediction that is the first class; determining that there is no classification prediction if the output of the first classification is not the first class; Equipped with.

[0029] According to another aspect, there is provided a classification system comprising one or more processors configured to perform the above method.

[0030] According to another aspect, there is provided a classification system comprising a classification model trained according to the above method.

[0031] According to another aspect, a carrier medium is provided comprising computer readable code configured to cause a computer to perform the above methods. The methods are computer implemented methods. Since some methods according to the embodiments may be implemented by software, some embodiments encompass computer code provided to a general purpose computer on any suitable carrier medium. The carrier medium may comprise any storage medium, such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transitory medium, such as any signal, for example, an electrical signal, an optical signal or a microwave signal. The carrier medium may comprise a non-transitory computer readable storage medium.

[0032] Classifier models have applications in a wide range of fields. For example, classifier models may be used in various medical applications. Classifier models may be used to provide medical diagnoses, for example, by taking physiological measurements as input. In another example, classifier models may be used to analyze medical image data, for example, to predict whether an input tissue image contains cancerous cells or to predict whether a particular biomarker is associated with the tissue image.

[0033] Such a classifier model may be trained using a training dataset. For example, the training dataset may comprise a number of example inputs along with labeled classification outputs. A classification method that uses a classifier model modeled on a training dataset, however, will have a certain level of inaccuracy when deployed on new input data.

[0034] For example, a binary classifier model outputs a score between 0 and 1 for each data point, which is converted to a classification (positive or negative) based on a prediction threshold. If the score is above the threshold, the output prediction is positive. If the score is below the threshold, the output prediction is negative. The threshold may be, for example, 0.5. A binary classifier model may be used, for example, to predict whether a whole slide image contains cancerous cells. An input whole slide image is predicted to contain cancerous cells only if the score given by the classifier model is greater than or equal to 0.5.

[0035] The classifier model output is compared to a prediction threshold to generate a classification method output (positive or negative). The accuracy of a classification method can be measured by the number of correct classifications as a proportion of the number of total classifications. This can be assessed, for example, on a test dataset, where the correct classification (positive or negative) is known for each input in the dataset.

[0036] In some use cases, e.g., medical use cases, high accuracy is required. The accuracy of classification can be improved by improving the classifier model performance, where the classifier model performance can be measured by AUC (described in more detail below). Such improvement can be achieved, e.g., by further training of the classifier model on a new training dataset. However, in some cases, further improvement to the classifier model performance is not possible, e.g., further training data cannot be easily obtained.

[0037] FIG. 1 shows a schematic diagram of a method for classification according to one embodiment.

[0038] At S101, input data is obtained. The input data may be, for example, physiological measurements or input medical images.

[0039] In S102, a first classification and a second classification are applied. The first classification and the second classification each output a classification into a positive class or a negative class. The positive class is also called the second class and the negative class is also called the first class. The positive class is the same for the first classification and the second classification. For example, the positive class can be a medical diagnosis, such as a diagnosis of a disease. Alternatively, the positive class can be the detection of cancerous cells or an indication of a particular biomarker, for example for an input image of tissue. The first classification is configured to correctly predict more negative cases than the second classification. The second classification is configured to correctly predict more positive cases than the first classification. The second classification is configured to have a higher true positive rate than the first classification, where the true positive rate is the proportion of positive cases that the method correctly predicts as positive.

[0040] Then, in S103, a classification prediction is determined based on the output of the first classification and the second classification. In particular, if the output of the first classification and the output of the second classification are the same, the classification prediction is output as the output of the first classification (or the second classification). If the output of the first classification and the output of the second classification are different, the classification prediction is not output. Then, some further steps, for example manual classification, can be performed. The accuracy of the classification prediction is improved because the cases with more uncertain prediction are excluded.

[0041] Next, an exemplary method of classification is described. First Example There are various indicators of the performance of a classification method. The sensitivity of a classification method is the true positive rate (TPR), which is the proportion of positive cases that the method correctly predicts as positive. The specificity of a classification method is the true negative rate (TNR), which is the proportion of negative cases that the model correctly predicts as negative. The specificity of a method is also equal to 1-FPR, where FPR is the false positive rate. FPR is the proportion of negative cases that the method incorrectly predicts as positive. The false negative rate (FNR) is the proportion of positive cases that the model incorrectly predicts as negative, which is equal to 1-TPR.

[0042] As explained above, a binary classifier model outputs a score between 0 and 1 for each data point, which is converted to a classification (positive or negative) based on a prediction threshold. If the score is above the threshold, the output classification is positive. If the predicted score is below the threshold, the output classification is negative. In many classifier models, e.g., deep neural networks, support vector machines, random forests, etc., the sensitivity and specificity of the classification method can be adjusted after training the classifier model by simply shifting the prediction threshold. Figure 2(a) is a schematic diagram of classifier model outputs for a dataset. The classifier model outputs for the positive class (probability values ​​from 0 to 1) are along the x-axis, in a discrete interval of 0.05. The frequency of each classifier model output for the dataset corresponds to the y-axis. Outputs corresponding to cases where the true value is positive (1) are shown in darker shading, while outputs for cases where the true value is negative (0) are shown in lighter shading.

[0043] A higher prediction threshold results in a classification method with higher specificity, which can correctly predict more negative cases but not more positive cases. The line "specificity threshold" in Figure 2(a) shows the threshold at which all true negative cases in the dataset are correctly predicted.

[0044] The same classifier model combined with a lower prediction threshold will result in a classification method with higher sensitivity that can correctly predict more positive cases but not more negative cases. The line "sensitivity threshold" in Figure 2(a) indicates the threshold at which all positive cases in the dataset are correctly predicted. The "specificity" threshold is higher than the "sensitivity" threshold.

[0045] A classification method using a classifier model and a higher threshold is configured to correctly predict more negative cases than a classification method using a classifier model and a lower threshold, and further, a classification method based on a sensitivity threshold of a classifier model is configured to correctly predict more positive cases than a classification method based on a specificity threshold of a classifier model.

[0046] Figure 2(b) shows a method according to a first example. Classification is performed on input image data corresponding to an image of tissue. A schematic diagram of the process including input image data 41 is shown in Figure 3. However, it will be appreciated that in other examples other types of input data may be used including, for example, one or more of other types of images (such as non-tissue medical images), patient data (such as age, sex, health status), liquid biopsy data (e.g. DNA sequence data), or blood test data (e.g. blood cell counts).

[0047] In this example, the input image is classified as to whether it corresponds to a particular biomarker, the ER biomarker. If the input image corresponds to the ER biomarker, it is classified into a positive class. However, it will be appreciated that in other examples, other types of classifications may be performed, for example to classify whether the image corresponds to a different biomarker, or, for example, to a cancerous cell.

[0048] The method according to the first example uses a first classification and a second classification. The first classification and the second classification use the same classifier. However, the prediction threshold is higher for the first classification than for the second classification. Thus, the first classification is configured to correctly predict more negative cases than the second classification. The second classification is configured to correctly predict more positive cases than the first classification, in other words, the second classification is configured to have a higher true positive rate than the first classification.

[0049] At S201, input data is obtained. In this example, the input data is input image data 41 comprising a plurality of pixels. The input image data 41 is, in this example, an image tile from a whole slide image of tissue, as shown in FIG. 3. In the following description, the pixel data is red-green-blue (with dimensions of height×width×3), but the pixel data may alternatively be, for example, grayscale (with dimensions of height×width×1). The input image data comprises a first number of pixels, where the first number is equal to height×width. The image data may be collected using a digital camera attached to a microscope that captures an image of the tissue (also called a histological section).

[0050] In the particular example described herein, the input image data 41 comprises tiles of images of histological sections stained with hematoxylin and eosin stains. An example of a whole slide image (WSI) of a histological section stained with hematoxylin and eosin stains is shown in FIG. 3 (WSI). A grid is overlaid on the WSI, where each grid square corresponds to a tile. A whole slide image scanner scans the entire tissue slice, resulting in an image of the histological section stained with hematoxylin and eosin stains with, for example, a height of about 60000 pixels by a width of 60000 pixels. The whole slide image is then split into contiguous parts, or tiles. The image parts (tiles) have a fixed height and width. The parts may be contiguous or overlapping within the image. For example, the image tile size may be 512×512 pixels. The whole slide image is split into tiles, and each tile is taken separately as an input in S201. Other tile sizes may of course be used. For example, tile sizes corresponding to powers of two may be used, for example 128x128, 256x256, 512x512, or 1024x1024 pixels. Each WSI image may be of different size, and therefore, depending on the size of the input WSI, a different number of parts may be extracted from the input WSI. The original image data may be taken as input, i.e., the original pixel values. Alternatively, some pre-processing may be performed on the original pixel values, for example for color normalisation. A cell segmentation step may be performed first, discarding tiles that contain only non-cancerous tissue. A pre-trained model may be used for cell segmentation.

[0051] In S202, input image data 41 (image tiles) is taken as input to a classification model. The classification model comprises a first classifier. In this example, the first classifier is a first convolutional neural network (CNN) 40. An exemplary first convolutional neural network 40 is shown in FIG.

[0052] The first convolutional neural network 40 comprises a first part 46 with at least one convolutional layer, and a second part 47, a classification part, that takes one-dimensional vectors as input. The second part 47 comprises at least one fully connected layer. The first CNN 40 comprises a first part 46 with convolutional filters applied to the image at various layers of depth and field of view, followed by a second part 47 with fully connected dense layers and / or pooling layers for data reduction. The filter weights are trainable parameters that are learned during the training phase. Lower level filters detect coarse structures such as edges and blobs, deeper levels capture more complex properties such as shape and texture, and finally, the top layer learns to generalize to objects of interest regarding biomarker identification information.

[0053] The first CNN 40 is a binary classifier. The first CNN 40 is used to determine whether an input image tile is associated with a particular molecular biomarker. The tiles are submitted to the first CNN 40. The pixel-by-pixel data is input directly to the first CNN 40. For each tile, the CNN outputs the probability that the tile is assigned to a positive class, i.e., the tile is associated with the biomarker.

[0054] The first CNN 40 shown in Figure 4 is based on a residual network architecture. The residual neural network comprises one or more skip connections. Although the figure shows a small number of layers for simplicity, the first CNN 40 may comprise, for example, more than 100 layers.

[0055] The first layer in the CNN is a convolutional layer, labeled in the figure as "Convolutional Layer 1." Each filter in the first layer has a depth that matches the depth of the input data. For example, if the input data is RGB, the filter depth in the first layer is 3. For simplicity, the CNN shown in FIG. 4 has an input data depth of 1 (i.e., grayscale input data).

[0056] The output volume of the first tier is determined by several factors.

[0057] The depth of the output volume of the first layer corresponds to the number of filters. For example, there may be 32 filters in the first layer, so the output of the first layer has a depth of 32. Thus, the filters in subsequent layers will have a depth of 32.

[0058] The height and width of the output volume are determined by the input height and width, the filter's receptive field size (both height and width), and the filter stride. When the stride is 1, the filter slides one pixel at a time. When the stride is 2, the filter slides two pixels at a time, producing a smaller output volume. Zero padding used at the border will also affect the output size.

[0059] Each filter is moved along the width and height of the input. At each location, an element-wise multiplication of the input value with a weight value is performed and the results are added to give a single output value for each location. The output values ​​for a filter form a 2D array. The output arrays from all the filters in a layer are stacked along the depth dimension and the resulting volume is input to the next layer.

[0060] Each convolutional layer is followed by an activation layer. The activation layer applies an element-wise activation function and leaves the size unchanged. The activation layer is not shown in the figure for simplicity. For example, the model may comprise one or more ReLU (rectified linear unit) layers that apply an element-wise activation function. A batch normalization layer is also implemented after each convolutional layer, where the activation layer is implemented after the batch normalization layer. The model comprises a unit comprising a convolutional layer, a batch normalization layer, and an activation layer, or a unit comprising, for example, a first convolutional layer, a first batch normalization layer, a second convolutional layer, a second batch normalization layer, and an activation layer.

[0061] The first CNN 40 comprises multiple layers whose output has smaller dimensions than the input, e.g., height and / or width are smaller than the input to the layer. In this way, the height and width decrease through some layers while the depth increases. The first CNN 40 has an "encoder / decoder" structure, whereby the layers first decrease the height and width while increasing the depth (e.g., via filter hyperparameters such as stride size), and then increase the height and width while decreasing the depth (e.g., via pooling layers and / or bilinear upsampling layers). This is illustrated in FIG. 4, which shows the output sizes of the layers.

[0062] The first CNN 40 further comprises one or more pooling layers included to vary the spatial size. The pooling layers may be used to increase the width and / or height and decrease the depth of the output. The pooling layers may be "average pooling" layers. The average pooling layer comprises a filter with a spatial extent and stride that is moved across the input and takes the average value at each position. However, functions other than average may be used, e.g., max pooling. An upsampling layer, e.g., one or more bilinear upsampling layers, may additionally or alternatively be included to increase the height and / or width.

[0063] The first CNN 40 further comprises at least one skip connection. In practice, the model may comprise multiple skip connections, but for simplicity, a small number of layers and a single skip connection are shown in FIG. 4. The second layer, “Convolutional Layer 2”, produces an output called output m. The fourth layer, “Convolutional Layer 4”, produces an output o that has the same dimensions as output m. The input to “Convolutional Layer 5” is produced from the output m of the first layer as well as the output o of the fourth layer. Inputting the output from an earlier layer directly to a later layer is a “skip connection”. The outputs in this example are combined by pixel-wise addition. Alternatively, concatenation may be used, where the outputs are, for example, of different sizes. Using one or more skip connections, information from downstream is directly fed to the upstream. This maintains high-level global and regional visual features throughout the network. Inputting features directly from earlier layers to later layers and skipping one or more intervening layers provides context.

[0064] A flattening layer is included after the last convolutional layer. The flattening layer converts the output data from the last convolutional layer into a one-dimensional vector x for input to the next layer. The layers before the flattening layer in this example form the first part 46 of the CNN.

[0065] One or more fully connected layers are included after the flattening layer. The final fully connected layer outputs a value corresponding to the positive class. An activation function, for example a sigmoid function, is applied at the output to give a probability value. The activation function takes as input the output from the last fully connected layer and normalizes it to a probability. Thus, the activation function outputs a value between 1 and 0 corresponding to the positive class. The fully connected layer(s) and the activation function form the second part 47 of the first CNN 40.

[0066] For each tile, the CNN 40 outputs a probability that the tile is assigned to a positive class, i.e., that the tile is associated with a biomarker. A biomarker is a naturally occurring molecule, gene, or characteristic from which a particular pathological or physiological process, disease, diagnosis, therapy, or prognosis can be identified. In a particular example described herein, the biomarker is a cancer biomarker, i.e., a naturally occurring molecule, gene, or characteristic from which a particular type of cancer, or a particularly effective cancer treatment, can be identified. Furthermore, in an example described herein, the biomarker is a molecular biomarker. In this example, the particular biomarker is an ER biomarker. However, other examples of biomarkers are described in more detail below.

[0067] In S203, a first threshold is applied to the output of the first classifier 40. In other words, the probability value output from the first CNN 40 is compared to the first threshold. In the example shown in FIG. 3, the probability value output from the first classifier is 0.764. The first threshold is a higher prediction threshold T1. In this example, the first threshold T1 is 0.6. If the output of the first classifier is higher than the higher prediction threshold, the first classification output is 1, which corresponds to a positive class. If the output of the first classifier is lower than the higher prediction threshold, the first classification output is 0. FIG. 6 is a schematic diagram showing the first prediction threshold indicated by a line T1 on a plot of frequency against classifier model prediction score.

[0068] In S203, a second threshold is also applied to the output of the first classifier. In other words, the probability values ​​output from the first CNN 40 are compared separately to the second threshold. The second threshold is a lower prediction threshold T2. In this example, the second threshold T2 is 0.45. If the output of the first classifier is higher than the lower prediction threshold, the second classification output is 1, which corresponds to a positive class. If the output of the second classifier is lower than the lower prediction threshold, the second classification output is 0. The second prediction threshold is indicated by the line T2 in FIG. 7.

[0069] The prediction threshold T1 for the first classification is higher than the prediction threshold T2 for the second classification, so that the first classification is configured to correctly predict more negative cases than the second classification, and the second classification correctly predicts more positive cases than the first classification.

[0070] The first predictive threshold T1 for the first classification is higher than the second predictive threshold T2 for the second classification. The values ​​of the thresholds are determined during a training phase carried out prior to use of the system. The training phase is described below with respect to FIG. 8(a).

[0071] In S204, a final classification prediction is determined based on the first classification output and the second classification output. In this step, if the first classification outputs 1 and the second classification outputs 1, the classification prediction is 1. If the first classification outputs 0 and the second classification outputs 0, the classification prediction is 0. If the first classification and the second classification output different classifications, no classification prediction is made. Some further steps may be performed to determine the final classification. For example, the image may be manually classified by an expert.

[0072] In this method, by using two different prediction thresholds, two different classifications can be implemented using a single trained classifier with a single training process. The second classification M2 uses a lower prediction threshold T2, thus providing classification with high sensitivity and generating more reliable positive predictions. The lower prediction threshold is also referred to herein as the negative threshold. The first classification M1 uses a higher prediction threshold T1, thus providing classification with high specificity and generating more reliable negative predictions. The higher prediction threshold is also referred to herein as the positive threshold. By simultaneously deploying two classifications, a higher degree of certainty can be achieved in classification predictions than by applying a single classification using the first classifier with a threshold of 0.5.

[0073] 8(a) is a schematic diagram of a method for training a system for classification according to a first example, which is used to train a system that can be used in the classification method according to the first example described above.

[0074] A training data set and a validation data set, each comprising a plurality of input images, are used in the training method. The images correspond to the intended type of input images for use in the classification method. In the example described here, the input images are image tiles from images of histological sections stained with hematoxylin and eosin stains, as described above. Thus, the training data set and the validation data set each comprise image tiles from images of histological sections stained with hematoxylin and eosin stains.

[0075] Each image tile is labeled according to whether it corresponds to a particular biomarker that the model is to detect. As explained above, the particular biomarker is the ER biomarker in this example. Each image tile is labeled with a 1 if it corresponds to an ER biomarker and with a 0 if it does not correspond to an ER biomarker.

[0076] To generate the labels, for example, information from the IHC staining process may be used. For some datasets, experts may review the IHC stained images and determine the ER status of the target images, for example, if they are not already available as metadata. These are then used as ground truth labels for the corresponding H&E images during training. Various tests of human samples from patients through means of genetic, transcriptomic, and / or immunological analysis may be used in addition or as an alternative. These tests are performed on human samples called biopsies in liquid and / or solid form, which then undergo a procedure to inform the molecular status of the sample. The results are then analyzed by experts, by pathologists for tissue biopsies, hematologists for liquid biopsies, cytopathologists for cytology samples, geneticists for genetic / transcriptomic analysis, etc., to generate a label 1 or 0 for the corresponding H&E image. The annotation may be performed by trained pathologists.

[0077] To prepare the dataset, the same image pre-processing is performed as intended for the input image. Thus, for each full slide image, multiple image portions (tiles) are obtained. Cell segmentation can be used to discard tiles that contain only non-cancerous tissue from the dataset.

[0078] Image annotation can be performed separately for each tile of the full slide image. Alternatively, annotation can be performed for the full slide image, and then each tile from the image is labeled with the annotation given to the full slide image. Thus, in this case, a step of pairing each tile with the label of their corresponding slide is performed.

[0079] Thus, the training and validation datasets each comprise a number of image tiles, each of which is labeled according to whether it corresponds to a particular biomarker that the model is to detect.

[0080] In S801, image tiles in the training dataset are submitted to a first classifier, a first CNN 40, which generates a probability that the image is assigned to a positive class in the same manner as described above with respect to the inference (deployment) stage.

[0081] The first CNN 40 has an associated parameter vector θ1. The parameters include filter weights for all of the convolutional layers in the first portion of the first CNN 40 as well as weights for the second portion of the first CNN 40. The objective of the training process in S801 is to find a parameter vector θ1′ such that the difference between the annotations of the images in the training dataset and the output of the first classifier for the images is minimized.

[0082] The optimal parameters are first assigned a random value as θ1, and then the gradient of the loss

[0083]

number

[0084] D1 represents the loss function, which is the "per tile" loss. In this example, the cross-entropy loss is used.

[0085]

number

[0086] where y i is the annotation value (1 or 0), and p iis the classifier output value for an example i in the training dataset. The gradient of the loss with respect to each of the trainable parameters of the classifier model is determined through backpropagation. The gradient is then used to determine updated parameters using an optimizer function. This family of update methods is generally known as gradient descent (GD), which is iteratively defined as follows:

[0087]

number

[0088] where μ1 is the learning rate that defines how quickly the parameters are updated. The updates may be performed based on batch averages. For example, a batch size of eight tiles may be used. The average loss over the eight tiles is calculated and the model parameters are updated based on that average loss. Thus, the classifier is applied for eight tiles before the parameters are updated.

[0089] The Adam optimization algorithm may be used. However, the optimization strategy selected may depend on the performance of each strategy for the use case. For example, one of the following optimization methods may be selected: Stochastic Gradient Descent (SGD) · AdaDelta Adam AdaMax Nesterov Adam Optimizer · RMSProp Training may be performed over multiple epochs, where the number of epochs is the number of times the classifier is applied to the entire training dataset during training. During each training epoch, the training examples in the training dataset may be input to the model in a random order, such that the training examples are input in a different order for each epoch.

[0090] The classifier model performance may be evaluated after each epoch using the validation dataset. A different subset of data from the validation dataset may be used for evaluation after each epoch. After each training epoch, a classifier model accuracy based on a threshold of 0.5 may be determined. Each image tile from the validation dataset is submitted to a first classifier, which generates a probability that the image is assigned to a positive class. The classifier model output is then compared to a threshold of 0.5 to give a classification for the image. It is then determined whether the classification is correct by comparing with the annotation value (label) for the image. Accuracy is determined as the number of correct classifications divided by the number of classifications performed on the validation data. The accuracy value is an indication of the classifier model performance. Once the classifier model achieves a desired performance, training is stopped (i.e., no further epochs are performed) and the classifier model parameters are fixed. The desired performance may be determined by meeting a minimum accuracy (e.g., 0.8 or 0.9) or by the change in accuracy between the current epoch and the previous epoch being less than a predetermined value. Instead of monitoring accuracy, AUC or loss may be used to monitor, and may be implemented, for example, in a similar manner.

[0091] Then, in S802, given the trained model, prediction thresholds T1 and T2 are determined using the validation dataset. In this step, it is assumed that the validation set is largely similar to the test set. In S802, the image tiles in the validation dataset are submitted to a first classifier, the first CNN 40, which generates a probability that the image is assigned to the positive class in the same manner as described above with respect to the inference (unfolding) stage. The classifier model prediction score, i.e., the probability value output from the classifier model, is then compared with the label of the image. In this example, from the validation dataset results, the highest probability value for an image with label 0 (in the negative class) is taken as the first prediction threshold T1. The lowest probability value for an image with label 1 (in the positive class) is taken as the second prediction threshold T2. Thus, the first prediction threshold T1 is set to a specificity threshold and the second prediction threshold T2 is set to a sensitivity threshold, as determined from the results of running the first classifier model on the validation dataset. By setting the first prediction threshold T1 to a specificity threshold and the second prediction threshold T2 to a sensitivity threshold, the classification method can generate reliable predictions for both positive and negative classes.

[0092] As described above with respect to FIG. 2(b), the performance of the full classification method, in which the first classification is applied using a first threshold T1 and the second classification is applied using a second threshold T1, may be assessed using a further test data set. For example, the accuracy of the full classification method may be determined from the test data set.

[0093] In this example, the first prediction threshold T1 is set to the specificity threshold and the second prediction threshold T2 is set to the sensitivity threshold. However, in an alternative example, the first prediction threshold T1 is set lower than the specificity threshold and the second prediction threshold T2 is set higher than the sensitivity threshold, while still ensuring that the first prediction threshold T1 is higher than the second prediction threshold T2. This increases the number of cases that can be predicted using the classification method. For example, it is common in medical diagnostic systems that an acceptable level of uncertainty or error can be introduced if the benefits outweigh the costs of such uncertainty or error. In a sensitivity / specificity context, this can be achieved by increasing the lower prediction threshold T2 (used by M2) to a value slightly above the sensitivity threshold and decreasing the upper prediction threshold T1 (used by M1) to a value slightly below the specificity threshold. By doing this, the number of cases that the classification method can predict is increased.

[0094] Assuming a trained classifier model with relatively good performance, the prediction threshold is determined in S802 as described above. In the example described above, which assumes that the validation set is largely similar to the test set, T1 is set to the specificity threshold and T2 is set to the sensitivity threshold for the validation data set to generate reliable predictions for both classes. With the threshold set to the sensitivity threshold, the classifier model achieves 100% PPV on the validation data set. PPV is the positive predictive value and is given by 100×TPR / (TPR+FPR). With the threshold set to the specificity threshold, the classifier model achieves 100% NPV on the validation data set, where NPV is the negative predictive value (NPV) and is given by 100×TNR / (FNR+TNR).

[0095] FIG. 5 is a schematic diagram showing a training pipeline and deployment of a trained classification system according to a first example. A first classifier is trained in S801 using a training dataset as described above. The first classifier is then operated on a validation dataset to generate a set of model prediction scores as shown in FIG. 5. A first threshold T1 and a second threshold T2 are then selected in S802 based on the set of model prediction scores. The first threshold T1 and the second threshold T2 are determined in an automated manner. For example, the first prediction threshold T1 is set to a specificity threshold, and the second prediction threshold T2 is set to a sensitivity threshold.

[0096] Then, in S501, the classification method is performed on a test data set to evaluate the accuracy of the classification method. In S502, it is determined whether the classification accuracy on the test data set, evaluated only for test cases where the first and second classifications match, is sufficient for the intended purpose. For example, depending on the intended application, an accuracy of 0.90 or 0.95 may be sufficient. If the accuracy is sufficient, the system is deployed as described above with respect to FIG. 2(b). If not, the system is discarded.

[0097] During development, an input image is received in S201 and analyzed by a first classifier in S202. Predictions of the first and second classifications are output in S203. If the predictions match, a final classification prediction is made in S204. If they do not match, further steps, such as manual testing, may be performed.

[0098] The performance of a predictive classifier model may be measured by one or both of the following: Accuracy: the number of correct predictions / predictions made by a classifier model using a particular prediction threshold value (e.g. 0.5), and · AUC: The area under the receiver operating characteristic curve (ROC curve), which serves as a measure of accuracy across all prediction thresholds.

[0099] An ROC curve is a plot of true positive rate (TPR) against false positive rate (FPR) for varying thresholds. The true positive rate, also known as sensitivity, is the proportion of positive cases that the classifier model correctly predicts as positive using a threshold. The false positive rate is the proportion of negative cases that the classifier model incorrectly predicts as positive using a threshold. The accuracy and AUC (for a particular threshold, such as 0.5) of a trained classifier model may be evaluated using a test data set. In a medical use case, a clinical-grade medical system may use a classifier model with an accuracy and AUC of, for example, 0.94-0.98.

[0100] Classifier models with lower accuracy and AUC may be used for applications where the intended use allows for some margin of error, or where the benefits of true prediction outweigh the risks and impacts of false prediction, or where there is a human in the loop to verify the prediction. However, in many use cases, having such a classifier model means that the device is not economically feasible, for example, because a human must verify the prediction. This may prevent the adoption of the device in clinical practice. Classifier model performance may be improved, for example, by further training of the classifier model on a new training data set. However, in some cases, further improvements to the classifier model performance are not possible, for example, further training data cannot be easily obtained.

[0101] However, the sensitivity and specificity of a classification method can be adjusted by selecting the prediction threshold used with the classifier model. Figure 9 shows a schematic diagram of the transition from perfect sensitivity (no false negatives) to perfect specificity (no false positives) as observed by traversing the ROC curve from right to left.

[0102] In the classification method described above, a first classification with high or perfect sensitivity (few false negatives or no false negatives) and a second classification with high or perfect specificity (few false positives or no false positives) are used, which are assessed on a validation data set.Accuracy of the classification method can thus be improved.

[0103] In the example described above, the first and second prediction thresholds are determined using a validation data set. However, in an alternative example using a training set, a validation set, and a test set, training is performed using the training set in S801, the performance of the training is verified using the validation set, and then the thresholds are determined using the test set.

[0104] In the training method described with respect to FIG. 8(a), the training data set and the validation data set correspond to the same population. However, in alternative embodiments, the training data set and the validation data set may correspond to different populations. For example, the training data set may correspond to a general population for which a large amount of data is available. The validation data set may correspond to a specific population for which a smaller amount of data is available and which more closely corresponds to the intended use case population. For example, the training data set may correspond to people from one country or region and the validation data set may correspond to people from another country or region. The validation data set may correspond to patients in a particular hospital. The validation data set may correspond to patients with a particular feature or characteristic, such as an age range. In this way, the first classifier is trained using the large training data set, in other words the parameters of the first classifier are updated using the large training data set. The validation data set is then used to determine the first and second thresholds. This may be considered a "calibration" phase, where the first and second thresholds are determined based on the validation data set which corresponds to a population closer to the intended use population.

[0105] 8(b) is a schematic diagram of a method for training a system for classification according to a first example, where the validation data set corresponds to a different population than the training data set. The method is used to train a system that may be used in the classification method according to the first example described above.

[0106] A training data set and a validation data set, each comprising a plurality of input images, are used in the training method. The images correspond to the intended type of input images for use in the classification method. In the example described herein, the input images are image tiles from images of histological sections stained with hematoxylin and eosin stains, as described above. Thus, the training data set and the validation data set each comprise image tiles from images of histological sections stained with hematoxylin and eosin stains. The training data set comprises images corresponding to a first population P1, and the validation data set comprises images corresponding to a second population P2. The second population P2 is a target population, in other words, the second population P2 corresponds more closely to a population for the intended use case than the first population P1. For example, the training data set may comprise images of tissues from patients of all ages, while the validation data set comprises images of tissues from patients in a particular age range.

[0107] Each image tile is labeled according to whether it corresponds to a particular biomarker that the model is to detect, as described above. As described above, the particular biomarker is an ER biomarker in this example. Each image tile is labeled with 1 if it corresponds to an ER biomarker, and with 0 if it does not correspond to an ER biomarker. Thus, the training dataset and the validation dataset each comprise a plurality of image tiles, and each tile is labeled according to whether it corresponds to a particular biomarker that is to be detected.

[0108] In S801, image tiles in the training dataset are submitted to a first classifier, the first CNN 40, which generates a probability that an image is assigned to a positive class. As explained above, an updated parameter vector θ1′ is determined such that the difference between the annotation of the image in the training dataset and the output of the first classifier for the image is minimized. The untrained first classifier is trained using a training dataset corresponding to the first population P1. The output of S801 is a trained classifier trained on the first population P1.

[0109] As explained above, after each training epoch, the classifier accuracy based on a threshold of 0.5 may be determined. Each image tile from the validation dataset is submitted to a first classifier, which generates a probability that the image is assigned to a positive class. The classifier output is then compared to a threshold of 0.5 to give a classification for the image. The results are shown diagrammatically in a first graph, labeled "Result 1" in the figure, which shows the model prediction scores. All scores above 0.5 are predicted as positive and all scores below 0.5 are predicted as negative. The true classification is also shown in the first graph, with darker shading indicating that the true value is positive.

[0110] It is determined whether the classification is correct by comparing with the annotation values ​​(labels) for the image. Accuracy is determined as the number of correct classifications divided by the number of classifications performed on the validation data. The accuracy value is an indication of the classifier model performance. Once the classifier model achieves the desired performance, training is stopped (i.e., no further epochs are performed) and the classifier parameters are fixed.

[0111] Then, in S802, given the trained model, prediction thresholds T1 and T2 are determined using a validation data set. The validation data set corresponds to a second population P2, the target population. Thus, S802 can be considered as a calibration process. The trained model is taken as input and prediction thresholds are determined.

[0112] As previously described, in S802, the image tiles in the validation dataset are submitted to a first classifier, which generates a probability that the image is assigned to a positive class in the same manner as described above with respect to the inference (unfolding) stage. The classifier model prediction score, i.e., the probability value output from the classifier model, is then compared with the label of the image. The classifier prediction score is shown in the graph "Result 2". In this example, from the validation dataset results, the highest probability value for an image with label 0 (in the negative class) is taken as the first prediction threshold T1. The lowest probability value for an image with label 1 (in the positive class) is taken as the second prediction threshold T2. Thus, as determined from the results of running the first classifier model on the validation dataset, the first prediction threshold T1 is set to a specificity threshold and the second prediction threshold T2 is set to a sensitivity threshold. By setting the first prediction threshold T1 to a specificity threshold and the second prediction threshold T2 to a sensitivity threshold, the classification method can generate reliable predictions for both the positive and negative classes.

[0113] As described above with respect to FIG. 2(b), the performance of the complete classification method, in which the first classification is applied using a first threshold T1 and the second classification is applied using a second threshold T1, may be assessed using a further test data set. For example, the accuracy of the complete classification method may be determined from the test data set. The test data set in this example also corresponds to the second population P2.

[0114] In this example, the first prediction threshold T1 is set to the specificity threshold and the second prediction threshold T2 is set to the sensitivity threshold. However, in an alternative example, the first prediction threshold T1 is set lower than the specificity threshold and the second prediction threshold T2 is set higher than the sensitivity threshold, while still ensuring that the first prediction threshold T1 is higher than the second prediction threshold T2.

[0115] The calibration step in S802 allows the model to be adjusted, for example, based on the target hospital population, in other words, adjusted for population shift. This is done by determining prediction thresholds T1 and T2 that should be adjusted for the distribution change of the prediction scores resulting from the population change. An intermediate step of adjusting the weights of the model to adjust for the population change may also be included. For example, a second training data set corresponding to a second population P2 may be used to perform a further training process after the training process performed using the first training data set. In this further training process, the parameters of the first classifier are further adjusted based on the data corresponding to the target population. The first and second thresholds are then determined in S802 as described above.

[0116] In the above described example, the first classifier is trained, and then the first and second prediction thresholds are determined according to the target population. During the calibration stage S802, the input image is taken as the input to the first classifier. The value output from the last layer before prediction is output as the predicted score. Thus, a target population predicted score distribution is generated. Then the first and second thresholds are determined.

[0117] Table 1 shows results produced using a first method of classification according to a first example, which classifies images from a test data set as corresponding to an ER biomarker. The method uses the classifier model described above along with a first and second threshold. Table 1 also shows results produced using a first method of classification according to a comparative example, which classifies images from a test data set as corresponding to an ER biomarker. The method uses the same classifier model as described above along with a single threshold of 0.5.

[0118] Table 1 also shows results produced using a second method of classification according to a first example, which classifies images from a test data set as corresponding to a PR biomarker. The method uses the classifier model described above with a first and second threshold. Table 1 also shows results produced using a second method of classification according to a comparative example, which classifies images from a test data set as corresponding to a PR biomarker. The method uses the same classifier model described above with a single threshold of 0.5.

[0119] [Table 1]

[0120] In Table 1 above, PLR is the positive likelihood ratio and NLR is the negative likelihood ratio. Modification of the first example 10(a) is a schematic diagram of a method for training a system for classification according to a modification of the first example, where a lower prediction threshold T2 can be set to a value above the sensitivity threshold and an upper prediction threshold T1 can be set to a value below the specificity threshold. By doing this, the number of cases that the model can predict can be increased.

[0121] In S1001, the first classifier 40 is trained using the training data set in the same manner as described above with respect to S801.

[0122] In S1002, the sensitivity and specificity thresholds are determined using the validation data set in the same manner as described above with respect to S802.

[0123] In S1003, the first threshold and the second threshold are determined using an optimization process. In the method according to the modification of the first example, instead of setting the first threshold and the second threshold as the specificity and the sensitivity threshold, respectively, an optimization process is performed to determine the first threshold and the second threshold.

[0124] An exemplary optimization process that may be implemented in S1003 will now be described with respect to FIG. 10(b).

[0125] In S901, a first set of possible values ​​is obtained for the first threshold and a second set of possible values ​​is obtained for the second threshold. All of the values ​​in the first and second sets are greater than or equal to the sensitivity threshold and less than or equal to the specificity threshold. The first set of possible values ​​is obtained by taking a fixed number of evenly spaced values ​​between the sensitivity threshold and the specificity threshold, including the specificity threshold but not including the sensitivity threshold. The second set of possible values ​​is obtained by taking a fixed number of evenly spaced values ​​between the sensitivity threshold and the specificity threshold, including the sensitivity threshold but not including the specificity threshold.

[0126] In this example, the specificity threshold is 0.65 and the sensitivity threshold is 0.4. A set of evenly spaced values ​​in increments of 0.025 are taken as the first set of values ​​and the second set of values. The first set of values ​​comprises the values ​​{0.425, 0.450, 0.475, 0.500, 0.525, 0.550, 0.575, 0.600, 0.625, 0.650}. The second set of values ​​comprises the values ​​{0.400, 0.425, 0.450, 0.475, 0.500, 0.525, 0.550, 0.575, 0.600, 0.625}.

[0127] At S902, an accuracy value is calculated for each of a plurality of combinations of a possible first threshold and a possible second threshold. The accuracy is determined from a test data set. In this step, each value from the first set is combined with each lower value from the second set. Inputs from the test data set are run through a first classifier, and output probability values ​​from the first classifier are compared to the first and second thresholds in the combination to give a classification prediction. The accuracy for the combination is then determined as the number of correct classification predictions divided by the total number of classification predictions. This is repeated for each of the plurality of combinations.

[0128] The result of this step is shown in this example in Figure 10(c). Values ​​from the first set {0.425, 0.450, 0.475, 0.500, 0.525, 0.550, 0.575, 0.600, 0.625, 0.650} are shown along the top of the grid. Values ​​from the second set {0.400, 0.425, 0.450, 0.475, 0.500, 0.525, 0.550, 0.575, 0.600, 0.625} are shown along the left side of the grid. The remaining cells in the grid either show accuracy values ​​(for cells corresponding to possible combinations) or are empty (for cells corresponding to combinations where the value from the second set is lower than the value from the first set). Each value from the first set is combined with each lower value from the second set, resulting in combinations with corresponding accuracy values: (0.425, 0.400), (0.450, 0.400), (0.450, 0.425), ... (0.650, 0.600), (0.652, 0.600). For each combination, the first threshold is higher than the second threshold. Figure 10(c) shows the accuracy values ​​calculated for each of the combinations.

[0129] In S903, all combinations of the first and second thresholds that result in a precision score greater than or equal to the desired precision for the intended application are retrieved. The desired precision is a predetermined precision value. In this example, the desired precision is 0.9. In this example, the combinations (0.575, 0.400), (0.600, 0.400), (0.600, 0.425), (0.625, 0.400), (0.625, 0.425), (0.625, 0.450), (0.625, 0.475), (0.650, 0.400), (0.650, 0.425), (0.650, 0.450), (0.650, 0.475) result in a precision greater than 0.9. Thus, these combinations are retrieved in S903.

[0130] In S904, one combination is selected from the retrieved combinations. The combination is selected based on coverage. For example, the combination among the retrieved combinations that predicted the classification for the largest number of results from the test data set is selected. Alternatively, the combination with the largest difference between the first threshold and the second threshold is selected. In this example, the combination (0.625, 0.475) is selected as the one with the largest coverage. Therefore, in S1003, the first threshold is set to 0.625, and the second threshold is set to 0.475. Second Example Fig. 11(a) shows a method of classification according to a second example. In the method described with respect to Fig. 11(a), the classification method is performed on the same input image data as described with respect to the first example. Fig. 11(b) shows a schematic diagram of the method including the input data.

[0131] The method according to the second example also uses a first classification and a second classification. However, the first classification and the second classification use different classifiers. The first classification uses a first classifier that is configured to correctly predict more negative cases than the second classifier used for the second classification. The second classifier is configured to have a higher true positive rate than the first classifier. In the second embodiment, two separate classifiers are trained, one of which has high specificity and low sensitivity, and the other of which has high sensitivity and low specificity.

[0132] In S1101, input data is obtained in the same manner as described above with respect to S201.

[0133] In S1102, input image data 41 (image tiles) are taken as input to a classification model. The classification model comprises a first classifier and a second classifier. Thus, the input image data 41 (image tiles) is taken as input to the first classifier and, separately, as input to the second classifier. In this example, the first classifier is a convolutional neural network (CNN) 40, as described above with respect to FIG. 4. The second classifier is also a convolutional neural network (CNN) 40, as described above with respect to FIG. 4. In this example, the first classifier and the second classifier have the same model architecture, but may be trained using different training data sets and therefore have different parameter values. The structure of the classifiers, as described above with respect to FIG. 4, is the same. However, in alternative embodiments, the first classifier and the second classifier have different model architectures. For example, the first and second classifiers may have a different number of layers, a different number of filters in a layer, or different skip connections.

[0134] In S1103, a first threshold is applied to the output of the first classifier 40a. In other words, the probability value output from the first CNN 40a is compared to the first threshold. In the example shown in FIG. 11(b), the probability value output from the first classifier 40a is 0.764. In this example, the first prediction threshold used with the first classifier 40a is a specificity threshold for the first classifier 40a determined from the validation data set. If the output of the first classifier is higher than the first prediction threshold, the first classification output is 1. If the output of the first classifier is lower than the first prediction threshold, the first classification output is 0.

[0135] In S1103, a second threshold is applied to the output of the second classifier 40b. In other words, the probability values ​​output from the second CNN 40b are compared to the second threshold separately. In this example, the second prediction threshold used with the second classifier 40b is a sensitivity threshold for the second classifier 40b determined from the validation data set. If the output of the second classifier 40b is higher than the second prediction threshold, the second classification output is 1. If the output of the second classifier is lower than the second prediction threshold, the second classification output is 0.

[0136] In this example, the first predictive threshold used with the first classifier 40a is a specificity threshold for the first classifier 40a determined from the validation data set, and the second predictive threshold used with the second classifier 40b is a sensitivity threshold for the second classifier 40b determined from the validation data set. How these thresholds are determined is described below with respect to FIG. 12. However, in an alternative example, the first and second thresholds are set to the same predetermined value. For example, the first and second thresholds may be set to 0.5.

[0137] The first classification is configured to correctly predict more negative cases than the second classification. The second classification is configured to correctly predict more positive cases than the first classification. A first classifier 40a used for the first classification, configured to correctly predict more negative cases, is obtained during a training phase performed prior to deployment. A second classifier 40b used for the second classification, configured to correctly predict more positive cases, is also obtained during the training phase. The second classifier 40b is selected to have a higher true positive rate than the first classifier.

[0138] In S1104, a classification prediction is determined based on the first classification output and the second classification output. In this step, if the first classification outputs 1 and the second classification outputs 1, the classification prediction is 1. If the first classification outputs 0 and the second classification outputs 0, the classification prediction is 0. If the first classification and the second classification output different classifications, no classification prediction is made. Then, some further steps can be performed to determine the final classification. For example, the image can be manually classified by an expert.

[0139] 12(a) is a schematic diagram of a method for training a classification system according to the second example, which is used to train a classification system that can be used in the classification method according to the second example described above.

[0140] A training data set and a validation data set are used, each comprising a plurality of images, which correspond to an intended type of input image. As previously described, the training data set and the validation data set may correspond to the same population, or the training data set may correspond to a first population and the validation data set may correspond to a second, target population.

[0141] In the example described herein, the input images are image tiles from images of histological sections stained with hematoxylin and eosin stains. Thus, the training data set and the validation data set each comprise image tiles from images of histological sections stained with hematoxylin and eosin stains. As described with respect to the first example, each image tile is also labeled according to whether it corresponds to a particular biomarker that the model is to detect.

[0142] In the example described herein, the training data set is split into five separate training data subsets. Each image in the training data set is randomly assigned to one of the five separate training data subsets. Thus, each training data subset comprises different images. Each subset has the same or substantially the same number of images.

[0143] In S1201, five separate classifiers A, B, C, D and E are trained. Each classifier has the same model architecture but uses a different training data subset.

[0144] Image tiles from the first training data subset are submitted to classifier A, which generates a probability of being assigned to a positive class. Classifier A has an associated parameter vector θA. The optimal parameters are determined by assigning a random value for θA and then calculating the gradient of the loss

[0145]

number

[0146] θA is calculated by continuously updating θA by calculating θA and updating θA using the calculated gradients. DA represents the loss function, which is a "per tile" loss. As previously described, cross-entropy loss is used. The gradients of the loss with respect to each of the trainable parameters of the first classifier are determined through backpropagation. The gradients are then used to determine updated parameters using an optimizer function, as previously described.

[0147] Image tiles from the second training data subset are submitted to classifier B, which generates a probability of being assigned to the positive class. Classifier B has an associated parameter vector θB. The optimal parameter is to assign a random value as θB and then calculate the gradient of the loss

[0148]

number

[0149] The gradients are then used to determine the updated parameters using an optimizer function, as previously described. In this example, the same learning rate μ is used for each classifier.

[0150] The third, fourth and fifth training data subsets are used in a similar manner to train classifiers C, D and E, respectively. Thus, in this example, the only difference between the five classifiers is the training data used for the training process in S1201.

[0151] For each classifier, model performance is evaluated using data from the validation dataset, as previously described. Once training achieves the desired performance, model parameters are fixed for each classifier.

[0152] In S1202, the same data from the validation dataset is input to each of the five classifiers. Then, the prediction scores, i.e., the probability values ​​output from the classifiers, are compared with the image labels. Then, for each classifier, a specificity threshold and a sensitivity threshold are calculated. For each classifier, the validation dataset results are used to determine the highest output probability value for an image with label 0 (in the negative class), which is the specificity threshold. For each classifier, the lowest output probability value for an image with label 1 (in the positive class) is determined as the sensitivity threshold. The classifier corresponding to the highest specificity threshold is selected as the first classifier, which can correctly predict more negative cases than the other classifiers. The classifier with the lowest sensitivity threshold is selected as the second classifier, which can correctly predict more positive cases than the other classifiers. The first prediction threshold to be used with the first classifier is set to the specificity threshold of the selected classifier. A second prediction threshold to be used with the second classifier is set to the sensitivity threshold of the selected classifier. In this second example, two models are selected, a first model for predicting negative cases and a second model for predicting positive cases. A specificity threshold is then selected for the first model and a sensitivity threshold is selected for the second model. This approach can be used for applications where some models are very good at predicting only negatives and not predicting positives, while other models are very good at predicting positives and not predicting negatives.

[0153] The performance of the complete classification, in which the first classification is applied using the first classifier and the second classification is applied using the second classifier, as described above with respect to FIG. 11(a), may then be assessed using a test dataset.

[0154] In the above example, five separate classifiers are trained and the classifier corresponding to the highest specificity threshold is selected as the first classifier and the classifier with the lowest sensitivity threshold is selected as the second classifier. However, in an alternative example, only two classifiers are trained and the classifier corresponding to the higher specificity threshold is selected as the first classifier and the other classifier is selected as the second classifier. In an alternative example, three or more classifiers are trained and the classifier corresponding to the highest specificity threshold is selected as the first classifier and the classifier with the lowest sensitivity threshold is selected as the second classifier.

[0155] In the above example, the classifier corresponding to the highest specificity threshold is selected as the first classifier, and the classifier with the lowest sensitivity threshold is selected as the second classifier. However, alternatively, the classifier corresponding to the highest specificity using a threshold of 0.5 is selected as the first classifier, and the classifier with the highest sensitivity using a threshold of 0.5 is selected as the second classifier. Then, the first prediction threshold and the second prediction threshold are both set to 0.5.

[0156] In the above example, each of the multiple classifiers is trained with a separate subset of the training dataset, where each image is randomly assigned to one of the subsets. However, in alternative examples, the images may be assigned based on one or more properties of the images. For example, one or more of the subsets may have more positive examples than the others. For example, the images may be assigned such that two of the subsets have 80% positive examples, one of the subsets has 50% positive examples, and two of the subsets have 20% positive examples. In the above example, the subsets have substantially the same number of images, but alternatively, some of the subsets may have many more images than the others.

[0157] In the above example, each of the multiple classifiers has the same model architecture. However, in alternative examples, some or all of the classifiers can have different model architectures, e.g., the classifiers can have different numbers of layers, different numbers of filters in the layers, or different skip connections.

[0158] In the above example, each of the multiple classifiers is trained using the same hyper-parameters, however, in alternative examples, some or all of the classifiers may be trained using different hyper-parameters, e.g., the learning rates may be different.

[0159] FIG. 12(b) is a schematic diagram showing a training pipeline and deployment of a trained classification system according to a second example. Two classifiers are trained in S1201 using a training dataset as described above. The classifiers are then operated on a validation dataset to generate a set of model prediction scores for each classifier as shown in FIG. 12(b). A first classifier and a second classifier are then selected based on the set of model prediction scores. The first classifier and the second classifier are determined in an automated manner. For example, as determined based on the validation dataset results, the first classifier is selected as the classifier corresponding to the highest specificity threshold, and the second classifier is selected as the classifier with the lowest sensitivity threshold.

[0160] Then, in S501, the full classification method is performed on a test data set to evaluate the accuracy of the method. In S502, it is determined whether the accuracy on the test data set, evaluated only on the test cases for which it claims certainty (i.e., whether the first and second classifications match or not), is sufficient for the intended purpose. If the accuracy is sufficient, the system is deployed as described above with respect to FIG. 11(a). If not, the system is discarded.

[0161] During development, an input image is received in S1101 and analyzed by a first classifier and a second classifier in S1102. Predictions of the first and second classifications are output in S1103. If the predictions match, a final classification prediction is made in S1104. If they do not match, further steps may be performed, such as manual testing. Modification of the second example FIG. 12(c) is a schematic diagram of a method for training a classifier according to a modification of the second example.

[0162] In S1401, multiple classifiers are trained in the same manner as described above with respect to S1201.

[0163] In S1402, a first classifier and a second classifier are selected in the same manner as described above with respect to S1202. As part of this step, a sensitivity threshold for the second classifier and a specificity threshold for the first classifier are determined using a validation data set, as described with respect to S1202.

[0164] In the second example described above, the first prediction threshold used with the first classifier 40a is set as a specificity threshold for the first classifier 40a, determined from the validation data set, and the second prediction threshold used with the second classifier 40b is set as a sensitivity threshold for the second classifier 40b, determined from the validation data set. However, in a modified method of the second example, an optimization process is performed to determine the first and second thresholds.

[0165] In S1403, the first and second thresholds are determined using an optimization process. The exemplary optimization process described above with respect to FIG. 10(b) may be implemented in this step.

[0166] A first set of possible values ​​is obtained for the first threshold and a second set of possible values ​​is obtained for the second threshold. In this example, all of the values ​​in the first set are greater than or equal to the sensitivity threshold of the first classifier and less than or equal to the specificity threshold of the first classifier, while all of the values ​​in the second set are greater than or equal to the sensitivity threshold of the second classifier and less than or equal to the specificity threshold of the second classifier. For example, the first set of possible values ​​may be obtained by taking a fixed number of evenly spaced values ​​from the sensitivity threshold for the first classifier to the specificity threshold for the first classifier, including the specificity threshold and the sensitivity threshold. The second set of possible values ​​may be obtained by taking a fixed number of evenly spaced values ​​from the sensitivity threshold for the second classifier to the specificity threshold for the second classifier, including the sensitivity threshold and the specificity threshold.

[0167] Then, for each of a plurality of combinations of possible first thresholds and possible second thresholds, an accuracy is determined. In this step, each value from the first set is combined with each value from the second set. Inputs from a test data set are run through the first classifier and the second classifier. The first classifier model output is compared to the first threshold in the combination, and the second classifier model output is compared to the second threshold in the combination to provide a classification prediction. An accuracy is then determined for each of the combinations.

[0168] All combinations of the first and second thresholds that produce an accuracy score greater than or equal to the desired accuracy for the application are retrieved, and a combination from the retrieved combinations is selected based on coverage, e.g., the combination that classified the most results from the test data set is selected from among the retrieved combinations. Third Example In a first example, the classification method is applied tile by tile. An image tile is taken from the whole slide image and taken as input to a first classifier. A first classification using a first threshold and a second classification using a second threshold are applied. A classification prediction is then determined as to whether the image tile corresponds to an ER biomarker.

[0169] 13 is a schematic diagram of a method of classification according to a third example, where the classification method is applied on an image-by-image basis, in other words, a classification prediction is obtained as to whether every slide image corresponds to an ER biomarker.

[0170] In S1501, input data is obtained. The input image data, in this example, is an image tile from a whole slide image of tissue, as previously described. In this example, multiple image tiles are obtained from a whole slide image, as shown with respect to FIG. 13. Cell segmentation may be used to discard tiles that contain only non-cancerous tissue.

[0171] At S1502, each input image tile is taken separately as an input to a classification model that comprises a first classifier, which in this example is the first CNN 40 as described with respect to the first example. The first classifier outputs, for each image tile, a probability that the image tile is associated with an ER biomarker.

[0172] An aggregation step is then performed, which in this example comprises applying a pooling operator. The image tile that corresponds to the highest probability value is identified. This image tile is called the "top-ranked" image tile. This "top-ranked" image tile is then used to classify the images in the subsequent steps of the classification method.

[0173] In S1503, a first threshold is applied to the output of the first classifier 40 for the top-ranked image tile. In other words, the probability values ​​output from the first CNN 40 for the top-ranked image tile are compared to the first threshold to give an output of a first classification. A second threshold is applied to the output of the first classifier 40 for the top-ranked image tile. In other words, the probability values ​​output from the first CNN 40 for the top-ranked image tile are compared separately to the second threshold to give an output of a second classification. The first and second thresholds are determined during the training phase as previously described with respect to the first example.

[0174] In S1504, a final classification prediction for all slide images is determined based on the first classification output and the second classification output. In this step, if the first classification outputs 1 and the second classification outputs 1, the classification prediction is 1. If the first classification outputs 0 and the second classification outputs 0, the classification prediction is 0. If the first classification and the second classification output different classifications, no classification prediction is made. Some further steps may be performed to determine the final classification. For example, the images may be manually classified by an expert.

[0175] In the third example, during unfolding, the first CNN classifier 40 returns a probability for each tile, and these individual scores are aggregated through a max operator. Other non-trainable aggregation functions, such as averaging, may alternatively be used. The first classifier 40 is trained in the same manner as in the first example. The classification prediction for the entire slide image is taken as the classification prediction for a single "representative" tile.

[0176] In an alternative example, a trainable aggregation function is used. An indication of whether an input full slide image is associated with a biomarker is determined by inputting data from multiple tiles into an aggregation model. The aggregation model may comprise a recurrent neural network, such as a long short-term memory (LSTM) network. The output of the first classifier 40 may be used to select a reduced set of image tiles, e.g., the top k tiles based on the probabilities output from the first classifier 40, where k is an integer greater than 2. The reduced set of image tiles is taken as input to the classification model. First, a feature vector corresponding to each of the image tiles is generated using the first trained model. These feature vectors are then taken as inputs in the sequence to a second trained model, which may comprise, for example, an LSTM. The output of the LSTM corresponding to the last input in the sequence is input to a fully connected neural network layer, which outputs the probability that the input image is associated with the ER biomarker (corresponding to the positive class). This output value is then compared to a first threshold to give an output of the first classification, and to a second threshold to give an output of the second classification. A classification prediction for the whole slide image is then determined based on the outputs of the first and second classifications. During training, image tiles from the training dataset are taken as input to the classification model in the same manner as during inference. The output is compared to the labels for the whole slide image to train a first model and a second model, for example using a cross-entropy loss function and an optimizer function as previously described. In an alternative example, the classification model may comprise applying a first classification comprising a first trained model and a second trained model using a first threshold, and a second classification comprising a different first trained model and a different second trained model using a second threshold. Fourth Example In the above-described example, improved accuracy is obtained with a reduction in prediction coverage, i.e., the number of cases that the classification model can predict, for all classes.However, in some applications, the classification method is used to predict only some of the classes, for example only negative classes or only positive classes.For example, the classification method can be used to classify whether an image corresponds to a HER2 biomarker, where the model is only used to predict the negative class.

[0177] In such an example, the method of classification may use a single classifier and a single prediction threshold. For some classes, improved accuracy is obtained. For example, the classifier is only used to predict the first class, where improved accuracy is obtained for the first class. The threshold corresponding to the first class is used in the method. If the classification predicts the first class, a classification prediction of the first class is made. If the classification prediction does not predict the first class, no classification prediction is made.

[0178] As explained before, a higher prediction threshold results in a classification method with higher specificity, which can correctly predict more negative cases, but cannot correctly predict more positive cases. For example, the line "specificity threshold" in FIG. 2(a) shows the threshold at which all true negative cases in the dataset are correctly predicted. The same classifier combined with a lower prediction threshold results in a classification method with higher sensitivity, which can correctly predict more positive cases, but cannot correctly predict more negative cases. The line "sensitivity threshold" in FIG. 2(a) shows the threshold at which all positive cases in the dataset are correctly predicted. The "specificity" threshold is higher than the "sensitivity" threshold.

[0179] 16 shows a method according to the fourth example. As described with respect to the first example, classification is performed on input image data corresponding to an image of tissue. The input image is classified into a negative class if it does not correspond to the HER2 biomarker.

[0180] The method according to the fourth example uses the first classification.

[0181] In S1601, input data is obtained in the same manner as described in S201 with respect to the first example.

[0182] In S1602, input data is taken as input to a classification model. The classification model applies a first classifier. The first classifier may be a CNN, as described with respect to S202 in the first example.

[0183] In S1603, a threshold is applied to the output of the first classifier. In other words, the probability value output from the first classifier is compared to the threshold. The threshold corresponds to a threshold for the negative class in this example. For example, the threshold may correspond to a sensitivity threshold determined using a data set.

[0184] In S1604, a final classification prediction for the negative class is determined based on the first classification output from S1603. In this step, if the first classification outputs 0, the classification prediction is 0. If the first classification outputs 1, no classification prediction is made. Some further steps may be performed to determine the final classification. For example, the image may be manually classified by an expert.

[0185] In this example, the first classification is used to predict the negative class. In an alternative example, the first classification is used to predict the positive class. In this case, a higher prediction threshold is used, such that the higher prediction threshold corresponds to the threshold for the positive class. For example, the threshold may correspond to a specificity threshold determined using the data set.

[0186] 17 is a schematic diagram of a method for training a system for classification according to the fourth example, which is used to train a system that can be used in the classification method according to the first example described above.

[0187] A training data set and a validation data set, each comprising a plurality of input data, are used in the training method. As previously described, the input data correspond to the intended type of input data for use in the classification method. As previously described, the training data set may correspond to a first population, and the validation data set may correspond to a target population. In this example, as described with respect to the first example, the input data comprises images of tissues that are labeled according to whether the images of tissues correspond to a particular biomarker that the model is to detect. As described above, the particular biomarker is the HER2 biomarker in this example. Each image tile is labeled with a 1 if it corresponds to the biomarker, and with a 0 if it does not correspond to the biomarker.

[0188] In S1701, the image tiles in the training dataset are submitted to a first classifier, the first CNN 40, which generates a probability that the image is assigned to a positive class. As described above, an updated parameter vector θ1′ is determined such that the difference between the annotations of the images in the training dataset and the output of the first classifier for the images is minimized.

[0189] Then, in S1702, a prediction threshold is determined using the validation dataset, assuming a trained model. The image tiles in the validation dataset are submitted to a first classifier, the first CNN 40, which generates a probability that the image is assigned to the positive class in the same manner as described above with respect to the inference (deployment) stage. The classifier model prediction score, i.e., the probability value output from the classifier model, is then compared with the label of the image. In this example, from the validation dataset results, the lowest probability value for an image with label 1 (in the positive class) is taken as the prediction threshold. Thus, the prediction threshold is set to the sensitivity threshold, as determined from the results of running the first classifier model on the validation dataset. By setting the prediction threshold to the sensitivity threshold, the classification method can generate accurate predictions for the negative class, in other words, the false negative rate, i.e., the proportion of positive cases that the model erroneously predicts as negative is reduced.

[0190] In the examples described above, the prediction threshold is set to the sensitivity threshold determined from the validation data set. However, in some examples, the prediction threshold is set to a value higher than the sensitivity threshold. For example, the prediction threshold may be determined using an optimization process. A set of possible values ​​is obtained for the threshold. In this example, all of the values ​​in the set are greater than or equal to the sensitivity threshold of the first classifier and less than or equal to the specificity threshold of the first classifier, as previously described with respect to the modification of the first example.

[0191] A score is then determined for each possible threshold. The score may be, for example, FNR or NPV. Inputs from a test data set are run through a first classifier. The first classifier model output is compared to a threshold to give a classification prediction. A score is then determined for the threshold, for example by determining an FNR for the set of predictions. This is repeated for each possible threshold.

[0192] All thresholds that produce a score greater than or equal to the desired score for the application are retrieved, and a threshold from the retrieved thresholds is selected based on coverage, e.g., the threshold that classified the most results from the test data set is selected from the retrieved thresholds, or the highest threshold is selected.

[0193] In instances where the first classification is used to predict the positive class, the prediction threshold may be set to a specificity threshold determined from the validation data set to reduce the false positive rate. Alternatively, the threshold may be set by an optimization process in a manner similar to that described above.

[0194] In a fourth example, the method of classification may use a single classifier and adjust the threshold according to the target population, for example using a validation data set corresponding to the target population. The example uses one classifier with one threshold, where the method of classification is intended to predict, for example, only negative classes or only positive classes. In some examples, the classifier is trained on a training set, the performance of the training is verified on a validation set, and then the threshold is set on a test set.

[0195] Table 2 shows results generated using the first method of classification according to the fourth example, which classifies images from a test data set as not corresponding to the HER2 biomarker. The method uses the classifier model described above, where a single threshold corresponds to the negative class. Table 2 also shows results generated using the first method of classification according to a comparative example, which classifies images from a test data set as not corresponding to the HER2 biomarker. The method uses the same classifier model described above, with a single threshold of 0.5.

[0196] [Table 2]

[0197] By understanding the cancer in general and / or the patient's particular molecular profile, various treatments performed against the cancer, including hormone therapy, immunotherapy, or targeted drug therapy, among others, can be informed. Various medically relevant biomarkers, including either diagnostic, therapeutic, or prognostic markers, including mutation status, receptor status, or copy number polymorphism, among others, can be identified to determine how well a patient will respond to some therapies. Mutation status, receptor status, or copy number polymorphism are examples of molecular biomarkers. For example, in some cases, the molecular biomarker can be protein expression levels. For example, a particular biomarker can be estrogen receptor (ER), progesterone receptor (PR), or human epidermal growth factor receptor (HER2). These pillar biomarkers are specific to breast cancer. They are the most important biomarkers for breast cancer prognosis and based on targeted therapy. ER and HER2 are most commonly associated with the cancer treatments tamoxifen and herceptin, respectively. Patients can be tested for these two biomarkers to determine suitability for these treatments.

[0198] In the example described above, the input image is classified as to whether it is associated with the ER biomarker. Alternatively, the method may be used to determine whether the input image tile is associated with the HER2 biomarker, the PR biomarker, the EGFR biomarker (associated with lung adenocarcinoma), or the MSI (associated with colon adenocarcinoma). Various molecular biomarkers may be used to classify some cancers into categories, such as breast or colorectal. For example, breast cancer has five different molecular "subtypes," each determined based on the status of ER, PR, and HER2. For example, if ER, PR, and HER2 are all negative, the molecular subtype is "basal-like." Thus, by determining the presence or absence of multiple molecular biomarkers, a molecular subtype may be predicted. A "molecular subtype" is a way of categorizing a particular type of cancer based on the presence or absence, or in some cases, based on the level of one biomarker or a set of biomarkers.

[0199] The classification method can be used to detect various other biomarkers. For example, the antigen Ki-67 is also increasingly being tested as a marker for cell proliferation indicative of cancer aggressiveness. Thus, the specific biomarker can alternatively be Ki-67. For example, as an alternative to mitotic figure counts in the grading scheme when assessing tumor proliferation in HER2- and ER+ breast cancer, a labeling index based on IHC staining of the Ki67 nuclear antigen can be used with other IHC markers. It can provide additional information for therapeutic decisions, such as any requirement for adjuvant chemotherapy. In various studies, it has been shown to be a strong predictor of survival. For example, PREDICT is an online tool that shows how different treatments for early invasive breast cancer can improve survival after surgery. PREDICT model performance has improved with the involvement of Ki67 as a prognostic marker. The manual scoring method for interpreting IHC stained Ki67 slides includes counting invading cells in randomly selected regions of interest, such as in the periphery of the tumor, and determining the percentage of Ki67 staining for all invading tumor cells. As with the conventional molecular profiling techniques described above, this process is labor intensive, subject to human error, and open to inter / intraobserver variation. For example, by predicting Ki67 index from H&E images, such a process could be made shorter and the accuracy potentially improved.

[0200] The classification method can provide automated profiling of specific biomarkers relevant for cancer diagnosis, treatment and / or prognosis. The specific biomarkers can be mutation status, receptor status or copy number variation, among other examples. Profiling is performed from whole slide H&E images in the described example, although other images or inputs can be used.

[0201] A biomarker can be a molecule or a characteristic associated with one of one or more molecules, such as, for example, the amount of a particular molecule. In some cases, the biomarker is a molecule associated with a particular cancer treatment. The biomarker can be a clinically actionable genetic mutation. In this example, the particular biomarker is an ER biomarker. Other examples of biomarkers are described in more detail below.

[0202] Alternatively, the classification prediction may be based on other properties, in some examples, the classification method classifies whether an image contains cancerous cells, for example.

[0203] In the example described above, images of histological sections stained with hematoxylin and eosin stains are used, but various types of tissue images acquired using various methods can be processed using the described methods. For example, alternatively, images of histological sections that have undergone immunohistochemistry (IHC) staining can be taken as input. IHC staining involves selectively identifying antigens in cells of a tissue section. Antibodies specifically bind to antigens in biological tissues. The staining allows visualization of antibody-antigen interactions. For example, using chromogenic immunohistochemistry (CIH), antibodies are conjugated to enzymes that can catalyze a color-producing reaction.

[0204] In the above described example, a binary classification method is implemented. However, in an alternative example, the method of classification can be used for multi-way classification. In multi-way classification, the classification model applies a first classification and a second classification for each class of a plurality of classes, where the second classification is configured to have a higher true positive rate for that class than the first classification. Given the results from the validation data set, the true positive rate for a class is determined as the proportion of cases in the class that the method correctly predicts as being in the class.

[0205] Next, an example is described in which classification into three classes is performed. A three-way classifier model is applied to the input. The first classification for the first class comprises comparing the output for the first class to a first prediction threshold. The second classification for the first class comprises comparing the output for the first class to a second prediction threshold. The first prediction threshold is higher than the second prediction threshold. The first classification for the second class comprises comparing the output for the second class to a third prediction threshold. The second classification for the second class comprises comparing the output for the second class to a fourth prediction threshold. The third prediction threshold is higher than the fourth prediction threshold. The first classification for the third class comprises comparing the output for the third class to a fifth prediction threshold. The second classification for the third class comprises comparing the output for the third class to a sixth prediction threshold. The fifth prediction threshold is higher than the sixth prediction threshold. The first prediction threshold is set as the highest output probability value for the first class from the validation data set results for inputs not in the first class (i.e., in the second or third class). The second prediction threshold is set as the lowest probability value for the first class from the validation data set results for inputs in the first class. Thus, the first prediction threshold T1 is set to the specificity threshold for the first class and the second prediction threshold T2 is set to the sensitivity threshold for the first class, as determined from the results of running the classifier on the validation data set. The third prediction threshold is set as the highest output probability value for the second class from the validation data set results for inputs not in the second class. The fourth prediction threshold is set as the lowest probability value for the second class from the validation data set results for inputs in the second class. The fifth prediction threshold is set as the highest output probability value for the third class from the validation data set results for inputs not in the third class. A sixth prediction threshold is set for inputs in the third class as the lowest probability value for the third class from the validation data set results.

[0206] Next, an alternative example is described in which classification into three classes is performed. The first classification for the first class comprises applying a first binary classifier and using a first prediction threshold, and the second classification for the first class comprises applying the first classifier and using a second prediction threshold, where the first prediction threshold is higher than the second prediction threshold. The first binary classifier is trained to determine whether the input is in the first class (positive class) or not in the first class (negative class). The first classification for the second class comprises applying a second binary classifier and using a third prediction threshold, and the second classification for the second class comprises applying the second classifier and using a fourth prediction threshold, where the third prediction threshold is higher than the fourth prediction threshold. The second binary classifier determines whether the input is in the second class (positive class) or not in the second class (negative class). The first classification for the third class comprises applying a third binary classifier and using a fifth prediction threshold, and the second classification for the third class comprises applying a third classifier and using a sixth prediction threshold, where the fifth prediction threshold is higher than the sixth prediction threshold. The second binary classifier determines whether the input is in the third class (positive class) or not in the third class (negative class). As described with respect to the first example, each classifier is trained separately on the training data set and a threshold is selected.

[0207] Next, an alternative example is described in which classification into three classes is performed. The first classification for the first class comprises applying a first binary classifier and using a first prediction threshold, and the second classification for the first class comprises applying a second binary classifier and using a second prediction threshold. The first and second binary classifiers are trained to determine whether the input is in the first class (positive class) or not in the first class (negative class). The first classification for the second class comprises applying a third binary classifier and using a third prediction threshold, and the second classification for the second class comprises applying a fourth binary classifier and using a fourth prediction threshold. The third and fourth binary classifiers are trained to determine whether the input is in the second class (positive class) or not in the second class (negative class). The first classification for the third class comprises applying a fifth binary classifier and using a fifth prediction threshold, and the second classification for the third class comprises applying a sixth binary classifier and using a sixth prediction threshold. The fifth and sixth binary classifiers are trained to determine whether the input is in the third class (positive class) or not in the third class (negative class). As described above with respect to the second example, three or more classifiers corresponding to each class are trained separately on the training data set, and the classifiers used for the first and second classifications are selected for each class. For each class, the positive class is the same for the two classifiers.

[0208] 14 is a schematic diagram of a method for classification. The method comprises deploying a predictive medical device (PMD) that outputs not only a binary predicted value, but also an indicator of whether the prediction is certain to be correct based on a previous performance of the prediction on a test dataset. For cases where the PMD is certain, a prediction is given for the case. For cases where the PMD is not certain, no prediction is given and the case will proceed to the standard of care.

[0209] In S1301, an input medical sample or data to be analyzed is obtained. In S1302, the input is entered into the PMD. In S1303, a predicted value (1 or 0) is output along with a certainty. If the first and second classifications match, the certainty is 1 and the method moves to S1305. An analysis (prediction) is provided, avoiding a standard of care analysis. If the first and second classifications do not match, the certainty is 0 and the method moves to S1304. A standard of care analysis is then performed, which may include manual inspection. This analysis may be time consuming or expensive. Once an analysis is provided in either S1305 or S1304, the method moves to S1306 to perform subsequent processes.

[0210] Figure 15 shows a schematic diagram of a classification system 1, according to one embodiment. System 1 comprises an input 11, a processor 3, a working memory 9, an output 13, and a storage 7. System 1 takes input image data and generates an output. The output may comprise diagnostic information. In particular, the output may be an indication of whether the input image is associated with a biomarker.

[0211] System 1 is a computing system, for example an end-user system such as a general purpose computer, or a server.

[0212] The processor 3 is coupled to storage 7 and has access to working memory including RAM 9. The processor 3 may comprise logic circuitry that responds to and processes instructions in code stored in the working memory.

[0213] The computer program 5 is stored in a non-volatile memory 7. The non-volatile memory 7 is accessed by the processor 3, and the stored code 5 is retrieved and executed by the processor 3. In particular, the computer program code 5, which when executed performs the methods described herein, is represented as a software product stored in a working memory 9. The execution of the code 5 by the processor 3 causes the embodiments described herein to be implemented.

[0214] The processor 3 also has access to an input module 11 and an output module 13. The input and output modules or interfaces 11, 13 may be a single component or may be split into a separate input interface 11 and a separate output interface 13.

[0215] The input module 11 is connected to an input 15 for receiving data. The input 15 may be a receiver for receiving data from an external storage medium or through a communication network. Alternatively, the input 15 may comprise hardware such as an image capture device. Alternatively, the input 15 may read data from a stored image file, which may be stored on the system or on a separate storage medium such as a floppy disk, CD ROM, magnetic device or programmable memory device.

[0216] Connected to the output module 13 is an output 17. The output 17 may comprise hardware such as a visual display. Alternatively, the output may be a transmitter for transmitting data to an external storage medium or over a communications network. Alternatively, the output 17 may write data to a stored image file, which may be stored on the system or on a separate storage medium such as a floppy disk, CD ROM, magnetic device or programmable memory device.

[0217] Storage 7 is communicatively coupled to processor 3. Storage 7 may contain data used by code 5 when executed by processor 3. As shown, storage 7 is a local memory contained in the device. However, alternatively, storage 7 may be located fully or partially remotely, for example using cloud-based memory that may be accessed remotely via a communications network (such as the Internet). Code 5 is also stored in storage 7. Code 5 is located in working memory 9 when executed.

[0218] System 1 may be located in a common system with hardware for inputting and outputting data. Alternatively, system 1 may be a remote system 1 that receives image data transmitted from a separate unit (e.g., an image capture device) and transmits output data to another separate unit (e.g., a user computer with a screen). For example, the system may be implemented on a cloud computing system that receives and transmits data.

[0219] In the described system, a single processor 3 located in the device is used, but the system may comprise two or more processors, which may be located in the same system or remotely, configured to perform different parts of the processing and to transmit data between them. For example, the system may comprise a graphical processing unit (GPU) and a general central processing unit (CPU). Various operations are implemented by the GPU, while other operations are implemented by the CPU. For example, matrix operations may be performed by the GPU.

[0220] Usual procedures for loading software into memory and storing data in storage unit 7 apply. The code 5 may be embedded in the original equipment or may be provided in whole or in part after manufacture. For example, the code may be introduced as a computer program product, which may be in the form of a download in its entirety, or via a computer program storage medium such as an optical disc. Alternatively, modifications to existing dialog manager software may be made by update or plug-in to provide the features of the described embodiments.

[0221] The method of training a classification model and the method of classification using the trained classification model may be performed on the same system. For example, the method of training a classification model and the method of classification using the trained classification model may be performed on a system as described with respect to Figure 15, where the trained classification model is stored in memory 7 once trained. Alternatively, the method of training a classification model may be performed on a first system and the trained classification model is stored on a second system, such that the method of classification is performed on the second system.

[0222] It will be appreciated that although the described embodiments are applicable to any computing system, the exemplary computing system shown in Figure 15 provides means by which the embodiments described herein can be implemented. In use, the system 1 receives data through a data input 11. A program 5 running on a processor 3 outputs data through an output 13 in the manner described with reference to the figures above.

[0223] Although several embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. Indeed, the novel methods and apparatus described herein may be embodied in a variety of other forms, and various omissions, substitutions, and modifications of the forms of the methods and apparatus described herein may be made.

Claims

1. obtaining a first dataset comprising first input data corresponding to a first class; obtaining a second data set comprising second input data corresponding to the first class; training at least one classifier using the first dataset; inputting the second input data from the second dataset into the at least one classifier, and providing a classification model comprising a first classification and a second classification, wherein the first classification predicts a greater proportion of the second input data corresponding to the first class to be in the first class than the second classification. A computer-implemented method comprising:

2. 2. The method of claim 1 , wherein the first classification comprises applying a first classifier and using a first predictive threshold, and the second classification comprises applying the first classifier and using a second predictive threshold, wherein determining the first predictive threshold comprises determining a highest output value of the first classifier for inputs from the second data set that corresponds to the first class and setting the first predictive threshold to be less than or equal to the highest output value, and wherein determining the second predictive threshold comprises determining a lowest output value for inputs from the second data set that corresponds to the second class and setting the second predictive threshold to be greater than or equal to the lowest output value.

3. 2. The method of claim 1, wherein the at least one classifier comprises at least two classifiers, and providing the classification model comprises determining a first classifier of the at least two classifiers, wherein the first classifier predicts a largest proportion of the second input data corresponding to the first class of the at least two classifiers that will be in the first class; and determining a second classifier of the at least two classifiers, wherein the second classifier predicts a largest proportion of the second input data corresponding to the second class of the at least two classifiers that will be in the second class.

4. 4. The method of claim 2, wherein providing the classification model comprises determining a first predictive threshold and a second predictive threshold, and wherein determining the first predictive threshold and the second predictive threshold comprises obtaining a plurality of possible combinations of values ​​for the first predictive threshold and values ​​for the second predictive threshold, determining accuracy of a classification model using each of the possible combinations, and selecting a combination having an accuracy higher than a predetermined accuracy value.

5. obtaining a first dataset comprising first input data corresponding to a first class; obtaining a second data set comprising second input data corresponding to the first class; training a classifier using the first data set; inputting the second input data from the second dataset into at least one classifier and providing a classification model comprising a first classification, wherein providing the classification model comprising the first classification comprises determining a prediction threshold corresponding to the first class. A computer-implemented method comprising:

6. 6. The method of claim 1, wherein the first data set comprises first input data corresponding to a first population and the second data set comprises second input data corresponding to a second population.

7. Obtaining input data; inputting the input data into a classification model; and the classification model applying a first classification and a second classification, wherein the first classification is configured to correctly predict a greater proportion of the input data corresponding to the first class than the second classification. determining a classification prediction for the input data based on the output of the first classification and the second classification; A computer-implemented method comprising:

8. 10. The method of claim 1 or 7, wherein the first classification comprises applying a first classifier and using a first prediction threshold, and the second classification comprises applying the first classifier and using a second prediction threshold, wherein the first prediction threshold is higher than the second prediction threshold.

9. The method of claim 7 , wherein the first classification comprises applying a first classifier and the second classification comprises applying a second classifier.

10. 10. The method of claim 9, wherein the first classification uses a first predictive threshold and the second classification uses a second predictive threshold, wherein the first predictive threshold is higher than the second predictive threshold.

11. The method of claim 7 , wherein the input data comprises an image of tissue, and wherein determining the classification comprises determining information related to a medical diagnosis.

12. Obtaining input data; inputting the input data into a classification model; and the classification model applying a first classification, wherein the first classification comprises applying a first classifier and using a prediction threshold corresponding to a first class. determining a classification prediction for the first class based on an output of the first classification; 1. A computer-implemented method of classification comprising:

13. One or more processors configured to perform the method of any of claims 7 or 12. A classification system comprising:

14. A classification system comprising a classification model trained according to the method of any of claims 1, 5 or 7.

15. A carrier medium comprising computer readable code configured to cause a computer to perform the method of any of claims 1 or 5 or 7 or 12.

Images