Cell type classification

Abstract

A computer-implemented method for classifying cell types includes receiving image data representing a digital test image, the digital test image depicting cells; applying a classifier model to the received image data, the classifier model configured to output cell types of cells depicted in the digital test image, the classifier model being trained using training data including a plurality of digital records, each digital record including training image data representing digital training images depicting cells and labels indicating the cell types of the cells depicted in the digital training image; determining one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image; and generating instructions that, when executed by a display component of a computing device, cause the display component to display the cell types output by the classifier model and the determined one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image. Corresponding diagnostic support and similar computer-implemented methods are also provided.

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for classifying cell types, and more particularly to a computer-implemented method for presenting a user with images of cells that are similar to a classified cell to aid the user in understanding the cell classification. [Background technology]

[0002] Modern deep learning networks are very powerful and can be quickly trained to reach very high accuracy for image classification problems (Krizhevsky, Alex & Sutskever, Ilya & Hinton, Geoffrey. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Neural Information Processing Systems. 25. 10.1145 / 3065386.). These models consist of millions of parameters and are trained end-to-end, i.e., the training data (in the form of images and labels) is used directly to optimize those parameters. No features are extracted from the images before "showing" them to the model. For this reason, these kinds of models are often treated as black boxes that cannot explain classification.

[0003] The interpretability of deep learning models has been a subject of active research for many years. However, we have found that many proposed methods are at best useful as debugging tools and fail to provide meaningful explanations for why a particular classification was made. For example, saliency maps merely highlight the regions that most influence the classification (e.g., dogs are included in the dog classifier, but there is no explanation for why the model selected the class "dog" over other mammals). In some cases, proposed methods have been shown to be insensitive to the model's parameters (https: / / arxiv.org / abs / 1810.03292). Summary of the Invention

[0004] At a high level, the present invention provides a computer-implemented method for classifying cell types using a classifier model, the computer-implemented method further comprising identifying and displaying to a user cells from a reference set of cells that are most similar to the cell being classified, in this manner allowing a user, such as a clinician, to gain better insight into why a particular cell is classified as such.

[0005] Accordingly, a first aspect of the present invention provides a computer-implemented method for classifying cell types, the method comprising: receiving image data representing a digital test image, the digital test image depicting a cell; applying a classifier model to the received image data, the classifier model being configured to output cell types of cells depicted in the digital test image, the classifier model being trained using training data comprising a plurality of digital records, each digital record including training image data representing digital training images depicting a cell and a label indicating the cell type of the cell depicted in the digital training image; determining one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image; and generating instructions that, when executed by a display component of a computing device, cause the display component to display the cell types output by the classifier model and the determined one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image.

[0006] In various embodiments of the present invention, the reference image may take different forms. For example, in some cases, the one or more reference images may include a plurality of training images that form part of the training data. Additionally or alternatively, the one or more reference images may include reference images stored in or retrieved from a reference image database. The reference images stored in the reference image database may be the same as the plurality of digital training images, or may include additional reference images. Essentially, the term "one or more reference images" may be understood to refer to any set of existing digital images depicting cells.

[0007] The process by which the most similar reference images are identified will now be described in some more detail. Specifically, determining one or more digital training images depicting cells most similar to the cells depicted in the digital test image may include calculating a similarity metric value indicating the similarity between the digital test image and each of the plurality of digital reference images, and selecting a subset of the plurality of digital reference images whose similarity metric values ​​indicate the highest similarity with the digital test image, the selected subset being the one or more digital reference images depicting cells most similar to the cells depicted in the digital test image. In some cases, selecting a subset of the plurality of digital reference images may include ranking the plurality of digital reference images based on their similarity metric values, and selecting a predetermined number of the highest ranked digital reference images, where the selected predetermined number is the selected subset.

[0008] The similarity metric may be related to image data representing a digital training image or a digital reference image. Further details about similarity metrics are provided later in this application.

[0009] As used herein, the term "classifier model" refers to a machine learning model that is configured to take an image as its input and output a classification of an object, in this case, a cell, depicted in the image. Effectively training a machine learning model such as a classifier model often requires the use of thousands of training images. If the reference images are or include digital training images, after the digital test image has been classified, identifying one or more reference images that are most similar to the digital test image may require calculating similarity metric values ​​between the digital test image and potentially thousands of digital reference images, which may be computationally infeasible.

[0010] To avoid this, the digital training images may include a first subset of digital training images and a second subset of digital training images, with only the first subset of digital training images forming part of the plurality of digital reference images. In other words, selecting a subset of the plurality of reference images may include selecting a subset of the plurality of digital training images from only the first subset of digital training images. Establishing a similarity metric with only a subset of the digital training images can reduce computation time. The first subset may be specifically selected to represent a range of cell types to ensure that the one or more images displayed to the user contain cells of the same type as the cell type of the digital test image. Similarly, the reference image database may include a reduced number of reference images relative to the digital training images to reduce the computational power required to identify the most similar reference image(s). The reference image database may contain 10-100,000 images, 100-10,000 images, 500-5,000 reference images, 800-2,000 images, 900-1,500 images, or preferably about 1,000 images. As used herein, "about" may be interpreted to mean "within 10% of."

[0011] A classifier model may include various sub-models. For example, a classifier model may include an encoder sub-model (this term may be used interchangeably with simply "encoder" throughout this application) configured to generate a latent representation of received image data, represented by a plurality of latent variables. In the context of the present invention, the term "latent variables" refers to quantifiable characteristics of image data that are not necessarily immediately visible to humans. In other words, they can only be directly inferred through mathematical operations on the data and do not necessarily correspond to observable variables (although they may). The latent variables themselves may be in the form of vectors, and the latent representation may include multiple weights that can be applied to each of these latent variables to reconstruct the original digital image data. Inherent in the encoding process is that it is generally not possible to completely reconstruct the digital image data using the latent representation, but rather allows for an approximate reconstruction based on the most significant latent variables.

[0012] In those embodiments of the present invention that include an encoder-generated hidden representation of digital image data, a similarity metric may be calculated based on the latent representation of the received image data and the latent representation of the reference image data representing each of the digital reference images. More specifically, the similarity metric may be the distance in N-dimensional space between the latent representation of the digital image data and the latent representation of the image data representing each of the digital reference images. N may be the number of dimensions in the latent representation, i.e., the number of latent variables in the latent representation. Alternatively, only a subset of the latent variables may be used, in which case N may be any number less than the number of latent variables in the latent representation, selected as most appropriate. The difference may be the Euclidean distance (where the overall distance is calculated as the square root of the sum of the squares of the distances in each dimension, which generalizes to, for example, two-dimensional Pythagorean addition), the Manhattan distance (which is the sum of the lengths of the projections of the line segments between points onto the coordinate axes in an N-dimensional real vector space with a fixed Cartesian coordinate system (Equation 1)), or the cosine difference (the dot product of two vectors divided by the product of their lengths to give a cosine value indicating the similarity between the two vectors).

[0013]

number

[0014] The classifier model may further include a cell type classification sub-model configured to determine and output a cell type of a cell depicted in the digital test image based on the latent representation of the received image data generated by the encoder sub-model.

[0015] The classifier model may be an artificial neural network. In some cases, the encoder sub-model may include one or more convolutional layers. The cell type classification sub-model may include one or more fully connected layers. Thus, the classifier model may be in the form of or include a ResNet model, such as ResNet18, ResNet34, ResNet50, or ResNet101. It will be appreciated that other types of ResNet classifier models are available and suitable for this purpose.

[0016] The process by which a classifier model may be trained is outlined elsewhere in this application. As described, the classifier model may include an encoder and a classification sub-model. Training the classifier model may include both training the encoder and training the classification sub-model. Because the role of the encoder is solely to generate a representation of its training image data, represented by latent variables, training the encoder portion may rely solely on the training image data for each record. Training the classifier may include using spectral normalization to help regularize the classifier model, which may result in more refined clusters of semantically close data points.

[0017] The spectral normalization technique (https: / / doi.org / 10.48550 / arXiv.1802.05957), commonly used to train encoders as part of generative adversarial networks (GANs), is equally applicable to the present invention. It has the property that the Lipschitz constant is the only hyperparameter to be tuned. Amersfoot et al. (2021) document the feature collapse problem, where points collapse into lines, for unconstrained training with a simple classification objective (https: / / doi.org / 10.48550 / arXiv.2102.11409). The spectral norm of a matrix is ​​the largest singular value of a matrix M, i.e., M T is the largest square root of the eigenvalues ​​of M, and M Tdenotes the adjacency or transversal of the matrix M. This can be generalized to higher-order, i.e., tensor, operators. The use of spectral normalization keeps the weights within the bi-Lipschitz constraint, thereby avoiding feature collapse.

[0018] Training a classifier model may include retrieving or receiving training data, where each record of the training data includes image data representing a digital training image depicting a cell and a label indicating the cell type of the cell depicted in the digital training image. Then, in a first step, the training image data may be passed through a (currently untrained) classifier model to obtain a classification prediction for each image. Naturally, at this point, the prediction is unlikely to be correct because the model has not been trained. Therefore, in a next step, a loss function may be calculated for each digital training image using the label indicating the cell type of the cells depicted in that training image. Next, the gradient of the cost function (i.e., the average error across a batch of training data, which may include, for example, 32, 64, or 128 records of training data; in some cases, the average error may be calculated using images with a lower resolution than the training images) is calculated and used to update the weights across the classifier model. This process may then be repeated until a termination criterion is met, for example, until the model converges, or after a predetermined number of epochs. When spectral normalization is used, two additional steps may be performed before the process is repeated. First, the spectral norm (defined above) can be calculated for each layer of the classifier model. Then, the weights of each layer can be divided by the spectral norm. This ensures that the weights remain within the bi-Lipschitz constraint. After these steps are performed, the process can be repeated as usual.

[0019] The present invention is directed to the classification of cell types. Accordingly, a classifier model may be configured to classify a cell as one of a promyelocyte, a myelocyte, a metamyelocyte, a band neutrophil, a neutrophil, an eosinophil, a basophil, a monocyte, a lymphocyte, aberrant lymphocyte, atypical lymphocyte, a blast, and a nucleated red blood cell.

[0020] A further aspect of the present invention provides a diagnostic support system for identifying cell types in an image of a specimen, the diagnostic support system comprising a processor and a display component, the processor configured to perform the computer-implemented method of the first aspect of the invention, and the processor configured to send generated instructions to the display component.

[0021] A further aspect of the invention is a computer-implemented method for improving a trained classifier model configured to classify cells based on image data representing images depicting one or more cells, the method comprising: receiving training data including a plurality of digital records, each digital record including training image data representing digital training images depicting cells and labels indicating cell types of cells depicted in the digital training images; training the classifier model to classify cell types of cells depicted in the digital test images based on image data representing digital test images to generate a trained classifier model; receiving image data representing the digital test images, the digital test images depicting cells; and applying the trained classifier model to the received image data. a computer-implemented method for determining a cell type of a cell depicted in the digital test image, the computer-implemented method comprising: applying a classifier model configured to output a cell type of a cell depicted in the digital test image; determining one or more digital training images depicting cells that most closely resemble the cells depicted in the digital test image; generating instructions that, when executed by a display component of a computing device, cause the display component to display the cell type output by the classifier model, the determined one or more of the digital training images, and their respective cell types; receiving input from a clinician indicating whether the cell type output by the classifier model is correct; and updating the classifier model based on the one or more received clinician inputs, thereby improving or updating the classifier model. As used herein, "updating the classifier model" may include modifying or updating (e.g., adding to) training data based on the input from the clinician. The classifier model may then be retrained using the modified training data.A further aspect of the present invention may include a diagnostic support system for identifying cell types in an image of a specimen, the diagnostic support system comprising: a processor; and a display component, wherein the processor is configured to execute a computer-implemented method of improving a trained classifier model configured to classify cells, the trained classifier model being configured to transmit generated instructions to the display component.

[0022] Implementation of these aspects of the invention allows for the use of clinician input to improve the model.

[0023] a plurality of latent variables for each of the image data subsets; ... The computer-implemented method may further include generating instructions that, when executed by a display component of the computing device, cause the computing device to display an image corresponding to the identified subset of image data. An embodiment of this aspect of the invention allows a clinician to automatically identify cells similar to a selected cell when reviewing the specimen image. By displaying all cells similar to the selected cell, the clinician can make an evaluation or diagnosis based solely on the relevant information. The ergonomics of the diagnostic process are improved because the user can simply select all similar cells and then make a diagnosis based on those cells.

[0024] In some cases, in addition to providing a more ergonomic diagnostic process for clinicians, the computer-implemented method of this aspect of the present invention can be used to update the training dataset based on clinician input. Generally, this can be achieved by soliciting input from the clinician when the trained classifier model is unable to identify the cell type in question. The clinician can then be asked to identify the cell type, and then presented with images of cells similar to the cell type in question and asked to identify whether these cells should receive the same classification. More specifically, as described above, the trained classifier model can include an encoder and a cell type classification sub-model configured to determine and output the cell type of cells depicted in the digital test image based on the latent representation of the received image data generated by the encoder.The computer-implemented method then includes applying a cell type classification sub-model to a selected subset of the image data indicated in the first user input or to a subset of the image data representing the selected cells; determining whether the classification sub-model is capable of determining a cell type of the cells represented by the selected subset of the image data indicated in the first user input or to a subset of the image data representing the selected cells; and if the classification sub-model is unable to determine the cell type, generating instructions that, when executed by a display component of the computing device, cause the display component to request a second user input identifying the cell type; and, in response to receiving the second user input identifying the cell type, generating instructions that cause the display component to request a second user input identifying the cell type, wherein the similarity metric identifies the selected subset of image data or the selected cells. The computer-implemented method may further include retrieving stored data identifying one or more subsets of image data that show a highest similarity to the subset of image data representing the selected cell; and generating instructions that, when executed by a display component of the computing device, cause the display component to display images corresponding to the identified one or more subsets of image data whose similarity metric shows a highest similarity to the selected subset of image data or the subset of image data representing the selected cell, and for each image, a request for third user input to confirm whether the image should be classified as the same cell type indicated in the same classification received in the second user input; and updating the training data based on the plurality of received third user inputs. A further aspect of the present invention provides a diagnostic support system, comprising: a processor configured to execute the computer-implemented diagnostic support method; and a display component, wherein the processor is configured to send the generated instructions to the display component.

[0025] The classifier model is generally configured to output a classification and a score between 0 and 1 (or equivalent) representing the likelihood that the cell is of that type. For example, an output of 1 may indicate that the classifier model is confident in its classification, while an output of 0.5 may indicate only 50% confidence. In computer-implemented method embodiments of this aspect of the present invention, determining whether the classifier model is able to determine the cell type may be based on a threshold confidence, probability, or likelihood. Specifically, the classification sub-model may be configured to output a cell type and an auxiliary value indicating the probability, likelihood, or confidence that the output cell type is the correct cell type. Determining whether the classification sub-model is able to determine the cell type may include comparing the auxiliary value to a predetermined threshold, where if the auxiliary value is equal to or greater than the predetermined threshold, it is determined that the classification sub-model was able to determine the cell type. And, if the auxiliary value is less than the predetermined threshold, it is determined that the classification sub-model was not able to determine the cell type. Alternatively, the auxiliary value may correspond to an uncertainty score, where if the auxiliary value is less than the predetermined threshold, it is determined that the classification sub-model was able to determine the cell type. If the auxiliary value is equal to or greater than a predetermined threshold, it is determined that the classification sub-model was unable to determine the cell type.

[0026] The user input may include the selection of cells within the specimen image (e.g., using a computer mouse, etc.). The processor may generate instructions that, when executed by the display component, cause the display component to output a version of the specimen image in which a predetermined number of the most similar cells are highlighted or otherwise indicated. Alternatively, the most similar cells may be displayed in a separate window. A magnified version of the cells may be superimposed over their location within the specimen image, allowing the user to both view a larger version of the cells and record their location. In some cases (note that this applies to all aspects of the present invention), the determined most similar cell(s) may be displayed in a location close to the cell of interest (e.g., a cell indicated by user input or a cell depicted in a digital test image). For example, upon identification of similar cells, these cells may be displayed surrounding (e.g., consecutively) the cell of interest such that images of cells not determined to be the one or more most similar cells are farther from the cell of interest than the determined most similar cells. Diagnosis of various conditions is generally based on the classification of populations of cell types rather than individual cells. For example, leukemia is unlikely to be identified based on the classification of a single cell. This means that there will be multiple abnormal (or otherwise interesting) cells in a specimen image of a subject with a particular condition. By displaying cells that are most similar to the cell of interest (i.e., other abnormal cells) near the cell of interest and excluding dissimilar cells (i.e., cells that have not been determined to be the most similar cell or cells) from the similar region of the display component, clinicians can make a more ergonomic and, consequently, more accurate assessment.

[0027] The predetermined number may be, for example, a predetermined number N of cells having the highest value of the similarity metric. N may be adjustable based on user input. Alternatively, the predetermined number of cells may include all cells in the specimen image whose similarity metric exceeds a threshold. The threshold may be predetermined or adjustable based on user input. In these cases, the user input may be continuously adjustable using, for example, a tool on the graphical user interface, such as a slider. This allows the user flexibility in displaying various numbers of the most similar cells.

[0028] The present invention includes combinations of the described embodiments and preferred features except where such combinations are clearly unacceptable or explicitly avoided. [Brief explanation of the drawings]

[0029] Embodiments of the present invention will now be described with reference to the accompanying drawings.

[0030] [Figure 1] FIG. 1 is a schematic diagram of a system that may be configured to perform a computer-implemented method according to an aspect of the present disclosure. [Figure 2] FIG. 1 is a flow diagram illustrating a computer-implemented method according to one aspect of the present disclosure. [Figure 3] FIG. 1 is a flow diagram illustrating a process by which one or more similar reference images may be identified in some implementations of the computer-implemented method of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a classifier model that may be used in some embodiments of the computer-implemented method of the present disclosure. [Figure 5A] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 5B] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 5C] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 6A] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 6B] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 6C] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 7A] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 7B] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 7C] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 8A] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. [Figure 8B] 1 is a result obtained by a computer-implemented method according to a first aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] Aspects and embodiments of the present invention will now be described with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this text are incorporated herein by reference.

[0032] FIG. 1 illustrates a system 10 that can be used to implement various computer-implemented methods of the present invention. The system 10 includes a cell type classification system 100, an image capture device 200, and an output device 300. Alternatively, the cell type classification system 100 may be referred to as a diagnostic support system. These may all be separate components. In such cases, they may be connected via some type of network (not shown), via a wireless connection, a wired connection, or a mixture of the three. When the cell type classification system 100, the image capture device 200, and the display component 300 are connected via a network, the network may be a wireless network, such as a wireless internet connection, a Wi-Fi network, a cellular network, or any equivalent network. Alternatively, the network may be a wired network, such as a LAN, a wired internet connection, or a WLAN. Those skilled in the art will readily appreciate that other types of network connections are possible.

[0033] A cell type classification system 100 that can be used to implement the present invention will now be described in more detail. It should be noted that the cell type classification system 100 includes several “modules.” The cell type classification system 100 may be implemented as a whole in the form of custom hardware, or alternatively, the classification system 100 may be implemented in software, e.g., in the form of computer-readable code containing instructions that, when executed, cause a computer to perform various functions described herein. Similarly, the modules (described in more detail below) may also be implemented in the form of hardware modules within the processor 104, but may also be implemented in the form of software modules, each represented, e.g., by respective sections of computer code containing instructions that, when executed, cause a computer to perform a respective function associated with that module. In this sense, modules may be interpreted as “functional modules” that may be implemented in any computer-based manner so as to perform their associated functions. It should be noted, with due care, that the entire cell type classification system 100 may be implemented on a general-purpose computer, such as a desktop computer, laptop computer, smartphone, tablet, etc.

[0034] The cell type classification system 100 includes an image capture device interface module 102, a processor 104, a memory 106, and an output device interface module 108. As the names suggest, the purpose of the image capture device interface module 102 and the output device interface module 108 is to interface with the image capture device 200 and the output device 30, respectively. The image capture device interface module 102 and the output device interface module 108 may be implemented in any suitable form: a software module, a physical interface (such as a USB connection), or a network component configured to receive a signal containing data from the first image capture device 200 or the output device 300. The image capture device interface module 102 and the output device interface module 108 may be the same component.

[0035] The processor 104 includes various sub-modules, including a classification model 1040, a similarity determination module 1042, a training module 1044, and an output module 1046. It will be understood from the foregoing disclosure that the specific embodiment of the cell type classification system 100 shown in Figure 1 is merely an illustrative example, and that the processor 104 of the cell type classification system 100 need not include some or all of the modules, or may alternatively include any sub-combination of the modules. All sub-combinations are contemplated.

[0036] Memory 106 stores classifier model 1060, training data 1062, reference image data 1064, and buffer 1066. At a particular point in time, buffer 1066 may include test image data 10660 and latent representations 10662. It will be understood from the foregoing disclosure that the specific implementation of cell type classification system 100 shown in FIG. 1 is merely an illustrative example, and that memory 106 of cell type classification system 100 need not include some or all of the stored items shown, or may instead include any subcombination of the stored items. All subcombinations are contemplated.

[0037] We now turn to a description of the method itself, which may be performed by the system of Figure 1, specifically by the cell type classification system 100. Of course, the methods or computer-implemented methods of the present invention may be performed by hardware or software configured differently from the cell type classification system 100 of Figure 1, but we will refer to this example in the following disclosure.

[0038] 2 is a flowchart illustrating a computer-implemented method according to one aspect of the present disclosure. In a first step S200, image data is received, the image data representing a digital test image. The test image data may be (temporarily) stored in buffer 1066 of memory 106 as test image data 10660. Specifically, test image data 10660 may be received from image capture device 200 via image capture device interface module 102 of cell type classification system 100.

[0039] Then, in step S202, the classification module 1040 may retrieve the classifier model 1060 from the memory 106 and apply it to the temporarily stored test image data 10660. FIG. 4 shows a high-level schematic diagram of a classifier model that may be used in an embodiment of the computer-implemented method of the present invention. The classifier model 1060 generally includes five stages: an input 40, an encoder 42, a latent representation 44, a classifier 46, and an output 48. The input 40 may take the form of the test image data 10660. The encoder 42 preferably includes multiple layers, each including multiple nodes. The layers may be in the form of one or more convolutional layers. By applying one or more convolutional layers to the input in the encoder 42, a latent representation 44 of the test image data 10660 can be generated. The latent representation 44 is a representation of the test image data 10660 in terms of multiple latent variables. The latent variables do not necessarily correspond to variables that are discernible or observable by humans. In theory, it should be possible to recreate the test image data 10660 (or at least a good representation of it) using a combination of the latent variables in the latent representation 44. After the latent representation 44 is generated using the encoder 42, the classifier portion 46 of the classifier model 1066 is applied. The function of the classifier portion 46, as its name suggests, is to determine a cell type based on the latent representation 44. The cell type then forms the output 48. It should be emphasized that this is merely a high-level example of one type of classifier model 1060 that may be used in embodiments of the present invention. Other classifier models 1060, such as a Resnet model, may also be used.

[0040] Returning to FIG. 2 , in step S204, the most similar reference image in the reference image database 1064 is identified. This process may occur at any time after the latent representations 44 are generated through application of the classifier model 1060 in step S202. The cell type output 48 is not required to perform step S204; however, it may be performed later. Alternatively, step S204 may be performed in parallel with the layer stage of the classifier model 106. An example of a process by which the most similar image in the reference image database 1064 may be identified is shown in FIG. 3 and will now be described.

[0041] In step S300, a latent representation, e.g., 44, is generated. This may be stored as latent representation 10662 in buffer 1066 of memory 106. The generation in step S300 may be performed in the manner outlined above or using any other effective method. In step S302, similarity determination module 1042 may then determine the value of a similarity metric between test image data 10660 and reference image data 1064 stored in memory 106. The latent representation 10662 may be in the form of a plurality of values, each value representing a latent variable in the latent representation 10662. Thus, the latent representation 10662 may be in the form of a large, multidimensional array of data. A corresponding latent representation may be generated for each of the reference images, i.e., for each image represented in the reference image data 1064. In some cases, the latent representations, or more specifically, data representing these latent representations, may also be stored in the reference image data 1064 in association with the respective image data.

[0042] In step S302, a similarity metric may be calculated for each reference image based on the latent representation 10662 and the latent representation of each reference image. As described, the similarity metric may be in the form of Euclidean distance, Manhattan distance, or cosine distance, all of which are generalized distance metrics in multidimensional space. Next, in step S304, the reference images represented by the reference image data 1064 are ranked according to the value of the calculated similarity metric, and in step S306, the N most similar reference images are identified, where N is, for example, a predetermined value or adjustable by the clinician based on input. After the most similar reference images are identified, they are output in step S308.

[0043] 2, in step S206, output is generated, for example, by output module 1046. The output may include a cell type classification of the test image data 10660 and the N identified most similar reference images.

[0044] 5A-8B show examples of results that may be obtained using a computer-implemented method according to the present invention.

[0045] FIG. 5A shows a first example of a digital test image ("cell of interest") depicting a characteristic neutrophil. It also includes the predicted cell type, i.e., the output of the classification process, along with the probability that the cell is that type. In FIG. 5A, the cell is identified as a neutrophil with 100% confidence. The image also shows the expert cell classification (i.e., the classification determined by a human expert). FIG. 5B shows the set of eight cells from the training set (or equivalently, from the reference image set) that most closely match the cell of interest. From this, it can be observed that all of the cells are neutrophils that closely resemble the test image, thus demonstrating that the computer-implemented method of the present invention can effectively identify similar cells in a reference dataset. FIG. 5C shows the set of eight cells from the training set (or equivalently, from the reference image set) that least closely match the cell of interest. In this case, the most distant cells are all eosinophils, which are morphologically very different from neutrophils.

[0046] Figures 6A-6C demonstrate this. In Figure 6A, the cell is identified as a neutrophil with 100% confidence. The image also shows the expert cell classification (i.e., classification determined by a human expert). Figure 6B shows the set of eight cells from the training set (or equivalently, from the reference image set) that most closely match the cell of interest. From this, it can be observed that all of the cells are eosinophils that closely resemble the test image, thus demonstrating that the computer-implemented method of the present invention can effectively identify similar cells in a reference dataset. Figure 6C shows the set of eight cells from the training set (or equivalently, from the reference image set) that least closely match the cell of interest. In this case, the most distant cells are all neutrophils, which can be seen to be morphologically very different from eosinophils.

[0047] Neutrophils and eosinophils are relatively common in the white blood cell population. Basophils are rarer and therefore underrepresented in the training dataset. Figure 7A shows a digital test image of a basophil identified by the classifier model with 99% confidence. Figure 7B shows the set of eight cells from the training set (or equivalently, from the reference image set) that most closely match the cell of interest. From this, it is observed that seven of the cells are basophils (one is a damaged basophil that was incorrectly identified as a monocyte by the classifier) ​​and one is a monocyte. Figure 7C shows the least similar cells, all eosinophils. This is expected, as eosinophils are the most morphologically distinct from the other types of white blood cells.

[0048] Figures 8A and 8B show different interesting examples that demonstrate the effectiveness of the present invention as part of a diagnostic support tool. Figure 8A shows a cell of interest. One that was incorrectly classified by the classifier as a lymphocyte turns out to be a basophil. Meanwhile, Figure 8B shows the eight most similar cells from the reference dataset. Of these, six are basophils and two are lymphocytes. Thus, when implementing the present invention as part of a diagnostic support system, the clinician would be presented with a series of images of basophils, even though the classifier incorrectly identified the cells as lymphocytes. In this way, the present invention helps the clinician make a correct assessment.

[0049] The features disclosed in the foregoing description, or the following claims, or the accompanying drawings, and expressed in their specific form or as means for performing a disclosed function or as methods or processes for obtaining a disclosed result, may be utilized, individually or in any combination of such features, as appropriate, to realize the invention in diverse forms thereof.

[0050] While the present invention has been described in conjunction with the exemplary embodiments set forth above, many equivalent modifications and variations will be apparent to those skilled in the art given this disclosure. Accordingly, the exemplary embodiments of the present invention set forth above are considered to be illustrative and not limiting. Various changes may be made to the described embodiments without departing from the spirit and scope of the invention.

[0051] To avoid any misunderstanding, any theoretical explanations provided herein are provided for the purpose of enhancing the understanding of the reader, and the inventors do not wish to be bound by any of these theoretical explanations.

[0052] Any section headings used herein are for organizational purposes only and should not be construed as limiting the subject matter described.

[0053] Throughout this specification, including the claims which follow, unless the context requires otherwise, the words "comprises" and "includes," and variations such as "comprises," "comprising," and "including," will be understood to imply the inclusion of stated integers or steps or groups of integers or steps, but not the exclusion of any other integers or steps or groups of integers or steps.

[0054] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another embodiment. The term "about" with respect to numerical values ​​is arbitrary and means, for example, + / - 10%.

Claims

1. 1. A computer-implemented method for classifying cell types, comprising: receiving image data representing a digital test image, the digital test image depicting cells; applying a classifier model to the received image data, the classifier model configured to output a cell type of the cells depicted in the digital test image; The classifier model is an encoder sub-model comprising one or more convolutional layers, the encoder sub-model configured to generate a latent representation of the received image data represented by a plurality of latent variables, the encoder sub-model comprising one or more convolutional layers; and a cell type classification sub-model configured to determine and output a cell type of the cell depicted in the digital test image based on the latent representation of the received image data generated by the encoder sub-model, the cell type classification sub-model including one or more fully connected layers. Including, The classifier model is trained using training data that includes a plurality of digital records, each digital record comprising: training image data representing digital training images depicting cells; and Labels indicating the cell type of the cells depicted in the digital training images applying, determining one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image, wherein the reference images are pre-existing digital images depicting cells; generating instructions that, when executed by a display component of a computing device, cause the display component to: the cell type output by the classifier model; and the one or more digital reference images determined to depict cells that are most similar to the cells depicted in the digital test image. generating an instruction to display Including, 10. The computer-implemented method, wherein the classifier model is trained using spectral normalization.

2. Determining one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image includes: calculating a value of a similarity metric indicative of the similarity between the digital test image and each of a plurality of digital reference images; selecting a subset of the plurality of digital training images for which the value of the similarity metric indicates the highest similarity with the digital test image, the subset selected being the one or more digital reference images depicting cells that are most similar to the cells depicted in the digital test image; The computer-implemented method of claim 1 , comprising:

3. The computer-implemented method of claim 1 or claim 2, wherein the one or more reference images include the plurality of training images that form part of the training data.

4. the plurality of digital training images includes a first subset of digital training images and a second subset of digital training images; The computer-implemented method of claim 2 , wherein selecting a subset of the plurality of digital training images comprises selecting a subset of digital training images from only the first subset of digital training images.

5. The computer-implemented method of claim 1 , wherein the similarity metric is calculated based on the latent representation of the received image data and a latent representation of the reference image data representing each of the digital reference images.

6. 6. The computer-implemented method of claim 5, wherein for each digital reference image, the similarity metric is the distance in N-dimensional space between the latent representation of the received image data and a latent representation of that digital reference image, where N is the number of dimensions in the latent representation.

7. The computer-implemented method of claim 6 , wherein the distance in the N-dimensional space is a Euclidean distance, a Manhattan distance, or a cosine distance.

8. 8. The computer-implemented method of claim 1, wherein the classifier model is configured to classify the cells as one of promyelocytes, myelocytes, metamyelocytes, band neutrophils, neutrophils, eosinophils, basophils, monocytes, lymphocytes, abnormal lymphocytes, atypical lymphocytes, blasts, and nucleated red blood cells.

9. 1. A diagnostic support system for identifying cell types in a specimen image, comprising: a processor; Display components and Equipped with The processor is configured to perform the computer-implemented method of any one of claims 1 to 8, The processor is configured to transmit the generated instructions to the display component.

10. 1. A computer-implemented diagnostic support method, comprising: receiving image data representing an image of a specimen depicting a plurality of cells, the image data including a plurality of subsets of image data, each subset of image data representing a portion of the image of the specimen depicting a single cell of interest; applying an encoder of a trained classifier model to each of the subsets of image data, the encoder including one or more convolutional layers, the encoder being trained to generate a respective latent representation of each subset of image data, the latent representation being represented by a plurality of latent variables; receiving a first user input selecting a cell within the specimen image or a subset of image data representative of cells within the specimen image; determining, based on the respective latent representations of the subsets of image data, respective values ​​of similarity metrics indicative of the degree of similarity between the selected subset of image data or the subset of image data representing the selected cells and each of the other subsets of image data; identifying one or more subsets of image data where the similarity metric exhibits the highest similarity to the selected subset of image data or the subset of image data representing the selected cells; storing in memory data identifying one or more subsets of said image data wherein said similarity metric exhibits the highest similarity to said selected subset of image data or said subset of image data representing said selected cells; Including, 10. A computer-implemented diagnostic support method, wherein the classifier model is trained using spectral normalization.

11. the trained classifier model includes the encoder including one or more convolutional layers and a cell type classification sub-module including one or more fully connected layers, the cell type classification sub-module configured to determine a cell type output for the cells depicted in a digital test image based on a latent representation of the received image data generated by the encoder; The computer-implemented method comprises: applying the cell type classification sub-model to a selected subset of the image data or to a subset of the image data representing the selected cells indicated in the first user input; determining whether the classification sub-model is capable of determining the cell type of the cells represented by the selected subset of the image data indicated in the first user input or the subset of image data representing the selected cells; if the classification sub-model is unable to determine the cell type, generating instructions that, when executed by a display component of a computing device, cause the display component to request a second user input identifying the cell type; In response to receiving the second user input identifying the cell type, retrieving the stored data identifying one or more subsets of the image data where the similarity metric exhibits the highest similarity to the selected subset of image data or the subset of image data representing the selected cell; generating instructions that, when executed by a display component of a computing device, cause the display component to: images corresponding to the identified subset or subsets of image data, where the similarity metric indicates the highest similarity with the selected subset of image data or the subset of image data representing the selected cells; and a request for a third user input for each image to confirm whether the image should be classified as the same cell type indicated by the same classification received in the second user input; generating an instruction to display updating the training data based on the received third user inputs; The computer-implemented method of claim 10, comprising:

12. The computer-implemented method of claim 11 , further comprising retraining the classifier model based on the updated training data.

Images