Clinical support system and related computer-implemented method

Abstract

To provide a clinical support system and method for adapting and improving an output of a machine learning model to be used for automatic cell identification so as to collect meaningful information.SOLUTION: A method performs: the reception of image data of a plurality of cells acquired from an object of a human or an animal; the application of a deep learning neural network model to each subset of the image data, and the application of a trained deep learning neural network model to each subset of the image data; the derivation of a data set including three or fewer values about each subset of the image data, and the derivation of a data set including three or fewer values which are derived from activation values of a node in a bottleneck layer; and the generation of an instruction to make a display constitution element of a computer display plots of three dimensions or less of respective data sets of each subset of the image data when executed by a display constitution element of a clinical support system.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a clinical support system, a computer-implemented method for identifying cell types in an image containing a plurality of cells, and a computer-implemented method for training a deep learning neural network model. Other aspects of the invention are also contemplated. [Background technology]

[0002] Digital hematology is a field in which it is often necessary to be able to detect the presence of abnormal cells, for example in slide images. Morphologically, abnormal cells may differ from normal cells only in very subtle ways that require close inspection by an experienced clinician to identify. In recent years, the automatic identification of cells has become easier with the advent of advanced artificial intelligence-based techniques such as machine learning. Such techniques can be parameterized and "learn" hidden morphological or geometric features that do not necessarily correspond to easily visualized features.

[0003] It is useful for the clinician to have a complete picture of the types of cells present, as well as easily digestible information about the types of cells. For example, it may be useful for the clinician to understand some basic information about abnormal cells present, even if those cells cannot be classified using a model (e.g., because they are a rare cell type for which there is insufficient training data). Currently, classifiers are constructed such that the output of the classifier is in the form of classes, and in the case of abnormal cells for which there is no training data, they do not provide the clinician with any information that would enable them to make a diagnosis or other relevant decision.

[0004] Therefore, there is a need to adapt and refine the outputs of machine learning models used for automated cell identification to optimize them for human use, i.e., to allow meaningful information to be gleaned in ways not possible before. Such data would provide ergonomic improvements for clinicians. Summary of the Invention

[0005] Broadly speaking, the present invention provides a computer-implemented method that achieves these ergonomic improvements by extracting data from a bottleneck layer of a deep learning neural network model and presenting the extracted data in three dimensions or less. The introduction of a bottleneck layer into a deep learning neural network model causes the model to summarize essential aspects of the data into a small number of values. By generating output based on these values, clinicians can be presented with improved visualizations of the data that are better suited for human use.

[0006] More specifically, a first aspect of the present invention provides a clinical support system comprising a processor and a display component, the processor receiving image data representing an image of a plurality of cells obtained from a human or animal subject, the image data including a plurality of subsets of the image data, each subset including data representing a portion of the image data corresponding to a respective cell of the plurality of cells, and applying a trained deep learning neural network model to each subset of the image data, the deep learning neural network model including a plurality of convolutional neural network (CNN) layers, each including a plurality of nodes, and a bottleneck layer including 10 or less nodes, the processor applying the trained deep learning neural network model by applying the plurality of CNN layers followed by applying the bottleneck layer. and applying a trained deep learning neural network model to each subset of image data, where each node of a bottleneck layer of the machine learning model is configured to output a respective activation value for that subset of image data; deriving a dataset for each subset of image data including no more than three values, where the values ​​are derived from the activation values ​​of the nodes in the bottleneck layer; and generating instructions that, when executed by a display component of the clinical support system, cause a display component of the computer to display a plot of each dataset of each subset of image data in no more than three dimensions. The clinical support system may be in the form of, for example, a digital blood analyzer. The clinical support system may also be in the form of a laboratory information system or a hospital information system, more specifically a module (e.g., a physical hardware module, or a software module implemented in code) of the laboratory information system or the hospital information system.

[0007] The term "activation value" is used to represent the scalar output of a node in a neural network. It may be in the form of a positive or negative number. As a general rule, the activation value of each node in the bottleneck layer is a parameterization of the degree to which the cell in question displays a particular feature (such as a morphological or geometric feature). The features are usually hidden or latent features that do not necessarily correspond to easily identifiable features in reality. Providing a bottleneck layer effectively forces the deep learning neural network model to generate 10 or fewer hidden features and parameterize the morphology of the cell in terms of only those features. Similar cell types are likely to have similar or nearly similar activation values ​​in the bottleneck layer. This means that when a dataset is derived and displayed, populations of cells of the same or similar type will generally be clustered in the displayed plot. Clinicians generally know which location in the plot corresponds to which cell type. This means that clusters of abnormal cells that are not located in the expected location of the plot can be easily identified as abnormal cell populations in the sample. Furthermore, the location of the clusters in the plot also provides useful information to the clinician. For example, if a cluster falls between two expected clusters, this may indicate to the clinician that the unknown population contains cells with morphological characteristics that fall between the characteristics of the cells represented by the two expected clusters.

[0008] It will therefore be appreciated that by subsequently introducing a bottleneck into the deep learning neural network model and plotting a data set derived from a number of, preferably all, activation values ​​in the data set, it is possible to generate an output that is easily interpretable and therefore easily usable by the clinician. In particular, in contrast to the conventional output of a classifier, the output generated by the clinical support system of the first aspect of the present invention provides more information to the clinician and better enables the clinician to make clinical decisions, both in terms of making the population of abnormal cells easily identifiable and in terms of placing them in a position that shows some of their morphological characteristics compared to other cells. Thus, such a clinical support system is ergonomically improved compared to a clinical support system that uses only a simple classifier.

[0009] In a preferred case, the deep learning neural network model is a classifier model. The inventors have shown that the performance of a classifier model having 10 or fewer nodes is still sufficient to identify blood cell types or components of interest, such as neutrophils, basophils, lymphocytes, band neutrophils, blasts, debris, eosinophils, immature granulocytes, lymphocytes, monocytes, platelets, atypical lymphocytes, promyelocytes, myelocytes, metamyelocytes, and plasma cells. Thus, the machine learning model may be further configured to generate a classification output indicating the type of cells in the image corresponding to a subset of the image data based on the activation values ​​of the nodes in the bottleneck layer. Thus, the present invention allows a balance between ergonomic visualization of data on populations of cells and effective classification, greatly aiding the performance of the clinician's duties. This balance is particularly effectively achieved when the number of nodes in the bottleneck layer is either 5, 6, 7, 8, 9, or 10.

[0010] The bottleneck layer may include two or three nodes. In these cases, the processor may be configured to derive a data set including three or less values ​​from the nodes in the bottleneck layer by extracting activation values ​​for each of the two or three nodes in the bottleneck layer. If there are more than three nodes in the bottleneck layer (i.e., if there are 4 to 10 nodes in the bottleneck layer), the processor may be configured to derive a data set including three or less nodes by extracting activation values ​​from two or three nodes in the bottleneck layer. Alternatively, if there are more than three nodes in the bottleneck layer, the processor may be configured to derive a plurality of N data sets for each subset of data, each data set including three or less values, the values ​​of the given data set being derived from the activation values ​​of the respective subsets of nodes. The processor may then be configured to generate instructions that, when executed by a display component of the clinical support system, cause or enable the display of a plurality of plots, each of which is three or less dimensions. Each plot of the plurality of plots preferably corresponds to a data set derived from activation values ​​of the same node in the bottleneck layer. For example, if there are N nodes in the bottleneck tier, N may be equal to "N chooses 3" or "N chooses 2" or the sum of the two. (Here, "X chooses Y" is a way of selecting a subset of Y elements from a set of X elements, and may be defined as follows: (TIFF2024059600000002.tif1230) This means that multiple plots can be generated, each corresponding to the values ​​of different node combinations from the bottleneck layer. By outputting multiple plots in this way, a more useful output is provided to the clinician.

[0011] The term "bottleneck layer" is used to emphasize the fact that the number of nodes in the bottleneck layer is less than the number of nodes in the previous layer (i.e., the convolutional neural network layer). The trained deep learning neural network may be a residual network such as ResNet18, ResNet34, ResNet50, or ResNet101.

[0012] As already discussed, the main objective of the present invention is to provide a clinical support system that can better provide the user with an ergonomic output that allows clinical decisions and evaluations to be made. However, once the output is generated, the processor can be configured to identify one or more cell types present in the image of the plurality of cells based on clusters of points in the plot or in the plurality of data sets. For example, the processor can be configured to apply a clustering algorithm to the points in the plot, or to the plurality of data sets, to identify clusters of cells and classify the cells in each cluster. In some cases, abnormal cells may not appear in expected clusters. In those cases, the processor can be configured to detect the presence of abnormal cells in the image of the plurality of cells based on points that are not included in clusters associated with normal cells.

[0013] There are various other steps that may be performed by the processor after the generation and display of one or more plots. The processor may include an analysis module configured to perform various analysis functions based on the generated plot. Some examples are provided below, but it will be understood that generating and plotting the output of a deep learning neural network model as outlined here allows for more direct extraction of meaningful data in addition to a more ergonomic display of that data. The analysis module may be configured to perform quantitative analysis on the plot, or the data set that gives rise to the plot. For example, the analysis module may be configured to perform or apply a clustering algorithm to multiple points in the plot, or multiple data sets, to identify clusters of cells.

[0014] The analysis module may further be configured to perform an analysis on the clusters of cells. For example, the analysis module may be configured to determine the number of points in each of one or more clusters. Alternatively, the analysis module may be configured to determine the percentage of points that fall within each cluster. This may be used to identify conditions that result in an increased number or percentage of cells compared to a normal state, such as neutrophilia, erythrocytosis, or leukocytosis.

[0015] By tracking the movement of the clusters over time, diagnostic analysis, i.e., tracking the state or progression of a disease, may be performed. Thus, the processor may be configured to perform all of the steps required to generate the plots multiple times, and the analysis module may be configured to track the shifts in the clusters to generate shift data. The analysis module may be configured to determine a clinically meaningful output based on the shift data. The clinically meaningful output may include a score or other data indicative of the shift data, which may enable a clinician to make a relevant decision (e.g., a diagnosis). In some cases, the processor may be configured to generate instructions that, when executed by the display component, cause the display component to display multiple plots in chronological order, each plot forming a frame of a video or animation. In this way, the clinician may be able to more clearly visualize the shifts of the clusters, providing a more ergonomic view and enabling the clinician to make appropriate clinical decisions, for example, to determine the progression of a disease or to discern whether a treatment is working.

[0016] The analysis module may be configured to overlay the plot with one or more reference clusters (e.g., reflecting the 99th percentile of each respective normal cell type). As used herein, a "reference cluster" is used to refer to a region of space in a two-dimensional or three-dimensional plot that reflects a normal population of cells of a given type. This may be obtained, for example, by running a deep learning neural network model on a known normal cell population or on images known to be characteristic of cells of that type. By overlaying one or more reference clusters in this manner, the clinician is better able to detect deviations, thus improving the ergonomics of the output.

[0017] The processor, for example its analysis module, can be configured to augment the data set associated with a given cell with additional data generated for that cell using different analysis methods.For example, the processor (for example its analysis module) can be configured to augment the data set with segmentation data or CD (cluster of differentiation, a term well known in the art) marker analysis data.The data can include, for example, nuclear to cytoplasmic ratio.

[0018] We now consider ways in which a clinician or other user can advantageously interact with the generated plot, ways in which the ergonomics of the interface can be improved, and ways in which the clinician can acquire and process the data. In a computer-implemented method, a processor may be configured to generate instructions that, when executed by a display component of the clinical support system, cause a graphical user interface to display a plot (or multiple plots as described above, or an animation / video). The graphical user interface may be configured to receive user input, for example, in the form of a mouse click, a screen touch, or any other conventional input. The user input may include a selection of one or more data points in the displayed plot.

[0019] In response to a selection of one or more data points, the processor may be configured to generate instructions that, when executed by the display component, cause supplemental information regarding the one or more data points to be displayed. For example, the supplemental information may include an image of a cell represented by the data point. To do so, the processor may be configured to obtain a subset of image data corresponding to the data point in question, and to generate instructions that, when executed by the display component, cause an image to be rendered based on the subset of image data and the rendered image to be displayed.

[0020] In some cases, images of each cell in the population may be displayed in a gallery adjacent to the plot. Then, when one or more cells are selected, as outlined above, the processor may be configured to sort the images such that images corresponding to the selected data points are displayed first or in a more prominent position in the gallery view. In this manner, a clinician (or other user) may quickly obtain images of cells represented by data points that the clinician has identified as likely to be abnormal, for example because they are not located with clusters representing normal cells. This allows the clinician to quickly obtain data related to the cells of interest, based on which the clinician may be able to make a diagnosis or gather other clinically relevant information.

[0021] Where an image of a cell is displayed with the plot, the user input may include selection of the image, in response to which a data point corresponding to that cell may be highlighted in the plot.

[0022] The supplementary information may further include segmentation data, results of CD marker analysis, or other information about the cells, such as the nuclear to cytoplasmic ratio, as described above. This allows the clinician to quickly and directly access additional information about the data points, thus improving the ergonomics of the system to a greater extent.

[0023] A first aspect of the invention relates to a clinical support system. A second aspect of the invention relates to a corresponding computer-implemented method (which may be executed by a processor of the clinical support system of the first aspect of the invention). More specifically, the second aspect of the invention relates to a computer-implemented method of identifying cell types in an image of a plurality of cells, comprising: receiving image data, the image data representing an image of a plurality of cells acquired from a human or animal subject, the image data including a plurality of subsets of the image data, each subset including data representing a portion of the image data corresponding to a respective cell of the plurality of cells; and applying a trained deep learning neural network model to each subset of the image data, the deep learning neural network model including a plurality of convolutional neural network layers each including a plurality of nodes and a bottleneck layer including 10 or less nodes; applying the trained deep learning neural network model to the image data. a computer-implemented method comprising: applying a trained deep learning neural network model to each subset of image data, the training comprising applying a plurality of CNN layers to each subset of image data, followed by applying a bottleneck layer, where each node of the bottleneck layer of the machine learning model is configured to output a respective activation value for that subset of image data; deriving a dataset comprising no more than three values ​​from the nodes in the bottleneck layer for each subset of image data; and generating instructions, which when executed by a display component of the clinical support system, cause a display component of the computer to display a three-dimensional or less plot of the respective dataset for each subset of image data. It will be appreciated that any features described above with respect to the first aspect of the invention apply equally well to the second aspect of the invention. In particular, any features relating to operations performed (or configured to be performed) by a processor of the clinical support system of the first aspect of the invention may also apply to the second aspect of the invention.

[0024] Both the first and second aspects of the present invention rely on a trained deep learning neural network model that is applied to different subsets of image data. It will be appreciated that a deep learning neural network model is a machine learning model that must be trained before it can be used. A third aspect of the present invention provides a computer-implemented method for generating a deep learning neural network model that can be used in implementing the first and second aspects of the present invention. Specifically, a third aspect of the present invention provides a computer-implemented method for generating a trained deep learning neural network model for identifying cell types in an image of a plurality of cells, the computer-implemented method comprising: receiving training data including a plurality of sets of electronic data, each set of electronic data representing a respective image of a cell, each set of electronic data further including a label indicating a type of the cell depicted in the image; training a deep learning neural network model using the training data in a first training stage, the deep learning neural network model including an encoder portion including a plurality of convolution layers and an initial head portion configured to classify the image of the cell and output the cell type, thereby generating an intermediate deep learning neural network model; and replacing the initial head portion with a replacement head portion including a bottleneck layer including 10 or less nodes to generate an updated intermediate deep learning neural network model. The third aspect of the present invention may further comprise a second training stage of training the updated intermediate deep learning neural network model using the training data to generate a trained deep learning neural network model.

[0025] Here, the "encoder" portion of the deep learning neural network model is the portion or sub-network of the deep learning neural network model configured to learn a representation of input data in terms of hidden variables or features, and the "head" portion is the portion configured to perform a classification operation based on the values ​​of the hidden variables.

[0026] The training data may include a first subset of training data and a second subset of training data, each including a respective multiple set of electronic data as defined above. In these cases, the first training stage (on the untrained deep learning neural network model) may be performed using the first subset of training data and the second training stage (on the updated intermediate deep learning neural network model). Alternatively, the complete training data may be used to perform both the first and second training stages.

[0027] As mentioned above, the deep learning neural network model may include multiple convolutional layers. As such, the deep learning neural network model may be a convolutional neural network. The computer-implemented method of the third aspect of the present invention (or any aspect for that matter) is applicable to any kind of deep learning neural network model, in particular convolutional neural networks of any architecture. Examples of convolutional neural network architectures that may be used in the implementation of the present invention include residual neural networks (Resnet, e.g., Resnet34), ResNeXt neural networks, AlexNet neural networks, VGG neural networks, vision transformation neural networks, and squeeze-and-excite neural networks. This is by no means an exhaustive list, but is intended to convey the breadth of applicability of the computer-implemented method of the third aspect of the present invention. Preferably, the deep learning neural network model is a classifier model. The classifier model may be configured to classify cells or other structures in the image as one or more of neutrophils, band neutrophils, basophils, blasts, debris, eosinophils, immature granulocytes, lymphocytes, monocytes, platelets, atypical lymphocytes, promyelocytes, myelocytes, metamyelocytes, plasma cells. Specifically, the classifier model is preferably configured to classify cells or other structures in the image after a first training stage. Although the classification performance of the model inevitably decreases after the classification head is removed, it has been shown that the introduction of a bottleneck layer with 10 or fewer nodes can still provide adequate classification accuracy as well as ergonomically improved visualization of the output. In some cases, the bottleneck layer may have two or three nodes.

[0028] The bottleneck layer may be configured to output a data set that includes three or fewer values. In some cases, if the bottleneck layer includes two or three nodes, the bottleneck may be configured to output an activation value for each node. If there are more than three nodes, the bottleneck layer may be configured to select a subset of two or three nodes and output the activation values ​​for those nodes. See the "Overview" section for other examples of how this may be accomplished.

[0029] In some cases, the initial head portion may include one or more dense layers. In this context, a "dense layer" is a layer in which each node in the layer is configured or capable of receiving inputs from all nodes in the previous layer. In these cases, replacing the initial head portion with a replacement head portion may include removing one or more dense layers and adding an identity layer for each removed dense layer. In this context, an "identity layer" is a layer in which the output of a node is identical to the input of that node, i.e., does not make any changes to the input. For example, using an identity layer rather than simply removing one or more dense layers ensures that the representation space is not squeezed in an undesirable way.

[0030] Training the updated intermediate deep learning neural network model may include freezing the encoder portion of the deep learning neural network model and training the updated intermediate deep learning neural network model by changing only the parameters of the permutation head portion. In other words, the representation learned by the encoder portion remains, and only the last portion of the network is allowed to change the weights of the nodes. At this point, the classification performance of the network may improve, if only slightly. In these cases, the updated intermediate deep learning neural network model may be trained using a one-cycle policy (https: / / arxiv.org / abs / 1803.09820). During training of a machine learning model, it is important to select an appropriate learning rate (which determines the amount of gradients that are backpropagated). A small learning rate means that the model will converge slowly, while a large learning rate may cause the model to diverge. Traditionally, the learning rate is decreased as the model approaches convergence. However, when a one-cycle policy is used, the learning rate is increased for a first period, and the learning rate is decreased for a second period, preferably of the same length as the first period. Doing so prevents overfitting, avoids steep regions of the loss, and helps the model reach a flatter minimum.

[0031] Training the updated intermediate deep learning neural network model may further include unfreezing (i.e., allowing its values ​​to continue to change) the encoder portion of the updated intermediate deep learning neural network model, and training the entire updated intermediate deep learning neural network model using the training data. The training data may include a third subset of the training data used for this final training step. Again, classification performance may improve, if only slightly.

[0032] A fourth aspect of the present invention provides a clinical support system according to the first aspect of the present invention, wherein the deep learning neural network model is generated using the computer-implemented method of the third aspect of the present invention. A fifth aspect of the present invention provides a computer-implemented method according to the second aspect of the present invention, wherein the deep learning neural network model is generated using the third aspect of the present invention. It will be readily appreciated that any features described with respect to the third aspect of the present invention apply equally well to the fourth and fifth aspects of the present invention.

[0033] A further aspect of the present invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the computer-implemented method of the second, third or fifth aspect of the invention. Another aspect of the present invention provides a computer readable storage medium having stored thereon a computer program of the preceding aspect of the invention.

[0034] The present invention includes combinations of the described embodiments and preferred features unless such combinations are expressly not permitted or explicitly avoided. [Brief description of the drawings]

[0035] Embodiments of the present invention will now be described with reference to the accompanying drawings.

[0036] [Figure 1] This is an example of a clinical support system such as a digital blood analyzer. [Diagram 2] 1 is a flowchart illustrating a computer-implemented method for generating a trained deep learning neural network model. [Diagram 3] FIG. 1 is a schematic diagram of a trained deep learning neural network model. [Figure 4] 1 is a flow chart illustrating a computer-implemented method for identifying cells using a deep learning neural network model. [Diagram 5]4 is an exemplary output that may be generated in accordance with the present invention. [Figure 6] 4 is an exemplary output that may be generated in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Aspects and embodiments of the present invention will now be described with reference to the accompanying drawings. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this specification are incorporated herein by reference.

[0038] FIG. 1 is a schematic diagram of a clinical support system 1 according to, for example, a first embodiment of the present invention. The clinical support system 1 comprises a processor 12, a memory 14, and a display component 16. In FIG. 1, these components are all shown as being part of the same system, but it will be understood that the system may be a distributed system in which various components are located on different hardware, optionally in different locations. In these cases, each component (e.g., the processor 12, the memory 14, and the display component 16) may be connected via a network, not shown. The network may be a wired network, such as a LAN or a WAN, or a wireless network, such as a Wi-Fi network, the Internet, or a cellular network. Now, with reference to FIG. 2 and FIG. 4, the structure of the clinical support system 1 will be described before discussing operations that the clinical support system 1 is configured to perform. The processor 12 includes a number of modules. In this specification, the term "module" is used to refer to a functional module configured or adapted to perform a specific function. The modules may be implemented in hardware (i.e., they may be separate physical components in a computer), software (i.e., they may represent separate code sections that, when executed by the processor 12, cause the processor to perform a particular function), or a combination of both. Specifically, the processor 12 of FIG. 1 comprises a training module 120, a cell identification module 122, an analysis module 124, a data set extraction module 126, and a data visualization module 128. The functionality of each of these modules will be described in more detail shortly. The memory 14 may be in the form of a permanent memory or a temporary memory, or may include a combination of these. The memory 14 stores the deep learning neural network model 140 and the training data 124. The training data may include three subsets 1420, 1422, 1424. The subsets are preferably non-overlapping, i.e., they each include different data records.The display component 16 is preferably in the form of a VPU, screen, or monitor configured to visually render the data to a clinician for viewing results.

[0039] FIG. 2 illustrates a process performed by the training module 120 of the processor 12 of the clinical support system 1, in which a trained deep learning neural network model 140 is generated. In a first step S200, training data 142 is received, for example from a client device or some repository. The purpose of the deep learning neural network model 140 is to characterize and identify cells. The training data 142 thus comprises a plurality of sets of electronic data, each set of electronic data representing a respective image of a cell, each set of electronic data further comprising a label indicating the type of cell depicted in the image. The training data 142 thus configured allows the deep learning neural network model 140 to learn the correspondence between image data and cell types. It is worth mentioning that at this stage of the computer-implemented method, the deep learning neural network model 140 is a complete deep learning neural network model, such as a classifier neural network, nothing has been removed yet.

[0040] In step S202, the deep learning neural network model 140 is trained using the training data 142. This is the first training phase, which may be performed using a first subset 1420 of the training data 142. In this phase, the entire deep learning neural network model 140 is trained using the first subset 1420 of the training data 142, as described above. The first training phase, i.e., step S202, may continue for a predetermined number of epochs, for example 10. While training the classifier neural network 140 in this manner, the objective is generally to minimize the classification loss. However, other loss functions, such as reconstruction loss, may be used.

[0041] After step S202 is completed, in step S204, the classification head of the deep learning neural network model 140 is removed, and only the convolutional part of the deep learning neural network model 140 is retained. The convolutional part of the deep learning neural network model 140 is the part that learns the representation of the input data in terms of hidden variables or features. The classification head is the part of the network that identifies the cell type based on the hidden representation. In step S206, a bottleneck layer containing 10 or less nodes (or neurons) is added to the end of the convolutional part of the network to replace the classification head. The number of original output classes and the softmax layer are retained. The bottleneck layer causes the network to summarize the essential aspects of the data into 10 or less values. In contrast to an autoencoder, it is not necessary to be able to reconstruct the original image in a self-supervised manner. Rather, it is desirable to retain as much classification performance as possible. In step S208, a selected rectified linear unit ("ReLu") layer in the classification head is replaced by an identity layer to avoid squeezing the representation space in an undesirable way. Then, in step S210, the encoder portion (i.e., the convolutional portion) of the network is frozen, meaning that during further training phases, the weights associated with the nodes of the convolutional or encoder portion are fixed and do not change.

[0042] In step S212, a section training phase is performed. During this phase, the currently deep learning neural network model 140 is trained using a subset 1422 of the training data 142. Performance improvements may result from this additional training step S212. After this step is completed, in step S214, the trained deep learning neural network model 140 is stored in memory 14.

[0043] 3 is a highly schematic diagram of a deep learning neural network model 140 that may be generated in accordance with the present invention. The deep learning neural network model 140 includes a first plurality of convolution layers. Then there is a bottleneck layer, which in the illustrated example has only two nodes and is configured to output the activation values ​​of those nodes for display. Optionally, the deep learning neural network model 140 may then further include a fully connected layer that can provide a classification output. This is not required for implementations where the only desired output is visualization, but is advantageous when both visualization and classification are desired.

[0044] FIG. 4 is a flow chart showing steps of a computer-implemented method for identifying cell types in an image of a plurality of cells, corresponding to the computer-implemented method of the first aspect of the present invention. In a first step S400, input image data is received in the processor 12 of the clinical support system 1, for example from a client device. The input image data may be in the form of an image of a plurality of cells or a cell population. Thus, the image may include a plurality of individual smaller images, each showing an individual cell. Each of these smaller images is represented by a subset of the image data representing the entire image. In step S402, the cell identification module 122 is preferably configured to identify individual cells in the image and to identify a corresponding subset of the image data (herein "subset of data") representing each of these image portions. The cell identification module 122 may perform this operation using any of several conventional techniques for identifying individual cells in an image of a cell population, for example as described in WO 2014 / 164757, the entire contents of which are incorporated herein by reference.

[0045] After the cells are identified as outlined above, in step S404, the trained deep learning neural network model 140 is applied to each subset of data. After application of the convolutional layers of the deep learning neural network model 140, the bottleneck layer is reached. At this point, the cell's characteristics are summarized into 10 or less scalar values, each value quantifying some hidden variables. In step S406, three or less values ​​are extracted from the bottleneck layer by the dataset extraction layer 126. For example, as shown in FIG. 3, the bottleneck layer may include two or three nodes. In these cases, the dataset extraction module 126 may be configured to derive a dataset including three or less values ​​from the nodes in the bottleneck layer by extracting activation values ​​of each of the two or three nodes in the bottleneck layer. In cases where there are more than three nodes in the bottleneck layer (i.e., when there are 4 to 10 nodes in the bottleneck layer), the dataset extraction module 126 may be configured to derive a dataset including three or less nodes by extracting activation values ​​from two or three nodes in the bottleneck layer.

[0046] The values ​​extracted from the bottleneck layer in step S406 are the values ​​that are ultimately plotted in 2D or 3D. Thus, the data visualization module 128 is configured to generate instructions based on the values ​​extracted from the bottleneck layer for each of the input subsets of data that, when executed by the display component 16, cause the display component to display a 2D or 3D plot including a plurality of data points, each data point corresponding to the extracted data set. Finally, in step S410, these instructions are output to the display component 16 for execution and display.

[0047] Figures 5 and 6 show plots that can be generated using the computer-implemented method of the present invention. Figure 5 is a 3D plot showing that by extracting only three values ​​from the bottleneck layer, it is possible to generate a plot in which each different type of cell (or other component of blood) is associated with a distinct, dense cluster with a relatively small number of outliers. In this case, the clusters correspond to basophils, blast cells, debris, eosinophils, lymphocytes, monocytes, neutrophils, damaged cells, clumps (referring to clumps of platelets stuck together), cells that are out of focus in the image (desired to differentiate), giant platelets, "multiples" (multiple cells shown in a single image), and nucleated red blood cells.

[0048] Figure 6 illustrates the utility of the present invention for identifying abnormal or aberrant cell populations within a sample. Six clusters of cells are clearly visible: 1. Blast cells 2. Eosinophils 3. Lymphocytes 4. Monocytes 5. Basophils 6. Neutrophil

[0049] However, between clusters 3 and 4 there are swaths of data points that do not fit into any cluster. Since such cells are outside of well-defined clusters, it is clear to the clinician that such cells are abnormal. Furthermore, because the data points are extracted from the bottleneck of the classifier model, their location in the plot can provide some clinically relevant information, namely that the abnormal cells are morphologically somewhere between lymphocytes and monocytes. Thus, the present invention allows for a visualization of the output of the classifier model that is more ergonomically readable for clinicians such as hematologists.

[0050] The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, and expressed in a specific form or in terms of means for performing a disclosed function or a method or process for obtaining a disclosed result, may be utilized, individually or in any combination of such features, as appropriate, to realize the invention in its diverse forms.

[0051] While the present invention has been described in conjunction with the exemplary embodiments set forth above, many equivalent modifications and variations will be apparent to those skilled in the art given this disclosure. Accordingly, the exemplary embodiments of the present invention set forth above are considered to be illustrative and not limiting. Various modifications to the described embodiments may be made without departing from the scope of the present invention.

[0052] To avoid any misunderstanding, any theoretical explanations provided herein are provided for the purpose of improving the understanding of the reader, and the inventors do not wish to be bound by any of these theoretical explanations.

[0053] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0054] Throughout this specification, including the claims which follow, unless the context requires otherwise, the terms "comprise" and "include," as well as variations such as "comprises," "comprising," and "including," are understood to imply the inclusion of stated integers or steps or groups of integers or steps but not the exclusion of any other integers or steps or groups of integers or steps.

[0055] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another embodiment. The term "about" with respect to numerical values ​​is arbitrary and may mean, for example, + / - 10%.

Claims

1. A clinical support system comprising a processor and a display component, The aforementioned processor Receiving image data, wherein the image data represents images of multiple cells obtained from a human or animal subject, and the image data comprises multiple subsets of the image data, each subset comprising data representing a portion of the image data corresponding to each of the multiple cells, Applying a trained deep learning neural network model to each subset of the image data, wherein the deep learning neural network model comprises a plurality of convolutional neural network (CNN) layers, each containing a plurality of nodes, and a bottleneck layer containing 10 or fewer nodes, and the processor is configured to apply the trained deep learning neural network model to each subset of the image data by applying the plurality of CNN layers and then the bottleneck layer, and each node of the bottleneck layer of the deep learning neural network model is configured to output its respective activation value for the subset of the image data. For each subset of the aforementioned image data, a dataset containing three or fewer values ​​is derived, wherein the values ​​are derived from the activation values ​​of the nodes in the bottleneck layer, and the dataset contains three or fewer values. When executed by the display component of the clinical support system, the command is generated to the display component of the clinical support system to display a plot of three dimensions or less for each dataset of each subset of the image data. A clinical support system configured to perform the following actions.

2. The clinical support system according to claim 1, wherein the trained deep learning neural network model includes at least an encoder portion of a deep learning neural network model configured to classify images of cells, and the encoder portion includes the plurality of CNN layers.

3. The clinical support system according to claim 2, wherein the deep learning neural network model is configured to classify cells as one or more of neutrophils, basophils, lymphocytes, band neutrophils, blasts, fragments, eosinophils, immature granulocytes, monocytes, platelets, atypical lymphocytes, promyelocytes, myelocytes, metamyelocytes, and plasma cells.

4. The clinical support system according to claim 1, wherein the bottleneck layer includes two or three nodes.

5. The clinical support system according to claim 1, further comprising identifying one or more cell types present in the images of the plurality of cells based on clusters of points in the plot or the plurality of datasets.

6. The clinical support system according to claim 5, further comprising detecting the presence of one or more abnormal cells in images of the plurality of cells based on points that are not included in clusters associated with normal cells.

7. A computer implementation method for generating a trained deep learning neural network model for identifying cell types, Receiving training data comprising multiple sets of electronic data, wherein each set of electronic data represents an image of a cell, and each set of electronic data further comprises a label indicating the type of cell shown in the image; Using the aforementioned training data, a deep learning neural network model is trained that includes an encoder portion containing multiple convolutional layers and an initial head portion configured to output cell types, classify cell images, and output cell types again, thereby generating an intermediate deep learning neural network model. The initial head portion is replaced with a replacement head portion that includes a bottleneck layer containing three or fewer nodes, thereby generating an updated intermediate deep learning neural network model. Computer implementation methods, including those mentioned above.

8. The computer implementation method according to claim 7, wherein the bottleneck layer includes two or three nodes.

9. The computer implementation method according to claim 7, wherein the deep learning neural network model is a classifier model.

10. The computer implementation method according to claim 9, wherein the classifier model is configured to classify cells as one or more of neutrophils, basophils, lymphocytes, band neutrophils, blasts, fragments, eosinophils, immature granulocytes, monocytes, platelets, atypical lymphocytes, promyelocytes, myelocytes, metamyelocytes, and plasma cells.

11. The computer implementation method according to claim 7, further comprising training the updated intermediate deep learning neural network model using the training data to generate the trained deep learning neural network model.

12. Training the aforementioned updated intermediate deep learning neural network model Freezing the encoder portion, The updated intermediate deep learning neural network model is trained by changing only the parameters of the replacement head portion. The computer implementation method according to claim 11, including the method described in claim 11.

13. Training the aforementioned updated intermediate deep learning neural network model Decompressing the encoder portion of the updated intermediate deep learning neural network model, The updated intermediate deep learning neural network model is trained using the aforementioned training data. The computer implementation method according to claim 12, further comprising:

14. A computer implementation method for identifying cell types in images of multiple cells, Receiving image data, wherein the image data represents images of multiple cells obtained from a human or animal subject, and the image data comprises multiple subsets of the image data, each subset comprising data representing a portion of the image data corresponding to each of the multiple cells, Applying a trained deep learning neural network model to each subset of the image data, wherein the deep learning neural network model comprises a plurality of convolutional neural network (CNN) layers, each containing a plurality of nodes, and a bottleneck layer containing 10 or fewer nodes, and applying the trained deep learning neural network model to each subset of the image data is Applying the aforementioned multiple CNN layers, Next, the bottleneck layer is applied, wherein each node of the bottleneck layer of the deep learning neural network model is configured to output its respective activation value for the subset of the image data. Applying a trained deep learning neural network model, including the above, to each subset of the image data, For each subset of the image data, the following are performed: deriving a dataset containing three or fewer values ​​from the nodes in the bottleneck layer; and, when executed by the display component of the clinical support system, generating a command to the display component of the clinical support system to display a plot of three dimensions or less for each dataset of each subset of the image data. Computer implementation methods, including those mentioned above.

15. The clinical support system according to any one of claims 1 to 6, wherein the trained deep learning neural network is generated using the computer implementation method according to any one of claims 7 to 13.

16. The computer implementation method according to claim 14, wherein the trained deep learning neural network is generated using the computer implementation method according to any one of claims 7 to 13.

Images