Apparatus and method

Abstract

A method of training a machine learning algorithm, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and training the machine learning algorithm comprising incremental learning using the set of sample data.

Description

[0001] APPARATUS AND METHOD

[0002] Field

[0003] The present invention relates to analytical instrument data analysis.

[0004] Background to the invention

[0005] In the United Kingdom, there are currently around 327,800 people diagnosed with a blood cancer related malignancy. Blood cancer has a five-year survival rate of 90% and those who live with blood cancer require regular testing. Rates of blood cancer are rapidly increasing, outpacing rates of other cancer. Furthermore, COVID-19 has led to a 10-year backlog on diagnosis of cancers. As a result, blood cancer survival rates in the UK are set to decrease for the first time ever, speeding up diagnosis times to reduce the backlog would help prevent this.

[0006] Flow cytometry is commonly used for diagnosing blood cancer. Collecting flow cytometry data is a straightforward process. The gold standard method for analysis of FC data is manual gating. However, the data analysis is manual, time consuming and requires a highly skilled operator, generating a substantial bottleneck in the diagnostic pipeline. Automated flow cytometry data analysis which is faster would result in more cost-effective diagnosis of diseases and allow patients to receive the treatment they need sooner, the single most important factor for improving patient outcomes.

[0007] Attempts to automate flow cytometry data analysis have been ongoing for over 20 years, however automated methods have not been widely adopted in clinical practice. Most methods have proved too inaccurate, such as those integrated in the current state-of-the-art software solutions such as FlowJo and Cytobank. Other methods only work under specific conditions which are inappropriate for all but a minority of users, such as I nfinicyt - Euroflow Profiles and Hema.to. Users report that the protocols required are too prescriptive, requiring specific expensive equipment or analysing data in a different way to what is desired.

[0008] Hence, there is a need to improve flow cytometry data analysis. More generally, there is a need to improve analytical instrument data analysis that includes manual gating.

[0009] Summary of the Invention

[0010] A first aspect provides a method of training a machine learning algorithm, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and training the machine learning algorithm comprising incremental learning using the set of sample data.

[0011] A second aspect provides a method of inferring classifications using a machine learning classification algorithm trained according to any previous clam, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a dataset, wherein the dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and inferring respective classifications for at least some of the plurality of single cell and / or particle events of the first sample data using the trained machine learning classification algorithm.

[0012] A third aspect provides a computer comprising a processor and a memory configured to implement a method according to any of the first aspect and / or the second aspect, a computer program comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of the first aspect and / or the second aspect or a non-transient computer-readable storage medium comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of the first aspect and / or the second aspect.

[0013] Detailed Description of the Invention

[0014] According to the present invention there is provided a method of training, as set forth in the appended claims. Also provided is a method of inferring. Otherfeatures ofthe invention will be apparent from the dependent claims, and the description that follows.

[0015] Method of inferring: The first aspect provides a method of training a machine learning algorithm, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and training the machine learning algorithm comprising incremental learning using the set of sample data.

[0016] In this way, the machine learning algorithm is trained by, for example partly or fully, incremental learning using the set of sample data, including the first sample data for the corresponding first sample, wherein the first sample data comprises the plurality of single cell and / or particle events and respective parameters, such as acquired using flow cytometry. Particularly, the inventors have determined that incremental learning of machine learning algorithms using such sample data, comprising a plurality of single cell and / or particle events in respective parameters, is effective, providing a trained machine learning algorithm for rapid, automated analytical instrument data analysis having a relatively higher accuracy of inference and / or trained using relatively fewer sample data, compared with machine learning algorithms that are trained conventionally. Since the analytical instrument data analysis is rapid and automated, a highly skilled operator is not required, thereby eliminating the substantial bottleneck of manual data analysis while providing diagnosis of disease is relatively sooner. Since the accuracy of inference is relatively higher, for example comparable or better the manual data analysis such as manual gating, the trained machine learning algorithm provides effective diagnosis of disease, without requiring a highly skilled operator. Since relatively fewer sample data are required, the machine learning algorithm maybe trained specifically for the particular protocol used for acquisition of the analytical instrument data, thereby enabling implementation of the analytical instrument data analysis specifically for the particular protocol, for example on a per laboratory basis while the effort required for training, such as required sample data and / or skilled operator during training, is reduced or minimised. In this way, cost-effective diagnosis of disease is provided, allowing patients to receive treatment sooner, thereby improving patient outcomes.

[0017] Machine learning algorithm: The method is of training the machine learning algorithm. Training of machine learning algorithms is known generally. In one example, the machine learning algorithm comprises and / or is an incremental machine learning algorithm, for example a clustering machine learning algorithm or a classification machine learning algorithm that inherently supports incremental learning and / or adapted to support incremental learning, as described in more detail below. In one example, the machine learning algorithm comprises and / or is a classification machine learning algorithm or an ensemble thereof, for supervised learning using labelled sample data, tree-based machine learning algorithm. Suitable classification machine learning algorithms are known, for example Hoeffding Trees (HT) are tree-based classification algorithms that learn incrementally while adaptive Random Forest (ARF) are Random Forest (RF) algorithms which are made up of an ensemble of incremental learning tree-based classifiers, such as HTs. In one example, the machine learning algorithm comprises and / or is a clustering machine learning algorithm or an ensemble thereof, for unsupervised learning using unlabelled sample data. Labelling of the determined clusters may be provided subsequently, for example by a skilled operator via the computer. Suitable clustering machine learning algorithms are known.

[0018] Computer: The method is implemented by the computer comprising the processor and the memory. Suitable computers are known. In one example, the computer comprises and / or is a remote computer, such as provided on or by the cloud. In one example, the computer comprises and / or is a network, for example comprising a remote computer, such as provided on or by the cloud, and a local computer, for example provided in the laboratory for the analytical instrument.

[0019] Obtaining: The method comprises obtaining the training dataset, for example from storage, as understood by the skilled person.

[0020] Training dataset: The training dataset comprises the set of sample data, including the first sample data, for the corresponding set of samples including first sample. It should be understood that the first sample data corresponds with the first sample i.e. the first sample data is acquired for the first sample, for example using the analytical instrument according to a specific protocol, which may include sample preparation, acquisition and / or sample data preprocessing. It should be understood that the set of sample data are acquired using the same analytical instrument or type of analytical instrument and according to the specific protocol. In one example, the set of sample data comprises replicates, for example replicates for the first sample, whereby multiple sample data correspond with the first sample. In one example, the set of sample data comprises calibration sample data corresponding with one or more calibration samples, as understood by the skilled person. In one example, the set of sample data does not comprise calibration sample data.

[0021] In one example, the set of sample data includes M sample data, including the first sample data, and the corresponding set of samples includes N samples, including the first sample, wherein M and N are (for example each and / or separately) natural numbers greater than or equal to 1 , for example 1 , 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000 or more. In one example, M and / or N are at most 100, preferably at most 50, more preferably at most 25, most preferably at most 15 or 10. In this way, the machine learning algorithm is trained using relatively fewer sample data, thereby enabling implementation of the analytical instrument data analysis specifically for the particular protocol, for example on a per laboratory basis while the effort required for training, such as required sample data and / or skilled operator during training, is reduced or minimised.

[0022] In one example, the set of sample data includes M sample data, including the first sample data, and wherein the corresponding set of samples includes N samples, including the first sample, wherein M and N are natural numbers greater than or equal to 1 , for example 1 , 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000 or more; and wherein incremental learning using the set of sample data comprises incremental learning using the M sample data, for example iteratively. In this way, the machine learning algorithm is trained using incremental learning using the first sample data (i.e. of the first sample).

[0023] Sample data: The first sample data comprises the plurality of single cell and / or particle events and respective parameters.

[0024] By way of example, flow cytometry (FC) is an analytical instrument technique used to detect and measure physical and chemical characteristics of a population of cells or particles. A sample containing cells or particles is suspended in a fluid and injected into the flow cytometer (i.e an analytical instrument). The sample is focused to ideally flow one cell or particle at a time through a laser beam, where the light scattered is characteristic of the cells or particles and their components. Respective parameters include cell size, morphology, and protein expression. Cells or particles may be labelled with fluorescent markers so light is absorbed and then emitted in a band of wavelengths. Tens of thousands of cells may be quickly analysed and the sample data acquired are processed by a computer.

[0025] In one example, the set of sample data comprises flow cytometry data, mass cytometry data, scRNA-seq data, citeseq data, imaging flow cytometry and / or microscopy data. More generally, in one example, the set of sample data comprises data conventionally manually analysed using gating.

[0026] In one example, the set of samples comprises biological samples, for example animal samples such as blood, bone marrow, tissue and / or bodily fluid, or plant samples such as algae. In one example, the set of samples comprises samples Including calibration beads, for example calibration standards.

[0027] Incremental learning: The method comprises training the machine learning algorithm comprising incremental learning using the set of sample data.

[0028] Generally, incremental learning is a method of machine learning in which input data are continuously and / or successively and / or iteratively used to further train the machine learning algorithm. Incremental learning is or represents a dynamic technique of supervised learning and unsupervised learning that may be applied when training data becomes available over time or the size thereof is out of system memory limits. Machine learning algorithms that may facilitate incremental learning are known as incremental machine learning algorithms.

[0029] Many traditional machine learning algorithms inherently support incremental learning. Other algorithms can be adapted to facilitate incremental learning. Examples of incremental algorithms include decision trees (IDE4, ID5R and gaenari), decision rules, artificial neural networks (RBF networks, Learn++, Fuzzy ARTMAP, TopoART and IGNG) or incremental SVM.

[0030] The aim of incremental learning is for the learning machine algorithm to adapt to new data without forgetting its existing knowledge. Some incremental machine learning algorithms have built-in some parameter or assumption that controls the relevancy of old data, while others, called stable incremental machine learning algorithms, learn representations of the training data that are not even partially forgotten over time. Fuzzy ART and TopoART are two examples for this second approach.

[0031] In one example, incremental learning using the set of sample data comprises incremental learning using the first sample data, for example initially using the first sample data (i.e. wherein the machine learning algorithm first uses the first sample data set forthe incremental learning) or only using the first sample data (i.e. wherein the machine learning algorithm only uses the first sample data set for the incremental learning). In this way, the machine learning algorithm is trained using incremental learning using the first sample data (i.e. of the first sample).

[0032] Batch learning: In one example, training the machine learning algorithm comprises batch learning using a first subset of the set of sample data, for example before incremental learning using a second subset of the set of sample data. In this way, the machine learning algorithm may be pre-trained by batch learning and subsequently, further trained using incremental learning for example using sample data acquired according to the specific protocol.

[0033] In one example, training the machine learning algorithm does not comprise (i.e. excludes) batch learning. In one example, training the machine learning algorithm consists of incremental learning.

[0034] Labelled and unlabelled data: In one example, the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating, for example via the computer such as by a skilled operator, the first sample data. In this way, the machine learning algorithm may be trained, for example initially, using this labelled sample data (i.e. supervised learning for example wherein the machine learning algorithm comprises and / or is a classification algorithm).

[0035] In one example, the first sample data comprises and / or is unlabelled sample data. In this way, the machine learning algorithm may be trained, for example initially, using batch learning and subsequently trained using this unlabelled sample data (i.e. unsupervised learning for example wherein the machine learning algorithm comprises and / or is a clustering algorithm, wherein the clusters are subsequently labelled, as described previously).

[0036] Curating: In one example, the first sample data comprises and / or is unlabelled sample data; and wherein training the machine learning comprises: inferring respective classifications for at least some of the plurality of single cell and / or particle events of the first sample data using the machine learning algorithm, whereby the first sample data comprises and / or is labelled sample data, having inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data; curating, via the computer, the inferred respective classifications provided forthe at least some of the plurality of single cell and / or particle events of the first sample data; and wherein incremental learning using the set of sample data comprises incremental learning using the curated inferred respective classifications provided forthe at least some of the plurality of single cell and / or particle events of the first sample data.

[0037] In this way, the respective classifications are inferred by the machine learning algorithm, for example wherein the machine learning algorithm is partially trained, the inferred respective classifications are curated, for example verified or reclassified, and the machine learning algorithm further trained using the curated inferred respective classifications. In this way, an accuracy of inferring by the machine learning algorithm may be assessed during training while training of the machine learning algorithm is improved using the curated classification since accurate classifications are verified while inaccurate classifications are reclassified, thereby reinforcing learning or correcting learning respectively.

[0038] In one example, curating, via the computer, the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data comprises reclassifying, via the computer such as by a skilled operator, at least one of the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data. In this way, inaccurate classifications are reclassified.

[0039] In one example, curating, via the computer, the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data comprises verifying, via the computer such as by a skilled operator, at least one of the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data. In this way, accurate classifications are verified i.e. determined to be accurate.

[0040] Iterative incremental learning: In one example, the set of sample data includes second sample data and wherein the corresponding set of samples includes a second sample, wherein the second sample data comprises a plurality of single cell and / or particle events and respective parameters; wherein the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating the first sample data; and wherein the second sample data comprises and / or is unlabelled sample data; and wherein training the machine learning algorithm comprises: incremental learning using the first sample data; inferring respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data using the machine learning classification algorithm trained using the first sample data, whereby the second sample data comprises and / or is labelled sample data, having inferred respective classifications provided for at least some of the plurality of single cell and / or particle events of the second sample data; curating, via the computer, the inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data; and incremental learning using the curated inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data.

[0041] In this way, the machine learning algorithm is initially trained using the labelled first sample data, the initially trained machine algorithm is used to infer classifications of the unlabelled second sample data, these inferred classifications are curated and the initially trained machine algorithm is further trained using the curated inferred classifications, thereby improving training of the machine learning algorithm. It should be understood that the steps of inferring, curating and further training may be repeated (i.e. iterated) for further samples, as described below.

[0042] In one example, the set of sample data includes M sample data, including the first sample data, and wherein the corresponding set of samples includes N samples, including the first sample, wherein M and N are natural numbers greater than or equal to 2, for example 2, 3, 4, 5, 6, 7 ,8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 or more; wherein the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating the first sample data; and wherein the second to M sample data comprises and / or is unlabelled sample data; and wherein training the machine learning algorithm comprises: incremental learning using the first sample data; and iteratively incremental learning using the second to M sample data comprising: inferring respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data using the machine learning algorithm trained using the first sample data, whereby the second to M sample data comprises and / or is labelled sample data, having inferred respective classifications provided for at least some of the plurality of single cell and / or particle events of the second sample data; curating, via the computer, the inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data; and incremental learning using the curated inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data.

[0043] In one example, iteratively incremental learning using the second to M sample data comprises iteratively incremental learning using the second to M sample data until a predetermined threshold accuracy of inferring the respective classifications is achieved. In this way, the machine learning algorithm may be trained to achieve a desired accuracy i.e. the predetermined threshold accuracy.

[0044] Acquiring: In one example, the method comprises acquiring the set of sample data, for example using an analytical instrument such as a flow cytometer, a mass cytometry, scRNA-seq data, citeseq data, a imaging flow cytometer and / or microscope, for example according to a particular protocol.

[0045] It should be understood that the steps of acquiring, inferring, curating and further training may be repeated (i.e. iterated) for further samples, as described above mutatis mutandis. That is, the machine learning algorithm may be further trained after the sample data for each sample are acquired, for example.

[0046] Method of inferring: The second aspect provides a method of inferring classifications using a machine learning classification algorithm trained according to any previous clam, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a dataset, wherein the dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and inferring respective classifications for at least some of the plurality of single cell and / or particle events of the first sample data using the trained machine learning classification algorithm.

[0047] Computer, computer program, non-transient computer-readable storage medium: A third aspect provides a computer comprising a processor and a memory configured to implement a method according to any of the first aspect and / or the second aspect, a computer program comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of the first aspect and / or the second aspect or a nontransient computer-readable storage medium comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of the first aspect and / or the second aspect.

[0048] Definitions

[0049] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processor circuits. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0050] Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” may mean including the component(s) specified but is not intended to exclude the presence of other components.

[0051] The optional features set out herein may be used either individually or in combination with each other where appropriate and particularly in the combinations as set out in the accompanying claims. The optional features for each aspect or exemplary embodiment of the invention, as set out herein are also applicable to all other aspects or exemplary embodiments of the invention, where appropriate. In other words, the skilled person reading this specification should consider the optional features for each aspect or exemplary embodiment of the invention as interchangeable and combinable between different aspects and exemplary embodiments.

[0052] Although a few example embodiments have been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims.

[0053] Brief description of the drawings For a better understanding of the invention, and to show how exemplary embodiments of the same may be brought into effect, reference will be made, by way of example only, to the accompanying diagrammatic Figures, in which:

[0054] Figure 1 : Data instances were loaded in from one sample at a time. The data from each sample was first used to test the model, then used to train the model. This was repeated 20 times, with a different order of the shuffled samples. Both ARF and HT models were trained in this manner.

[0055] Figure 2: Median (IQR) F1 score of converged models of ARF and HT, following incremental training and evaluation on data generated from 152 FC samples. Mann-Whitney U test indicated that ARF showed significantly higher F1 scores than HT (U = 277,127, n1 = 860, n2 = 1 ,380, p < 0.001).

[0056] Figure 3: Rolling mean (window = 15) of median (IQR) F1 score of ARF (green) and HT (blue) models, over incremental training and evaluation on data generated from 152 FC samples. Vertical line indicates the point at which model convergence was calculated. The darker region indicates overlapping IQR.

[0057] Figure 4: Histogram of F1 score calculated after every evaluation of ARF (top) and HT (bottom) models following incremental training and evaluation on data generated from 152 FC samples.

[0058] Figure 5: F1 score of 20 HT (left) and ARF (right) models, over incremental training and evaluation on data generated from 152 FC samples. Each line indicates an individual model.

[0059] Figure 6: Rolling mean (window = 15) of median (IQR) training time per sample of ARF (green) and HT (blue) models, over incremental training and evaluation on data generated from 152 FC samples. Vertical line indicates the point at which model convergence was calculated.

[0060] Figure 7: Median (IQR) training time for each sample on converged models of ARF and HT, following incremental training and evaluation on data generated from 152 FC samples.

[0061] Figure 8: Rolling mean (window = 15) of median (IQR) prediction time per sample of ARF (green) and HT (blue) models, over incremental training and evaluation on data generated from 152 FC samples. Vertical line indicates the point at which model convergence was calculated.

[0062] Figure 9: Median (IQR) prediction time of each sample on converged models of ARF and HT, following incremental training and evaluation on data generated from 152 FC samples.

[0063] Figure 10: Boxplots denoting F1 (1st row), precision (2nd row) and recall (3rd row) for each class following evaluation of a converged HT (left) and ARF (right) models. Incremental training and evaluation were performed on data generated from 152 FC samples. Centre line denotes the median value. Each box contains the 25th to 75th percentiles of data. Whiskers denote the 5th and 95th percentile. Datapoints outside whiskers are plotted as individual points.

[0064] Figure 11 : Aggregated confusion matrices (mean ± SD) of all converged models for HT (left) and ARF (right). Colour indicates proportion of predicted labels normalised by number of true labels per class. Following incremental training and evaluation on data generated from 152 FC samples.

[0065] Figure 12: Rolling mean (window = 15) of median F1 score for each class on HT (left) and ARF (right) models, over incremental training and evaluation on data generated from 152 FC samples. Vertical line indicates the point at which model convergence was calculated. Figure 13: UMAP embedding generated using the descriptive statistics of the 152 FC files making up the dataset. Each point represents a sample. Each subplot represents a class and the colour denotes the proportion of instances in a sample which belonged to that class.

[0066] Figure 14: UMAP embedding generated using the descriptive statistics of the 152 FC files making up the dataset. Each point represents a sample. Each subplot represents median F1 (1st row), precision (2nd row) and recall (3rd row) calculated from converged HT (left) and ARF (right) model evaluations.

[0067] Figure 15: Linear correlations between the median F1 score calculated from each converged HT and ARF model evaluation and the proportion of instances in a sample which belong to a class. Solid line denotes the linear regression. Points denote each model evaluation and proportion of class in that sample. Shaded region denotes the 95% confidence interval estimated using 1 ,000 bootstrapping resamples. Horizontal dashed line denotes median class proportion of all samples. Each subplot also details Pearson’s coefficient, r, and p-value.

[0068] Figure 16: F1 score of each of the 152 samples for every evaluation on converged ARF and HT models, ordered from left to right by descending median F1 score. Each box represents a sample in the dataset. Centre line denotes the median F1 score the model evaluated for that sample. Each box contains the 25th to 75th percentiles of data. Whiskers denote the 5th and 95th percentile. Datapoints outside whiskers are plotted as individual points.

[0069] Figure 17: Labels of the 2 worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. The bottom middle plot shows the ambiguously labelled dense B cell cluster.

[0070] Figure 18: Labels of the 5 worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. The bottom middle plot shows the ambiguously labelled dense B cell cluster.

[0071] Figure 19: Labels of the 4 worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. The bottom middle plot shows the ambiguously labelled dense B cell cluster.

[0072] Figure 20: Labels of the 3 worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. There were no clear labelling errors in the authors opinion.

[0073] Figure 21 : Labels of the 6 worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. There were no clear labelling errors in the authors opinion.

[0074] Figure 22: Labels of the worst performing sample over all evaluations of ARF and HT models. Each plot shows the gating strategy for the Lymphoid Screening Tube panel, each colour indicates a different label given to each event. Initially, lymphocytes are identified from all events (top left and right plot), before being gated into B, T and NK cell sub-types. There were no clear labelling errors in the author’s opinion.

[0075] Figure 23: A screenshot of the prototype CytoLearn software, showing an fcs file loaded in containing data generated by the LST panel. Two-dimensional scatter and density plots have been generated and gating has been performed using a variety of tools to identify populations of interest. Statistics calculated from the identified populations can be displayed in a table which can also be downloaded as a csv file.

[0076] Figure 24: Model performance for benchmarked methods and incremental learning methods. Bars represent mean macro F1 score ± standard deviation. ARF is referred to as “CytoLearn (RF)”, HT is referred to as “CytoLearn (DT)”.

[0077] Figure 25: Prediction time for benchmarked methods and incremental learning methods. Bars represent mean time taken to make a prediction for each sample in seconds ± standard deviation. ARF is referred to as “CytoLearn (RF)”, HT is referred to as “CytoLearn (DT)”.

[0078] Figure 26: A flowchart of a method of training a machine learning algorithm for improving flow cytometry analysis.

[0079] Figure 27A: A table of results showing the median (IQR) F1 score for each class of converged classifier models.

[0080] Figure 27B: A table of results showing the median (IQR) precision for each class of converged classifier models.

[0081] Figure 27C: A table of results showing the median (IQR) recall for each class of converged classifier models.

[0082] Detailed Description of the Drawings

[0083] Summary: This study introduces incremental learning (IL) as a generalised solution to automated FC analysis, for example, as well as CytoLearn prototype software for users to trial the method. IL classifiers can be trained using batches of data to continuously improve a model’s knowledge. Hoeffding Trees (HT) are tree-based classification algorithms that learn incrementally. Adaptive Random Forest (ARF) are Random Forest (RF) algorithms which are made up of an ensemble of IL tree-based classifiers, in this instance HTs. This study evaluated the effectiveness of HT and ARF for the automated analysis of FC data. A trained operator labelled events of 152 FC samples with one of nine cell types. Twenty HT and ARF models were trained, each on a random order of the 152 samples. Each sample was first used to evaluate model performance, before being used for training. ARF showed higher median F1 scores than HT, though training took 7x longer and prediction 3x longer, on average. ARF models averaged F1 scores of 0.87 after being trained on only 15 samples, and with furthertraining, performance was similar to the batch learning (BL) models utilised in the previous study. IL algorithms show excellent potential as a generalised solution for automated FC analysis of known populations, offering the flexibility of unsupervised clustering methods, being able to analyse data generated from any antibody panel, combined with the accuracy of supervised classification methods.

[0084] Introduction: Previous study by the inventors demonstrated how tree-based classification algorithms show the potential to provide a solution to automated FC analysis of known populations. These algorithms showed high agreement with a manual operator’s labelling of FC events, short prediction times and CART and RF showed short training times. However, it did not present a generalised solution to FC data beyond the specific dataset used.

[0085] The models which were trained during the previous study are only able to make predictions on the same input features and predict the same event labels which they were trained on, therefore only FC data from the LST, which the dataset was derived from, could be used to predict the events which were labelled. While this may be valuable to users of the LST panel who aim to identify the same events which were used in the dataset, this is of little utility to other FC users. The LST panel is used specifically for the identification of lymphocytes, though FC has a wide variety of uses, including, but not limited to vaccine development, detecting microorganisms, biomarker detection and drug discovery, where each specific usage requires a different antibody cocktail depending on the events to be identified. Gating is time consuming, and the previous study relied upon 152 gated fcs files, followed by pre-processing steps to organise the data. Collecting labelled datasets to train classification models for every antibody panel used for FC is unfeasible, and even if it were feasible, there is variation between laboratories of how data from the same panel will be analysed, thus trained models would not even be applicable for every lab using that panel. It is therefore unsurprising that previous attempts at automated FC analysis have focused on unsupervised clustering algorithms rather than supervised classification methods. The advantages and disadvantages of both have been discussed in more detail in previous study, but a key advantage of unsupervised methods over supervised methods is that, as no training is required, any input data, and therefore any FC panel, can be used with these algorithms. However, based on the results of the previous study, supervised classification algorithms are more effective at automated FC analysis than unsupervised clustering algorithms and therefore could be more likely to provide a solution to automated FC analysis of known populations.

[0086] The approaches utilised in the previous study were that of batch learning (BL), where all training instances are made available to the model during model training. If any new training data were to be introduced to the model, the model would be required to retrain on the entire dataset to adapt to the newly introduced training data. Incremental learning (IL) is a dynamic form of supervised learning where training data becomes increasingly available over time, allowing for continuous model learning and prediction improvement. IL is largely motivated by two challenges. Firstly, machine learning models are frequently reliant on large amounts of data which may present a computational challenge to store and process as a batch, acting as a barrier to entry for those who wish to train machine learning models. IL has reduced memory requirements compared to BL, as the entire dataset is not required to be loaded into memory during training as it is with BL. Instead, when training IL models, once the data has been used for training it can be discarded, therefore reducing the computational requirements for machine learning. The second key motivator behind IL is the analysis of data streams. A data stream is the continuous generation or collection of a sequence of data, rather than data arriving in a single batch. Some examples of data streams include spam e-mail classification and presenting social media users with content which maximises click-through rates. In both examples, new data is being gathered constantly which a machine learning model makes predictions on and then learns from. Data streams are considered “non-stationary” if the underlying distribution of the data changes over time, or “stationary” if it does not. Non-stationary data streams can suffer from “concept drift”, where the data distribution changes to such an extent over time that early training data becomes out-of-date. Detection and mitigation of concept drift is required to avoid model performance becoming degraded by out-of- date knowledge informing predictions. In the biomedical field there seem to be very limited examples of ILs application to date. One example demonstrates the utility of IL for the automated recognition of pulmonary disease from chest X-rays. The published IL system had the distinct advantage over an analogous BL system that it could learn to predict new diseases as they were introduced to the model, without them having to be present in the original training dataset. The second example demonstrates how IL can be used to detect atrial fibrillation from electrocardiogram (ECG). The advantage over the previously published system was that the time and cost associated with labelling and preparing training data was reduced by 90%, as the labelled data could be collected directly from clinicians as part of the diagnostic pathway and the model could be trained on that data in situ. This would not be possible with BL as data collection is required prior to any model training, whereas IL models can continue to learn after being deployed.

[0087] The previous study demonstrated the effectiveness of CART and RF for automated FC analysis due to excellent accuracy, and prediction and training times. While these are BL algorithms which cannot be utilised for IL, there are IL equivalents. A Hoeffding Tree (HT) is an IL decision tree algorithm which is capable of learning from stationary data streams i.e., data streams where the distribution of data does not change over time. CARTs are generated by first choosing a root node using the best mathematical split, usually using the gini index, on a single feature of the entire training dataset, then iteratively defining internal nodes and finally leaf nodes using the same method with the data available at the node. HTs are generated in a similar manner. However, as the trees do not have access to the entire dataset during training, the goal becomes to ensure that the chosen split, based on the training data available, is that which would be chosen if the entire training dataset was available. Therefore, the challenge is defining how many training instances are required to be seen by a root node, and later leaf nodes, before the nodes are replaced by a split which is believed to be optimal. Given a variable of numerous observations, the Hoeffding bound defines the number of observations which require sampling before the mean of those observations does not differ from the mean of another sample of randomly chosen observations from the same variable. HTs use the Hoeffding bound to quantify the confidence required to be met before a root node, and later leaf node, is replaced with a more optimal split. Given a variable r, with the range R, with n independent observations made, with their computed mean f. The Hoeffding bound states with a probability 1 - 5, that the true mean of the variable will not differ from f by more than e, where: ) / ?2ln (1 / 5) J 2n

[0088] When using gini impurity as the split criterion, R = 1.0. For a split to be made, the difference between the best performing split and the second-best performing split must be > e. In practice, this means that the more training instances a leaf node sees, the more likely an optimal split is to be made. As well as the lower 8 is set, then more training instances are required before a split is made, though that split has a greater guarantee of being optimal (1 - 5). The Hoeffding bound possesses other desirable qualities. Assuming the data stream is indeed stationary, HTs offer a guarantee that the resultant tree is almost identical in performance to that of a non-incremental CART when infinite training instances are available. As HT are the IL analogue to CART, Adaptive Random Forest (ARF) are the IL analogue to RF. As is the case with batch learning RFs, ARFs are made up of an ensemble of HTs, with each tree varied due to having been limited to searching over a random subset of features on resampled training data when generating decision rules, with the output being the class voted by most trees.

[0089] This study introduces the concept of manual gating FC software which offers the functionality of currently used FC software, with the novel addition of an IL classification algorithm. The IL algorithm considers the FC data being analysed as a data stream, from which it can learn the label an operator is giving each event. The trained model is then able to make predictions on unseen FC data as it is loaded into the software for analysis, enabling automated FC analysis. Theoretically, IL can offer the flexibility of unsupervised learning, as a new model can be trained for differing input requirements (antibody panels) and output requirements (event labels) in situ, combined with the superior accuracy of supervised models, as demonstrated in the previous study. While all tree-based methods showed high agreement with manual operators and fast prediction times in the previous study, XGB models were slow to train, making them a less favourable solution for deployment in this context. Therefore, this study evaluates the use of HT and ARF for automated FC analysis of known populations and introduces CytoLearn, a generalised software solution for the task.

[0090] Methods

[0091] Dataset: The dataset was that used in the previous study where it has been more fully described. In brief, it consisted of LST panel FC data collected from 152 peripheral blood samples collected from patients with either an abnormal full blood count result or suspected haematological malignancies for whom FC had been performed as part of the diagnostic pathway. Each event in a sample had been labelled by a single FC operator, the author. The events the operator was confident of had been given one of nine labels: “Not lymphocyte”, “Kappa B cell”, “Lambda B cell”, “CD4+ T cell”, “CD8+ T cell”, “CD4+ / CD8+ T cell”, “CD4- / CD8- T cell”, “Gamma delta T cell”, “CD56+ NK cell”. Labelled events from each sample were combined and fluorescence channels were transformed using the Logicle transformation, while FSC and SSC remained linear. The dataset is heavily imbalanced with the majority class “Not lymphocyte” making up 60.99% of the dataset and the minority class “CD4+ / CD8+ T cell” (dual positive T cell) making up only 0.03% of it. Model Training and Testing: A list containing the name of each sample which was used to generate the dataset was shuffled and the result recorded. This was repeated 20 times. Each shuffled list dictated the order which data from each sample was used to first test, and then train, a HT and an ARF model. Testing was performed on every sample afterthe first one, which was only used fortraining. For each model, each sample was only used for training and testing once, therefore each model was trained and tested on 152 samples (Figure 1). This approach simulated how manual analysis software is used by FC operators, analysing a single sample one at a time and how an IL algorithm could learn the labels given by the operator, before making a prediction on the next sample. This resulted in 20 individual HT and ARF models being trained and evaluated, one for each shuffled order of sample names.

[0092] Default parameters were used for o, in the case of HT this was set at 1 e-7 and 0.01 for ARF. Therefore, the ARF makes splits faster, and can therefore make predictions faster than the HT. Though this comes at the cost of each split being less likely to be optimal, this is offset by the ARF being more forgiving if non-optimal splits are made, compared to HT, due to the increased number of trees making predictions, thus averaging out any errors. This follows the same principle on which batch learning RFs are built, being forced to make non-optimal splits for a dataset by being trained on resampled data and only being able to make splits on a random subset of features. This is the mechanism by which RFs are less likely to overfit data than CART.

[0093] ARF offers the capability to train trees in the background if concept drift is detected and use these trees to replace current trees. However, the data stream in the study was assumed to be stationary, as there seems no reason why it would not be, given all data is from the same FC panel. Therefore, this feature was not enabled.

[0094] ARF models were trained with ten trees in the ensemble. This is the default number suggested by River, the ARF implementation used. In contrast, the sklearn implementation of RF uses 100 trees by default. Sklearn allows each tree in an ensemble to be trained in parallel. River does not currently offer this functionality, therefore trees must be trained asynchronously, considerably increasing training and prediction times. Fewer trees are used by default to offset this.

[0095] Gini impurity was chosen as the split criterion by which nodes were chosen as it performs similarly to information gain in practice, but is faster to compute. Gini impurity is calculated as the likelihood of misclassifying a new instance. It is calculated for every combination of splits on each feature and the split with the lowest Gini impurity, i.e., lowest levels of misclassifications, is used to choose the node. The probability of misclassifying a new instance is defined as: where ptis the probability of classifying an instance as each class.

[0096] Evaluation Metrics: The approach of using incoming data from a stream, firstly for testing, then secondly fortraining, is known as “prequential evaluation”. “Prequential”, being derived from “predictive sequential”, refers to the prediction of events in a sequential manner, first by predicting the next event, or set of events, and once that has occurred, then by predicting the subsequent event, or set of events. Prequential evaluation is the most commonly used method for evaluating IL models. By using prequential evaluation, performance metrics can be calculated at any time during the models training based on the predictions the model has made to date, allowing for the evaluation of performance over time. In the study, models were trained, and performance was evaluated, on the data from one sample at a time, i.e., several thousand instances rather than a single instance. Therefore, it was decided performance would be evaluated per sample, rather than using typical prequential evaluation and considering all predictions made to date. The advantage of this was that it offered an understanding of model performance for each sample, without that information being lost as it would be if evaluations were averaged over the entire data stream. The evaluation metrics calculated for each sample were those used in previous study: precision- the proportion of instances of a class which are identified correctly, recall- the proportion of actual class instances which are identified correctly, and F1 score- the harmonic mean of precision and recall. As in the previous study, macro averaged F1 score was calculated (the mean F1 score across all classes present in the sample), as it is the preferred metric when classes are imbalanced. Training and prediction time in seconds was also recorded. Measurements for training time were made using a cluster with a pool of 40 CPUs and 185GB RAM, though the evaluation of each model was a single process. The pool was managed using the multiprocessing library in Python. All analyses were performed in Python (version 3.9). The River package (version 0.12.1) was used for ARF and HT.

[0097] Following model evaluation, F1 score was calculated for all 20 ARF and HT models after training on each sample. Rolling mean calculated over the number of samples models were trained on was calculated for median F1 scores of the 20 models, for both ARF and HT. A window of 15 samples was used to calculate the rolling mean, i.e., the mean of the median F1 scores for the previous 15 samples was used. A rolling mean is used to emphasise long term trends in data by smoothing shortterm variations. This was used to understand the overall trend of model performance as a function of the number of samples trained on. It was also used to indicate the point at which models converged, the point at which model performance no longer continued to improve despite additional training. This was not true model convergence as it was not calculated for each model, but instead calculated using the median of all models of the same algorithm. However, it allowed quantification of how many samples were required to be trained on before model performance no longer improved. This also allowed the evaluation of optimally trained models only, without considering evaluation instances where the model is still learning.

[0098] Convergence was defined as the first evaluation point at which the rolling mean F1 score was equal or greater than the rolling mean F1 score at the final evaluation point. Larger windows have the effect of greater smoothing by averaging more data, which may make long term trends easierto identify, though it considers earlier model performance therefore offering a more pessimistic view of the point of convergence. A window of 15 was deemed a suitable compromise.

[0099] Statistical Analyses of Incremental Learning Algorithm Performance: A Mann-Whitney U test was used to compare the mean F1 score of converged ARF and HT models. Statistical significance was set as p < 0.05. Data in all figures, tables and text in the study are presented as median (interquartile range [IQR]) unless otherwise specified. Evaluating Model Performance on Individual Samples: Uniform Manifold Approximation Projection (UMAP) is a non-linear dimensionality reduction technique which projects high-dimensional data into lower dimensional space. One advantage of doing so is allowing high-dimensional data to be plotted in two-dimensions while preserving local and global structure i.e., data points which are more similar are plotted closer together. UMAP has received much attention in the field of FC and wider field of single- cell data. This is partly due to high dimensionality of this data, but UMAP has received particular interest due to its improved ability to preserve the structure of non-linear data when compared to other commonly used dimensionality reduction techniques such as Principal Component Analysis (PCA) and t-distributed Stochastic Neighbourhood Embedding (t-SNE). FC data is generally not linearly dependent as each antibody selected for a panel is done so to acquire unique information from the sample due to the cost associated with antibody redundancy. In this study UMAP was used, not directly on event data, but on descriptive statistics of each sample. For each sample in the dataset, the mean, median, standard deviation (SD) and IQR of each feature in the FC data used for model training i.e. marker expression and scatter, was calculated. This generated descriptive statistics of the sample, an approach which has previously been used to classify samples by patient malignancy. Each variable was scaled to have a mean of 0 and standard deviation of 1 . A UMAP model was fit to the scaled data and the two-dimensional embeddings were repeatedly plot with colour denoting median F1 score on that sample and the proportion of each class that made up the sample. These plots were visually inspected to understand if classifier performance was dependent on the events which made up a sample.

[0100] To further examine this relationship, Pearson’s correlation coefficient, r, was calculated to establish the strength of association between the median F1 score on a sample following ARF and HT models prediction, and the proportion of each class which made up that sample. Statistical significance was set as p < 0.05. If it were possible to establish relationships between classifier performance and sample characteristics, it may be possible to predict if the model would perform poorly on certain samples before they were analysed. This could prove useful in the instance where an IL algorithm were built into FC analysis software and was responsible for automated analysis, as samples which it suspected would be analysed poorly by the model could be flagged to the user before the poor predictions were shown.

[0101] Prototype Software Development: The aim of the software, that we named “CytoLearn”, was to allow the trialling of IL for automated FC analysis with FC users. CytoLearn was developed to be browser-based to allow ease of access. The software was to contain basic manual gating features, such as the viewing of histograms, two- dimensional scatter plots and density plots. Users should be able to interact with these plots to gate data using tools to make an array of shapes, such as squares, ellipses and polygons.

[0102] The events enclosed within the gates should be able to be labelled and further gated, until all the events within a sample had been labelled by the user. FC data can contain one million events, or more, in a single sample. Rendering even a fraction of these events on plots can be a challenge. For this reason, the graphical user-interface (GUI) was to be developed using PixiJS, a fast, two- dimensional WebGL Tenderer, that allows the rendering of a large number of data points and for users to interact with them, such as for manual gating.

[0103] Once users had labelled all the events they wished to within a sample, the software was to be designed to allow the training of an IL model on the labelled data. Following model training, the user should be able to make predictions on new samples or update the model with new training data. These requests were to be handled using slurm. Slurm is a workload manager for job scheduling on Linux clusters. A request should be sent from the CytoLearn software to either, train a model or make a prediction using a model. Slurm should add this job to a queue for an HPC to process and the user be notified once the job was complete.

[0104] Results

[0105] Model Performance: Twenty ARF and HT models were trained, then evaluated, one sample at a time on randomly shuffled orders of 152 labelled LST samples. Once converged, both ARF and HT models showed high F1 scores (median = 0.93, IQR = 0.06 and median = 0.87, IQR = 0.10, respectively), though Mann-Whitney U test indicated that ARF showed significantly higher F1 scores than HT (U = 277,127, n1 = 860, n2 = 1 ,380, p < 0.001), while also showing less variation in performance (Figure 2). This means that ARF generally performed better at predicting the correct event labels in a sample than HT, but also did this more consistently. Figure 3 shows rolling mean F1 scores of ARF and HT models over the number of samples trained on. Both models appear to learn at similar rates and performance appears to plateau at a similar point, though convergence for the HT models was calculated as taking place after 82 training samples, whereas for ARF it was 108 samples. In theory this means, that while the ARF demonstrated higher F1 scores than the HT, it required more training data to reach that point. In practice, the plot shows this effect to be negligible, if it is indeed the case. Both models showed high F1 scores quickly, meaning that the models appear to learn quickly and require few training samples before demonstrating good performance. The median F1 score over the first 15 samples was 0.87 (0.09) for ARF models and 0.81 (0.12) for HT models. Model performance continues to improve after being introduced to more training data as the models appear to continue learning, before plateauing after being trained on around 90 training samples, at which point it seems that the introduction of more training data no longer improves model performance. The final rolling mean of F1 scores after 152 training samples was 0.94 (0.07) for ARF models and 0.87 (0.11) for HT models.

[0106] Though ARFs showed less variation in model performance than HTs, both algorithms showed a high degree of variability. While the majority of F1 scores are very high, there is a negative skew of the distributions, and some F1 scores are as low as 0.55, especially for HT (Figure 4). It was for this reason median and IQR are used to describe the data, as they are more robust to outliers, than mean and SD. This variation appeared to be within models rather than between models. Figure 5 shows each model’s F1 score changes from sample to sample. Rather than the variation being caused by some models performing particularly well and other models performing particularly poorly over the course of the entire dataset. Furthermore, there is a denser area of lines towards the top of the plot, and fewer lines below, indicating the skewed performance of data is present throughout evaluation, and not just a product of poorer model performance during the early phases of training. Training and Prediction Time: Median training time changed little throughout the course of HT model training (Figure 6). In contrast, during ARF model training, training time gradually increases from around 900 seconds per sample, to 1 ,100 seconds per sample from the commencement of model training to the model being trained on 90 samples. At this point, model training time decreases to around 950 seconds per sample after around 100 samples, before increasing slightly, and then continuing to decrease to around 600 seconds per sample when training ends after 152 samples. There is a general trend of ARF training time increasing until around the same point at which F1 score starts plateauing, and then decreasing. Following model convergence, ARF took, on average, 7 times longer to train per sample than HT (ARF: median = 816 seconds, IQR = 674 seconds vs. HT: median = 117 seconds, IQR = 45 seconds) (Figure 7).

[0107] Prediction time follows a similar pattern to training time. HT stays stable throughout model training (Figure 8). However, ARF training time remains relatively consistent (~250 seconds) until around 80 samples have been trained on. At which point, prediction time begins decreasing to around 225 seconds until around 1 15 samples. Then, model training time increases slightly, before decreasing to just below 150 seconds per sample, after the full dataset has been used for training. Similarly to training time, there is a trend of ARF prediction time beginning to decrease at around the similar time to when F1 score starts plateauing. Following model convergence, ARF models took an average of 3 times longer (ARF: median = 196 seconds, IQR = 118 seconds vs. HT: median = 65 seconds, IQR = 24 seconds) (Figure 9).

[0108] Individual Class Performance: Figure 10 shows the F1 , precision and recall per class for every evaluation of a model, after convergence. In every class, median F1 was higher and showed less variation in ARF models compared to HT. The classes with the highest median F1 scores from models of both algorithms were “CD4+ T cell”, “CD8+ T cell” and “Not lymphocyte” (Figure 27A). Dual positive T cells and B cell classes had the lowest median F1 scores for both algorithms. This indicates these were the most and least challenging classes for the algorithms to classify, respectively. Median precision for every class varied greatly between ARF and HT (overall precision - ARF: median = 0.91 , IQR = 0.12 vs. HT: median = 0.80, IQR = 0.17) (Figure 27B). As was the case with F1 score, ARF showed greater precision than HT for every class. In HT, precision was lowest in dual positive T cells and B cell classes, though especially low in dual positive T cells where precision was only 0.41 (0.62). In contrast, while B cell classes had the lowest precision resulting from ARF models, dual positive T cells had a median precision of 1 .0, though this was variable (IQR: 0.2). This means that both ARF and HT models frequently mislabelled events as B cells, which were not B cells. While HT models frequently mislabelled events as dual positive T cells, if ARF identified an event as this class, it was usually correct. Recall was generally high across all classes, across models from both algorithms (Figure 27C). Despite showing low precision, recall was high for B cell classes, though showed less variation in ARF. This indicates that events which were B cells, were generally labelled as such. Though also, that some of the events labelled as B cells, had been misclassified. Dual positive T cells had the lowest median recall of any class, for models of both algorithms, though in ARF the median was lower and the variation greater, than for HT (ARF: median = 0.78, IQR = 0.35 vs. HT : median = 0.88, IQR = 0.25). This indicated that ARF models frequently did not label dual positive T cell events correctly, and instead mislabelled them as another class.

[0109] Confusion matrices shows how instances of each class were labelled by the models (Figure 11). Each row and column represent a class, and each cell represents the proportion of total instances belonging to that class (y-axis) and the label the model predicted them to be (x-axis). Each evaluation of the model generates a confusion matrix. Figure 1 1 is the aggregated confusion matrices generated following model convergence, each cell shows mean ± SD, rather than median (IQR) as is preferred throughout the rest of the study. Despite the data being negatively skewed, and the mean being therefore more pessimistic of mislabelling errors, the mean offers a more intuitive reading of the aggregated confusion matrices compared to median, as rows sum to one and therefore values represent proportion of instances of that class. Furthermore, as it is more pessimistic, the mean highlights mislabelling errors more so than the mean. The confusion matrices show that when B cell sub-types were incorrectly predicted, they were most frequently predicted as the other sub-type, by both models. When dual positive T cells were incorrectly predicted, the HT models would be equally likely to confuse them with “CD4+ T cell”, “CD8+ T cell” or “Not lymphocyte” classes. ARF models, however, occasionally confused dual positive T cells instances with “Not lymphocyte”, but were around 4 times more likely to confuse them with “CD4+ T cell” or “CD8+ T cell” classes.

[0110] Figure 12 shows how class performance changes over training time. “CD4+ T cell”, “CD8+ T cell” and “Not lymphocyte” classes all perform very well, very quickly, achieving almost perfect F1 scores when averaging over the first 15 evaluations of models, for both algorithms. In both ARF and HT, dual positive T cell performance starts poorly (ARF: median = 0.57, IQR = 0.36 vs. HT: median = 0.48, IQR = 0.43), though in ARF it quickly improves to around 0.8 after 60 samples or so, before staying relatively consistently around that level, suggesting more training samples were required than with other classes to achieve similar performance. However, in HT, performance only slightly improves with model training and stays around 0.6. B cell sub-type performance starts quite well around 0.75 for HT and 0.85 for ARF, though quickly improves, especially in ARF. However, B cell performance remains variable throughout model evaluations, especially in the case of HT, suggesting they were particularly challenging to predict despite continued model training.

[0111] Sample Performance: UMAP was fit to descriptive statistics of marker expression and scatter of each sample. Samples with similar class proportions were generally projected closer together, indicating class proportion of samples reflected the samples descriptive statistics (Figure 13). Samples with very high proportions of “CD4+ T cell” labels were located closer together. Samples with higher proportions of CD8+ T cells were positioned more towards the centre of the plot, as generally were samples with higher proportions of other T cells. Samples with higher proportions of events which were not lymphocytes were located towards the top of the plot. Samples with the highest proportions of these events were the most heterogenous of samples which had a high proportion of any class. Samples with a high proportion of B cell sub-types were located towards the bottom left of the plot and were closely clustered together, indicating those samples were particularly similar.

[0112] Figure 14 shows the same UMAP embedding as the previous figure, though with colour indicating median F1 , precision and recall of each sample following HT and ARF model convergence. Despite samples with similar class proportions generally being closer in proximity on the UMAP embedding, there was generally no discernible pattern between performance metrics and samples with higher class proportions. This indicated that there was little relationship between the proportion of classes which made up a sample and how well that sample was evaluated. Generally, there seemed more samples which had higher values for median F1 score, precision and recall in the ARF models, than the HT models.

[0113] No linear correlations between the class proportion of a sample and median F1 score of a sample showed strong relationships (Figure 15). The strongest correlation was -0.13 and none of the associations were significant, indicating that sample class proportion was not correlated with sample performance, and therefore did not heavily influence it.

[0114] Figure 16 shows a boxplot of F1 score for each sample over all converged ARF and HT models. Six samples were evaluated with median F1 scores of less than 0.85. Plotting the labels of these samples show that three have a single large cluster of B cells. The centroid of these clusters appears fairly equal in anti-Kappa and anti-Lambda expression suggesting their identity is not clearly either Kappa or Lambda B cell, though the events in the entire cluster have been labelled as either “Kappa B cell” or “Lambda B cell” (Figures 17 - 19). The models could be misclassifying these ambiguous events as the other B cell class which could be leading to decreases in F1 score. Other than the B cells in these three files, none of the other samples showed any clear discrepancies, such as labelling errors (Figures 20 - 22).

[0115] Prototype Software Development: Prototype CytoLearn software was developed. Users can upload fcs 2.0 or 3.1 files for analysis. Once uploaded, the data is automatically compensated if a spillover matrix is available in the fcs metadata. Fluorescence channels are transformed using the Logicle transformation, and these values are cached for use with model training, if required. Users can create two-dimensional scatter plots, density plots and histograms of the data. Data can be viewed either linearly or through log and Logicle transformations. Users can draw sequential gates on plots using a range of tools, allowing for event labelling. Summary statistics of the data can be displayed in a table and exported as a csv (Figure 23).

[0116] The label and event data are then used to train ARF models, which can then be used to make predictions on unseen fcs files when these are loaded in for analysis. The predicted labels are converted into gates, which users can adjust if required. This saves the users analysis time, even if the models predictions do not exactly match the labels the user would assign themselves, and the gates need adjusting. The corrected labels and data are then used to perform further model training. Users can train new models if they wish to analyse data generated from different antibody panels or if they wish to label different events. CytoLearn reads the metadata of fcs files and suggests a compatible model, if available, which can be used to train and predict data derived from that antibody panel. Each model is also paired to a template. A template is a configuration, specified by a user, which defines an order of plots. Each plot is defined by its type, the parameters plotted, and the transformations applied to each axis. This further helps speed up manual analysis as users are not required to repeatedly set these parameters prior to gating every file. CytoLearn is currently being trialled with several FC user groups including: for blood cancer diagnosis, for Zika vaccine development, (using FC for quantifying T cell response), and for minimal residual disease (MRD) analysis of chronic lymphocytic leukaemia for clinical trials.

[0117] Discussion

[0118] Incremental learning algorithms are effective for automated flow cytometry analysis: The aim of this study was to evaluate the use of ARF and HT for automated FC analysis of known populations. Both algorithms showed high F1 scores after being trained on 152 samples (ARF: median = 0.94, IQR = 0.07 vs. HT: median = 0.87, IQR = 0.11). Models required few training samples to perform well, median F1 score over the first 15 samples was 0.87 for ARF models and 0.81 for HT models. This demonstrated how IL models can be trained quickly, and assuming the antibody panel, gating strategy, and labels are the same, and samples are somewhat heterogenous, ARF and HT would be effective at automating FC analysis of known populations. Furthermore, considering the findings of the previous study where no benchmarked method achieved F1 scores ofgreaterthan 0.6, these IL algorithms would likely be more effective than previously published automated methods at this task. ARF models performed particularly well, showing similar performance to batch learning RF in the previous study (mean ± SD ARF F1 score over converged models: 0.93 ± 0.05 vs. BL RF performance: 0.95 ± 0.05). ARF models demonstrated higher and less variable F1 scores than HT models, which may make them a preferred choice in practice.

[0119] RF models tend to outperform decision trees, as they are less likely to overfit training data. RF performance was not significantly better than CART in the previous study, however HT may have been outperformed by ARF in the study due to the differences in the way BL and IL trees are generated. CART splits are made by calculating the optimal split on the entire training data, HT makes splits once it is believed enough training instances have been seen to be reflective of the entire dataset. This is determined by the range of the data and o, a parameter which dictates the probability that a split is optimal. A greater 8 offers a greater guarantee that a split is optimal, though more training instances are required to meet this guarantee. If 8 were set lower than would have been ideal for HTs, splits would have been made earlier on in model training, though these have less guarantee of being optimal. HT models made predictions resulting in high F1 scores within the first few samples evaluated, suggesting splits were being made quickly, certainly before a wide variety of training samples were seen by the models. This could indicate that HT is in danger of overfitting the first few samples it is trained on. Furthermore, it highlights that the initial samples seen by an IL model are exceptionally important for training and therefore future model performance. Though low 8 could be responsible for poorer HT performance than its analogous BL model, 8 was set lower for ARF, though did not appear to substantially impair performance. This is likely because ARF and RF do not rely on each tree making optimal splits. In fact, it is a feature of the algorithms that individual trees are prevented from making optimal splits for a set of training data, as this will lead to overfitting, instead each tree is trained on a bootstrapped dataset and a subset of features. It is accepted that individual trees will produce errors, but hoped that the majority of trees will be accurate enough to result in the correct classification, as the resulting prediction is voted by the ensemble. This tends to result in a more generalised classifier, which is less likely to overfit data than CART. Therefore, despite 8 possibly not being optimised for ARF models, the nature of the algorithm suggests they may be somewhat resilient to this. Interestingly, it seemed that the ten trees which made up the ARF ensemble in the current study, did not perform substantially poorer than the 100 trees in the RF ensemble in the previous study. This may suggest that fewer than 100 trees are required for optimal model performance which would reduce memory requirements.

[0120] Though it was calculated that ARF models required more training samples before reaching convergence than HT, this seemed negligible in practice. The median F1 score of ARF and HT models appears to increase at a similar rate before plateauing at a similar point, following around 80 training samples. At which point model performance did not seem to consistently improve with further model training.

[0121] Despite ARF showing higher F1 performance than HT models throughout model training, converged ARF took seven times longer to train on average than converged HT models (ARF: median = 816 seconds, IQR = 674 seconds vs. HT: median = 117 seconds, IQR = 45 seconds). This could be problematic for software implementations of IL for automated analysis, as users would be waiting 13 and a half minutes on average for an ARF model to learn from the previously analysed sample. T raining time may be particularly important earlier on in the model training process when data from a single sample more greatly impacts model performance. Once models are performing well, it would be less important that models have finished being trained on all previously analysed data before the model is used for predictions on the next file. For a model that is already performing well, a single sample of training data would provide little additional benefit and further model training could occur in the background, while an earlier version of the model could be used for analysis with little performance detriment. The newly trained model could be deployed for predicting labels, in place of an older model, once training from all samples is complete. HT training time stayed constant throughout model training. ARF training time gradually increased until model performance plateaued, possibly as models became increasingly complex. Following model performance plateauing, training time per sample generally decreased. This could be because once model performance is no longer improving, models are learning less, meaning less changes are being made to the structure of trees, thus less processing. Alternatively, the decrease in training time may have been independent of model performance and it may have been, by chance, shuffling of the file names led to files containing less data being utilised later in the training process. If these samples contained less data, they would have taken less time to process.

[0122] ARF prediction speed was three times longer on average than HT models, per sample (ARF: median = 196 seconds, IQR = 118 seconds vs. HT: median = 65 seconds, IQR = 24 seconds). Both average prediction times were longer than the average prediction times of all other methods in the previous study, the longest of which was the BL RF which took 22.69 seconds per sample, on average. This is less time than the hours it can take to run some automated methods and, more importantly, the 20 minutes it takes to manually gate data, which is the gold standard analysis method. However, waiting over 3 minutes, on average, for ARF models to predict labels from a sample may still prevent a barrier to adoption in practice if users desire faster analysis. Prediction times for HT did not change substantially throughout model training, however for ARF, model prediction time started to decrease after the model performance began to plateau. It is unknown why prediction time would decrease due to model performance plateauing. A more likely explanation may be that it is indeed the case that samples with fewer events were used towards the end of model evaluation and that these samples took less time to predict. BL CART and RF models showed shorter prediction times in the previous study (CART mean ± SD: 0.20 ± 0.02 seconds and RF mean ± SD: 22.69 ± 2.29 seconds) than their IL counterparts, which suggests there is the potential for faster prediction times to be achieved if the algorithms can be optimised. Sklearn implementations of the BL models were used, whereas River implementations of IL models were used. Sklearn is a well-supported package and, as well as other optimisation features, allows for the use of parallel processing, which River (version 0.12.1) does not currently support which would result in faster prediction and training times.

[0123] Some classes perform better than others: As was the case with the tree-based classifiers in the previous study, both HT and ARF struggled to classify B cell sub-types. Both “Kappa B cell” and “Lambda B cell” classes are relatively abundant in the dataset, making up 12.29% and 1 1.67% of the dataset respectively, therefore lack of training instances in the dataset would not seem to explain poor performance. Reasons forthe poor performance of tree-based algorithms at classifying B cell sub-types has been discussed more in the previous study, though in brief, these events are often almost identical and are labelled by considering the surrounding events, whereas the classifiers used in the current studies have considered each event in isolation. Performance of both B cell sub-types was particularly variable over the course of model training, with the rolling mean even decreasing on occasion, suggesting that at a point, receiving increasing instances of training data did not improve the algorithms capacity to distinguish B cells. Fluctuating B cell performance over time may be explained by certain samples containing more B cells than other samples or by containing B cells which, either were particularly difficult to distinguish or had been incorrectly labelled, and therefore received low F1 scores and reduced the rolling mean.

[0124] In the previous study, mean (± SD) classifier recall and precision were similar for both B cell sub-types (recall - Kappa: 0.93 ± 0.06; recall - Lambda: 0.83 ± 0.16; precision - Kappa: 0.85 ± 0.13; precision Lambda: 0.9 ± 0.13), and median recall and precision were also high during the classifiers in the study (recall - Kappa: 0.99; recall - Lambda: 0.99; precision - Kappa: 0.96; precision - Lambda: 0.90). However, though standard deviation was low for median and recall in the BL classifiers, IQR was large for precision (Kappa: 0.98 and Lambda: 0.82) in the IL algorithms. High recall and variable precision suggest that, generally, B cells tend to be labelled correctly, however frequently other events are mislabelled as these sub-types as well. Given that for an instance to be labelled as a certain class, it cannot be labelled as another, it would be expected that recall would be lower in the mislabelled class, as it represents an instance which has not been correctly recalled. However, this does not seem to be the case, possibly suggesting that the instances which have been regarded as false negatives belong to a range of different classes, which may not have been enough to lower the recall of one particular class. Though, the confusion matrices appear to suggest this is not the case and most frequently B cell classes are confused as one another, and all other classes are rarely, if ever, mislabelled as B cells. Therefore, it is not known why precision is often low for B cell classes. Furthermore, the confusion matrices suggest that IL algorithms confuse B cell sub-types slightly less frequently than their BL counterparts in the previous study (ARF - Kappa (True), Lambda (Predicted): 0.03 and Lambda (True), Kappa (Predicted): 0.04 vs. RF - Kappa (True), Lambda (Predicted): 0.05 and Lambda (True), Kappa (Predicted): 0.13). This seems unlikely in practice, as any deficiency in BL tree-based algorithms which would lead to poor B cell performance, would also exist in IL tree-based algorithms. Instead, this seems to be related to the high recall shown by the IL algorithms. When confusion matrices are normalised to the counts of true labels, as they are in the study, confusion matrices reflect recall. And the calculation for the diagonal, where predicted class is also the true class, is equal to that of recall, TP / TP + FN. Therefore, though it may seem that B cell sub-types are confused less by IL algorithms, than BL algorithms, this appears to be a function of how confusion matrices are calculated.

[0125] Though higher in ARF than HT (median = 0.82, IQR = 0.27 vs. median = 0.60, IQR = 0.45), dual positive T cells had the lowest F1 score of all classes. This was also the case with BL classifier performance in the previous study (mean ± SD: 0.76 ± 0.11). This is possibly because it is the minority class of the dataset, making up only 0.03% of total instances. Dual positive T cell F1 score was lower in HT than ARF, though HT showed higher recall (median = 0.88, IQR = 0.25 vs. median = 0.78, IQR = 0.35). This suggested HT more frequently predicted dual positive T cells than ARF, but often these were of a different class. According to the confusion matrices these were usually either, CD4+ T cells, CD8+ T cells or events which were not lymphocytes. Dual positive T cells showed the greatest discrepancy of any class in F1 score between HT and ARF, largely explaining the difference between HT and ARF F1 score overall. The poorer HT performance with dual positive T cells may have been because non- optimal splits were made too soon. Splits are made once the Hoeffding bound indicates enough training instances have been seen which are representative of the entire dataset. This could be particularly problematic for minority classes in IL, as even fewer samples would be seen of that class by the model when a split is made, compared to BL. The fewer instances of a class are seen, the less likely those instances are representative of the wider class population. This could have been why dual positive T cell class performance did not particularly improve in HT models despite seeing increasing numbers of training samples. It could also indicate a wider issue; that IL particularly struggles with minority classes compared to BL. Once HT and ARF make a split, the split cannot be changed, therefore critical mistakes could be made early on which cannot be corrected. Extremely Fast Decision Tree (EFDT) are IL treebased algorithms which revisit early splits. If a better split becomes available, the split is replaced. Algorithms such as EFDT could favour minority class performance as sub- optimal splits made early on in model training can later be corrected.

[0126] F1 score was higher in ARF for dual positive T cells than HT, this may have been because, as discussed, if non-optimal splits are made by the trees within an ARF ensemble, they are less critical, as numerous trees vote for the classification. This allows for a more generalised model. Despite ARF generally performing well, ARFs median recall of dual positive T cells was the lowest median recall of any class for both algorithms (median = 0.78, IQR = 0.35). However, ARFs median precision of dual positive T cells was 1 . This indicated ARF somewhat rarely predicted the dual positive T cell class, even when instances were of that class. However, when the ARF did predict an event was a dual positive T cell, it was usually correct, as precision was high. When it was incorrect, the events which were predicted as dual positive T cells, were usually either CD4+ or CD8+ T cells, which they are very similar to. RFs tend to perform poorly on minority classes. This is because instances from a minority class are less likely to be sampled in a bootstrapped dataset, therefore splits are made considering fewer dual positive T cell instances. This may explain why the most frequent classes dual positive T cells were confused with were CD4+ and CD8+ T cells, as they were in a very similar feature space. In fact, only one further decision, or split, would be required to classify both CD4+ and CD8+ T cells as dual positive T cells, which was whether expression forthe opposite markerwas also high. Unlike dual positive T cell class performance with the HT, class performance continued to consistently improve with further training with ARF. This could be because the initial models which were generated by ARF from early training samples were less overfit and more general than the HT, then only once enough training samples were seen, did the trees in the ensemble make splits which they were more confident of. This meant the ARFs made a more gradual shift in their decision-making process than HTs, and this gradual shift offered even higher F1 scores.

[0127] ARF showed higher recall and lower precision forthe minority class and HT showed the inverse. Therefore, the decision between whether HT or ARF is preferred by the user may depend on the problem which they wish to solve. ARF may lead to more false negatives of the minority class, instead mislabelling the minority class as another class. While the HT may lead to more false positives of the minority class, mislabelling instances of other classes as the minority class. There are some instances in FC analysis where the minority class is the population of interest, for instance when attempting to identify very low levels of cancer in tissue, known as MRD detection. When used for this purpose, the HT may be preferred as, based on the results of this study, HT shows a preference for labelling the minority class, and while these were often false positives, in the case of MRD this may be preferable to failing to identify cancerous cells, which could negatively impact patient prognosis, if this meant they failed to receive the treatment they required soon enough.

[0128] “CD4+ T cell”, “CD8+ T cell” and “Not lymphocyte” classes showed the highest F1 score of all classes (All classes: median = 1 .0, IQR = 0.0), suggesting these were the least challenging to classify. Even after 15 training samples, all three classes averaged almost perfect F1 scores. This suggested that good quality splits in the trees were made quickly. This was likely due to several factors. Firstly, these events are not particularly challenging to manually gate, compared to B cell sub-types, for example. Therefore, they could be easier for a tree- based algorithm to discriminate, and the dataset was more likely to be correctly labelled for these instances. Secondly, these classes were all particularly abundant, making up the top five most abundant classes in the dataset. The other two classes in the top five were the B cell sub-types. This indicates that plenty of samples were seen early on by the models by the time initial splits were made. This further indicates the caution required with imbalanced classes during IL. As the most abundant classes tend to have performed best, while, as discussed, the least abundant, have shown poorest performance.

[0129] Some samples perform better than others: F1 score data was negatively skewed for both algorithms, though generally very high, there was a long tail of low scores. There were lower F1 scores associated with evaluation earlier on in the training process, however these early evaluations did not appear to solely explain the long tail. Plotting each individual model performance over increasing training samples showed that performance varied greatly within models, even once model performance plateaued suggesting models were fully trained. Throughout training, most evaluations of the model resulted in high F1 scores which, when each model evaluation was plotted as a line plot, resulted in a denser region of lines towards the top of the plot and occasional incidences where model evaluation was particularly poor.

[0130] UMAP and correlation analysis aimed to establish whether the poor performance that occasionally occurred across models was due to poorly performing samples containing a greater abundance of classes which the models struggled to discriminate i.e., B cell sub-types and dual positive T cells. UMAP suggested the descriptive statistics of marker expression reflected the proportion of classes which made up a sample. Samples containing higher proportion of classes were closer together on the projection. Though this was less apparent with “Not lymphocytes” than any other class, which were less closely grouped compared to other classes. This is understandable considering "Not lymphocytes” are a range of cell types, not a single cell type like all other classes were, therefore marker expression would be far more variable. The UMAPs did not appear to indicate any clear relationship between F1 score, precision, and recall, and the proportion of classes making up a sample. Furthermore, linear correlations between the proportion each class made up a sample and the F1 score of the sample were weak and insignificant. These findings suggest that the classes which performed poorly were not sufficiently problematic to negatively impact the performance of an entire sample. It also suggested that negative performance of a sample could not be predicted prior to evaluation based on the descriptive statistics of marker expression, which could have been a particularly useful feature for deploying IL models in practice, as advance warning could have been given to a user prior to automated analysis being made.

[0131] Overall, occasional poor sample performance did not appear to be explained by individual models performing poorly, nor the poor performance of certain classes, nor similar types of samples performing poorly. Plotting the labels of the poorest performing samples showed three of those samples had a large number of B cells which had been labelled as either Kappa or Lambda B cells, though were of ambiguous identity as anti-Kappa and anti-Lambda expression were similar. This could suggest that some of the samples in the dataset were incorrectly labelled by the operator and the models were “correctly” classifying these events, though being penalised for it. One indication of this is that there is a particularly distinct cluster of samples in the far bottom left corner of the UMAP plot. Some of these samples seem to contain very high proportions of Lambda B cells and others very high proportions of Kappa B cells, despite the samples seeming to be exceptionally similar. This may be unusual, as despite Kappa and Lambda B cells being similar, the marker expression of anti-Kappa and anti-Lambda would still likely be different enough to classify such a large number of events differently. Furthermore, in the UMAP indicating F1 score, this region of samples seems to generally have a slightly higher median F1 score from the HT models, than the ARF models, which is unusual considering ARF F1 score was higher than HT for the B cells. The most likely explanation is that the B cells in these samples were similar to those plotted in three of the six poorest performing samples, and that identity of the B cells was particularly ambiguous, or they had been incorrectly labelled.

[0132] Future research: The main limitations of the current study largely relate to the limitations of the dataset, which have been previously discussed. In short, only a single antibody panel was used to evaluate the IL algorithms. Now that CytoLearn software has been developed, the trialling of IL with other datasets is substantially easier, as users can upload their own data which they can gate and label as they wish. Rather than the labour-intensive process of the authors being required to generate their own labelled datasets, this can be outsourced to the users who benefit from the automated analysis of their FC data at the same time as testing CytoLearn. Problems also potentially arose from, what could have been, incorrectly labelled samples. A single analyst gated the samples to label the data due to staff constraints, though previous comparison studies have involved numerous operators labelling the data and a “consensus dataset” being generated from the majority label given by most operators. This is preferable as one operator’s bias is less likely to influence the labelling and errors are less likely to be made.

[0133] Future research should seek to further understand model performance, as certain findings could not be fully explained in the study. The IL models in the study generally showed higher recall, but lower precision than the BL models in the previous study, where precision and recall were similar to one another. It is unknown why this resulted as any decrease in precision would seemingly need to be accompanied by a proportional decrease in recall in a corresponding class, which does not seem to have been the case. To understand this further, the performance of individual samples and class performance within them should be examined.

[0134] The early samples used to train a model seem of particular importance as trees make splits based on them, which cannot be re-evaluated later in model training. Research to further understand this could help inform best practices for users to follow to prevent models making critical mistakes early on which are detrimental to model performance throughout the life of the model being used. Additionally, testing should be performed of algorithms which are able to re-evaluate splits, such EFDT, as these may provide a more robust alternative to HTs, when used individually and as the base classifier within an ensemble in ARF. These trees would be able to correct sub-optimal splits which were made early on in model training which could prevent models becoming ineffective. While revisiting splits made earlier on in training would likely be beneficial for improving performance, delaying splits until more training data has been seen would limit the likelihood of sub-optimal splits being made in the first place. In the study, default 8 was used for both HT and ARF. Future research could investigate optimising 8 to improve classifier performance. Models performing well after being trained on few samples, suggested splits were made quickly. A higher 8 would lead to splits being made later, which may reduce short-term model performance, but may prevent splits being made until sufficient training samples have been seen to better represent the entire data stream and prevent erroneous splits being made early on, leading to improved model performance long term.

[0135] There were indications that certain samples consistently showed poor F1 scores. Plotting the labels of these samples suggested that some of the samples may have been incorrectly labelled, however, some of them did not appear to be. The samples which performed poorly should be analysed along with the samples which performed well. Possibly by considering the distribution of each feature within a class to understand if samples which performed poorly were markedly different from other samples. As well as poor F1 scores, it was possibly the case that certain samples were associated with shorter prediction and training times. This could be as there were less labelled events in these samples which took less time to process than larger samples. There was evidence that IL algorithms struggled with the minority class. Over-sampling is a technique used to improve minority class performance, often where synthetic data is generated from the minority class and used for model training alongside the real data. Given the same 8 and the same range of data, the Hoeffding bound would be equal, and a split would be made after seeing the same number of training instances. However, if minority classes were over-sampled and greater in number, they would make up a greater proportion of the training instances, and the ensuing split would regard the minority class as more important compared to if it were not over-sampled. Gini impurity is calculated on combinations of binary splits, with the split with the lowest impurity chosen. If a class is more represented, then it has greater weight when Gini impurity is calculated, and the splits are more likely to favour the classification accuracy of that class. The trade-off is that the increased training data from up- sampling would likely lead to longer training and prediction times, though this could be balanced by down-sampling instances of majority classes i.e., removing data points. EFDT could also offer a solution to improve minority class performance as incorrect splits made early on in training, before enough instances of the minority class has been seen, could be revisited, and changed if required.

[0136] ARF was more accurate than HT, making it a more attractive proposition for use as an IL algorithm in an automated FC analysis software solution. Though, ARF had substantially longer training times (converged ARF: median = 816 seconds, IQR = 674 seconds) and prediction times (converged ARF: median = 196 seconds, IQR = 118 seconds) than HT. This may present a barrier to the adoption of IL as a solution for automated FC analysis, as long wait times for prediction or training could present a poor user experience. Several strategies exist which could reduce training and prediction time. Firstly, ten trees were used in the ARF ensemble, reducing this would reduce training and prediction time, however it could also reduce model performance. Secondly, if trees within the ensemble were trained and made predictions in parallel this would likely also reduce processing times, as this processing could occur simultaneously, rather than sequentially. Code performance improvements could also assist processing times, including the use of Python 3.1 1 which is up to 60% faster than previous Python versions, or using a low-level programming language such as C.

[0137] Conclusion: In conclusion, IL algorithms show excellent potential as a generalised solution for automated FC analysis of known populations, offering the flexibility of unsupervised clustering methods, being able to analyse data generated from any antibody panel, combined with the accuracy of supervised classification methods. A prototype software solution, CytoLearn, was introduced which is available for FC users to trial. CytoLearn supports the manual analysis of FC data derived from any antibody panel. Labels derived from the gating process were used to train IL models which could be used to predict events in unseen FC files. These models continue to be trained with increasing samples manually analysed. ARF was more accurate than HT, though took substantially longer to predict and train. Future research should prioritise the improvements to the IL algorithms. Minority classes appear to cause difficulty for HT and ARF algorithms. B cells were relatively poorly classified, as was the case in BL tree-based algorithms. Furthermore, certain samples appeared to be analysed with more poorly than others, however sample performance did not correlate to the proportion of labelled events in a sample, indicating that the poor performance of B cells and dual positive T cells was not detrimental to overall sample performance. Benchmarking

[0138] Dataset: The Haemato-Oncology Diagnostics Service (HODS) based at the Royal Liverpool University Hospital carries out primary reporting of haematological malignancies for the 2 million people within the Merseyside and Cheshire area. Flow cytometry (FC) is a commonly used diagnostic technique for such disorders. One hundred and fifty-two fcs 2.0 files were used for the dataset which had been collected by HODS between the dates of February 2018 and March 2020. The data was generated from peripheral blood samples from patients with suspected haematological malignancies for whom FC had been performed as part of the diagnostic pathway. Samples had been stained with the EuroFlow Lymphoid Screening Tube (LST) panel (BD Biosciences, San Jose, CA) for the purpose of identifying aberrant B, T and NK cell lineages. All samples were analysed on FACSCanto cytometers (BD Biosciences, San Jose, CA). Each file had been compensated and contained 100,000 events from a single patient sample. As part of HODS’ standard operating procedure, each file had undergone a quality control process within HODS which included samples being checked for any signs of degradation during histopathological examination and the ability to observe distinguishable clusters on SSC and CD45 parameters of FC data.

[0139] Gating: Each fcs file was gated by a single trained FC operator using manual gating software (FlowJo, BD Biosciences, San Jose, CA). Each sample was gated to label events the operator was confident of the identity of. One of nine labels was given to these events: “Not lymphocyte”, “Kappa B cell”, “Lambda B cell”, “CD4+ T cell”, “CD8+ T cell”, “CD4 / CD8 + T cell”, “CD4 / CD8 - T cell”, “Gamma delta T cell”, “CD56+ NK cell”. Each class of labelled event from a file was exported to a csv.

[0140] Pre-processing: The labelled events from each sample were combined and fluorescence channels were transformed using the Logicle transformation, while FSC and SSC remained linear. The Logicle transformation has been suggested as the preferred transformation for FC data, resulting in fewer misclassified events than other popular transformations. The transformed data from each sample was plotted in 2D scatter plots and checked for visual similarity to the equivalent unlabelled sample to ensure that transformation was successful. Table 1 summarises the labelled events in the dataset.

[0141] Table 1 : Frequency of each class within the dataset

[0142] CytoLearn: Classification and Regression Trees (DT) and Random Forest (RF) are tree-based classification algorithms. DT are binary trees which mathematically determine the best split on single features. Gini index was used as the split criterion. Random forests create an ensemble of trees, with the output being the class voted by most trees. Random Forest are less likely to overfit training data than a single decision tree.

[0143] Incremental learning involves training a model incrementally on small batches of data at a time, compared to a single batch of data as is used during batch learning. Hoeffding Trees (HT) are decision trees which can be trained incrementally and are guaranteed to converge on DT performance when the number of training instances is large enough. Adaptive Random Forest (ARF) is an incrementally learning RF made up of an ensemble of HT.

[0144] Despite limited use in the field of automated FC analysis thus far, classification algorithms which label each event in a sample possess properties which may make them an attractive proposition for use within clinical practice. Firstly, trained classification algorithms have been demonstrated to be more accurate on FC data than clustering algorithms which are entirely unsupervised. Secondly, the primary aim of FC analysis for the diagnosis of haematological malignancy is to identify known populations. Identifying unknown, novel populations, which clustering algorithms are well suited for, is not the goal. Furthermore, clustering approaches still require some manual intervention, as sub- populations require manual labouring, somewhat negating potential benefits of automated analysis. Tree-based classifiers could be particularly well suited to FC analysis, as the manual gating process is analogous to the decision-making process decision trees use, the label given to an event being dependent on whether fluorescent intensities are above or below certain thresholds.

[0145] However, classification methods typically require the unseen data to possess the same input features the model was trained using, meaning that a trained classifier could only be used on unseen data generated using the same antibody panel, reducing its utility. Furthermore, the labelling of datasets required as training data for classification algorithms is time consuming and expensive.

[0146] A panel agnostic pipeline was devised by which a trained analyst could perform manual gating on an FC sample. An incremental classifier could learn from the labelled data of the single sample. The classifier can then predict the labels of the events in a subsequent unseen FC sample. The operator could correct any mistakes made by the classifier by adjusting manual gates. The classifier could again learn from these data instances, further improving model performance. When the classifier predicts the identity of events in an unseen FC sample, it is hopefully even more accurate than previously and less manual correction is required from the analyst and so on, until the classifier converges and, ideally, minimal manual intervention is necessary from the operator.

[0147] Samples from the labelled dataset were shuffled into five different orderings. HF and ARF models were trained incrementally from all instances of one sample at a time. Incoming data is first used to test performance of the model, before being used fortraining. Performance was calculated over a window of the ten prequential samples.

[0148] Benchmarking: Past attempts at automated FC analysis have focused on the use of clustering algorithms to group similar events into sub-populations. Some of these attempts have gained substantial interest and have been subject to several benchmarking and comparison studies. To identify previously published automated FC analysis methods, PubMed was searched with the search term “("flow cytometry" OR "FCM") AND ("automat*" OR "clustering" OR "classification" OR "machine learning" OR "random forest" OR "decision tree*")”. Search results were assessed for relevance and relevant references were explored. The following criteria was used for selecting a short list of methods for benchmarking:

[0149] 1 . The method was designed to be used with FC data to identify or cluster similar events so they can be labelled.

[0150] 2. The method was freely available to use either as source code or application.

[0151] 3. The code must be reasonable to run, either detailing default or suggested parameters.

[0152] 4. Either the method had previously been independently assessed, or had been released since the most recently published critical assessment paper in 2016 i.e. there had been no opportunity for it to be independently assessed.

[0153] All methods which had previously been independently assessed were included in either one of two papers. Due to the large number of methods which had previously been assessed, only the three best performing methods based on mean F1 score from both papers were included for benchmarking. This resulted in a short list of nine methods selected for benchmarking (Table 2). Two of these methods could not be successfully installed for testing and were therefore excluded from benchmarking.

[0154] Table 2: Methods selected for benchmarking

[0155] Ten samples were randomly selected from the labelled dataset. Each method was run five times on each sample. Clusters identified by the methods were matched to manually gated populations using the Hungarian assignment algorithm which maximises the sum of F1 scores across populations, allowing no population to be matched more than once.

[0156] Evaluation metrics: Precision is the proportion of instances of a class which are identified correctly, and recall is the proportion of actual class samples which are identified correctly. F1 score is the harmonic mean of precision and recall. Macro averaged F1 score was calculated (averaged over the set of all classes) and is a preferred evaluation metric when classes are imbalanced. Prediction time, in seconds, of each classifier was recorded. Measurements for running time were made on a cluster with 50 cores and 100GB of RAM.

[0157] Results: Figure 24 shows mean F1 score for all methods. For HT and ARF mean F1 score was calculated only after the models have been trained on data from ten samples. The mean ± standard deviation F1 score for HT and ARF is 0.80 ± 0.12 and 0.88 ± 0.09, respectively. The best performing benchmarked method was cytometree (0.55 ± 0.06). FLOCK showed the fastest prediction time of all methods (0.99 ± 0.19 seconds), whereas HT and ARF took 34.16 ± 7.76 seconds and 141 .13 ± 36.97 seconds, respectively (Figure 25).

[0158] Method: Figure 26 shows a method of training a machine learning algorithm, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters (S2601); and training the machine learning algorithm comprising incremental learning using the set of sample data (S2602).

[0159] Notes:

[0160] Although a preferred embodiment has been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims and as described above.

[0161] Attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

[0162] All of the features disclosed in this specification (including any accompanying claims and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at most some of such features and / or steps are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features. The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims

CLAIMS1. A method of training a machine learning algorithm, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and training the machine learning algorithm comprising incremental learning using the set of sample data.

2. The method according to claim 1 , wherein incremental learning using the set of sample data comprises incremental learning using the first sample data.

3. The method according to any previous claim, wherein the set of sample data includes M sample data, including the first sample data, and wherein the corresponding set of samples includes N samples, including the first sample, wherein M and N are natural numbers greater than or equal to 1 , for example 1 , 2, 3, 4, 5, 6 ,7 ,8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 or more; and wherein incremental learning using the set of sample data comprises incremental learning using the M sample data, for example iteratively.

4. The method according to claim 3, wherein M and / or N are at most 100, preferably at most 50, more preferably at most 25.

5. The method according to any previous claim, wherein training the machine learning algorithm comprises batch learning using a first subset of the set of sample data, for example before incremental learning using a second subset of the set of sample data.

6. The method according to any previous claim, wherein the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating, for example via the computer, the first sample data.

7. The method according to any previous claim, wherein the first sample data comprises and / or is unlabelled sample data; and wherein training the machine learning comprises: inferring respective classifications for at least some of the plurality of single cell and / or particle events of the first sample data using the machine learning algorithm, whereby the first sample data comprises and / or is labelled sample data, having inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data;curating, via the computer, the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data; and wherein incremental learning using the set of sample data comprises incremental learning using the curated inferred respective classifications provided forthe at least some of the plurality of single cell and / or particle events of the first sample data.

8. The method according to claim 7, wherein curating, via the computer, the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data comprises reclassifying, via the computer, at least one of the inferred respective classifications provided for the at least some of the plurality of single cell and / or particle events of the first sample data.

9. The method according to any previous claim, wherein the set of sample data includes second sample data and wherein the corresponding set of samples includes a second sample, wherein the second sample data comprises a plurality of single cell and / or particle events and respective parameters; wherein the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating the first sample data; and wherein the second sample data comprises and / or is unlabelled sample data; and wherein training the machine learning algorithm comprises: incremental learning using the first sample data; inferring respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data using the machine learning classification algorithm trained using the first sample data, whereby the second sample data comprises and / or is labelled sample data, having inferred respective classifications provided for at least some of the plurality of single cell and / or particle events of the second sample data; curating, via the computer, the inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data; and incremental learning using the curated inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second sample data.

10. The method according to any previous claim, wherein the set of sample data includes M sample data, including the first sample data, and wherein the corresponding set of samples includes N samples, including the first sample, wherein M and N are natural numbers greater than or equal to 2, for example 2, 3, 4, 5, 6, 7 ,8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 or more; wherein the first sample data comprises and / or is labelled sample data, having respective classifications provided for at least some of the plurality of single cell and / or particle events of the first sample data by gating the first sample data; and wherein the second to M sample data comprises and / or is unlabelled sample data; andwherein training the machine learning algorithm comprises: incremental learning using the first sample data; and iteratively incremental learning using the second to M sample data comprising: inferring respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data using the machine learning algorithm trained using the first sample data, whereby the second to M sample data comprises and / or is labelled sample data, having inferred respective classifications provided for at least some of the plurality of single cell and / or particle events of the second sample data; curating, via the computer, the inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data; and incremental learning using the curated inferred respective classifications for at least some of the plurality of single cell and / or particle events of the second to M sample data.

11. The method according to claim 10, wherein iteratively incremental learning using the second to M sample data comprises iteratively incremental learning using the second to M sample data until a predetermined threshold accuracy of inferring the respective classifications is achieved.

12. The method according to any previous claim, wherein the set of sample data comprises flow cytometry data, mass cytometry data or scRNA-seq data and / or wherein the set of samples comprises biological samples, for example blood, bone marrow, tissue and / or bodily fluid.

13. The method according to claim 12, comprising acquiring the set of sample data.

14. A method of inferring classifications using a machine learning classification algorithm trained according to any previous clam, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a dataset, wherein the dataset comprises a set of sample data, including first sample data, for a corresponding set of samples including a first sample, wherein the first sample data comprises a plurality of single cell and / or particle events and respective parameters; and inferring respective classifications for at least some of the plurality of single cell and / or particle events of the first sample data using the trained machine learning classification algorithm.

15. A computer comprising a processor and a memory configured to implement a method according to any of claims 1 to 14, a computer program comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of claims 1 to 14 or a non-transient computer-readable storage medium comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of claims 1 to 14.

Images