Computer-implemented method for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a blood smear sample

Abstract

The present invention concerns a computer-implemented method for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample.

Description

[0001] Computer-implemented method for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a blood smear sample

[0002] Technical Field

[0003] The present invention concerns the field of genomic aberration detection by computer-implemented detection methods. In particular, the present method concerns the detection of genomic aberration from an image of a body liquid, blood, bone marrow or cytology sample mounted on a slide, in particular a whole slide image of a blood smear sample.

[0004] Background of the invention

[0005] Genomic aberrations refer to alterations in the DNA sequence or structure, which may involve deletions, duplications, inversions, translocations, gene fusions or point mutations. Such aberrations can have various consequences at the cellular level, manifesting as functional abnormalities, structural disorganization, or uncontrolled proliferation. These aberrations are of particular concern in oncology, where they can serve as biomarkers for cancer diagnosis, prognosis, and targeted therapy.

[0006] Traditional methods for detecting genomic aberrations include fluorescent in-situ hybridization (FISH), comparative genomic hybridization (CGH), real-time polymerase chain reaction (PGR), microarray-based technologies, such as SNP microarrays and Array- CGH, and sequencing technologies, such as next-generation sequencing (NGS) or Sanger sequencing. While these methods offer varying degrees of sensitivity and specificity, they are often limited by their complexity, cost, and time-consumption. Furthermore, the analyses usually involve extensive sample processing on expensive equipment and require specialized expertise, making them less accessible for routine clinical applications.

[0007] The particular appearance of cells in a sample is a reflection of their genomic integrity. Aberrations can lead to atypical cellular morphology, including irregular nuclei, changes in cytoplasm granulation, abnormal chromatin distribution, and changes in cell size or shape. These resulting features are commonly analysed in morphological cytological examinations under the microscope but usually require subjective interpretation by highly trained professionals. Furthermore, the exact link between a genomic aberration and changes in cell morphology is rarely well-characterized. Accordingly, a computer- implemented method that can objectively and reliably detect such changes would constitute a significant advance in the field.

[0008] Existing computer-assisted methods largely comprise machine learning algorithms or heuristic rules to analyse sequencing data that results from the extraction and processing of DNA from patient tissue samples. However, these approaches generally require access to expensive, specialized laboratory equipment and involve time-intensive processing of precious patient tissue samples. Therefore, there remains a need for a more streamlined, time-efficient, and cost-effective method for detecting genomic aberrations. There exist moreover, in the state-of- the-art, visually-based computer-implemented methods for the classification of solid tissue samples as comprising or not one or more genomic aberration. These methods are advantageous in their non- destructive use of slide-mounted solid tissue samples that are made available in the standard clinical workflow. Unfortunately, the said methods cannot be applied to samples of body liquids, as the multicellular architecture, or lack thereof, as well as prevalence and spatial distribution of evaluable cells differs greatly in liquid samples. For example, while solid tissue cancer cells typically aggregate in contiguous regions, cells from a blood sample are dispersed and distributed randomly across the slide. Further, in samples from body liquids, the cells bearing the genomic aberrations can be as rare as 1 :1000, as is the ratio of lymphocytes to red blood cells in the blood of leukemia patients. This different cell distribution between solid tissue cancer and body liquid samples is also reflected in the used digitization procedures: whereas solid tissue sample slides are generally digitized with a whole slide image scanner, blood sample slides are digitized with hematology morphology analyzers, such as CellaVision or Vision Hema that search, detect and photography a limited number of single cells of interest, e.g., lymphocytes [Kratz, A., Lee, S., Zini, G., et al.: Digital morphology analyzers in hematology: ICSH review and recommendations.

[0009] International Journal of Laboratory Hematology (2019). ]. The state of the art lacks a solution that can scale to the thousands of cells required to detect rare morphologies of interest while maintaining the single-cell approach that is most advantageous for liquid samples.

[0010] Liquid samples are however important for the detection of the presence of genomic aberrations that can be linked to specific diseases or pathologies, as well as for the monitoring of the tumour over the course of treatment. For instance, detection of the driver mutation TP53 is of primordial importance, being both prognostic for the aggressiveness of the tumor and predictive of the patient’s response to treatment. By addressing these gaps and limitations in the state of the art, the present invention aims to offer a novel solution that enhances the speed, cost, and automation of genomic aberration detection in samples from body liquids, thereby advancing the standard of care in genomic diagnostics and personalized medicine.

[0011] Summary of the invention

[0012] Thus, the object of the present invention is to propose novel computer-implement methods, with which the above-described drawbacks of the known methods are completely overcome or at least greatly diminished.

[0013] According to the present invention, these objects are achieved in particular through the elements of the independent claims. Further advantageous embodiments follow moreover from the dependent claims and the description.

[0014] In particular, the objects of the present invention are achieved by a computer-implemented method for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample, comprising the following steps: a. Segment at least a part of a whole slide image of the smear sample into single-cell images, advantageously by means of a first trained convolutional deep neural network; b. Embed each single-cell image into a feature vector by means of a second trained neural network, advantageously a trained convolutional neural network or a trained transformer neural network; c. Define at least one bag of feature vectors representative for the sample; d. Derive from the at least one bag of feature vectors a bag value by means of an attention-based Multiple Instance Learning third trained neural network, wherein from the bag value the at least one bag of feature vectors is labelled as indicative or non-indicative for the defined genomic aberration; e. Classify the sample as indicative of the one or more defined genomic aberrations when an indicative value dependent on the number of bags of feature vectors labelled as indicative meets or exceeds a predetermined threshold value; wherein the minimal size of the at least one bag is determined based on the detection limit of the ground truth, said detection limit of the ground truth being defined as the minimal percentage threshold of cells featuring the said one or more defined genomic aberrations.

[0015] Thanks to this method, it is possible to reliably and automatically detect the presence of one more genomic aberration in whole slide images ( WSI) of liquid samples containing cells. This detection method differs from a method for classifying single-cell images as exhibiting a genomic aberration, as the ultimate goal of the present invention is to allow the diagnosis of a disease associated with a specific genomic aberration. To achieve this objective, it is crucial that the number of cells analyzed is representative of the sample. If the cell count is insufficient, there is a risk of producing false negative results, which must absolutely be prevented. The critical step in this process is defining a 'bag' of feature vectors that accurately represents the sample. In particular, the minimum size of the bag is essential to ensure reliable detection. A small bag may not capture sufficient variability within the sample, leading to inaccurate or incomplete results. In contrast, a method for classifying cells as exhibiting one or more defined genomic aberrations does not require consideration of the representativeness of the cells or of the bags of cells within the sample. This is because such methods are primarily focused on classification tasks, where the objective is to assign a specific category or label to each analyzed bags of cells and often to each cell in the bags without any consideration on the representativeness of the bags for the sample. .

[0016] According to the present invention, the minimal size of the at least one bag is determined based on the detection limit of the ground truth, said detection limit of the ground truth being defined as the minimal percentage threshold of cells featuring the said one or more defined genomic aberrations. Advantageously, the minimal size is determined based on the detection limit of the ground truth and the yield of the segmentation step. Even more advantageously, the minimal size is determined based on the detection limit of the ground truth, the yield of the segmentation step and an error value reflecting the statistical variation in bag sampling. In more details, the minimum bag size can be calculated from the detection limit of the ground truth, the theoretical best possible detection limit of the model, and the yield of the segmentation step. The model's theoretical best detection limit for a bag size of M feature vectors is 1 / M, and the segmentation yield Ys is the percentage of cells in the sample examined by the model. If a limit of detection for the ground truth D, for instance of 10%, is known, the bag size M that must be examined is found by: 1 / (M*Ys) >= D + E, where E is an error allowing for statistical variations in bag sampling. This yields a minimum bag size of M=1 / ((D+E)*YS) .

[0017] The variations accounted for by the error E can be due to systematic biases arising from digital or mechanical sample processing, or in response to natural variations stemming from insufficient statistical power. Systematic bias from digital methods can arise for instance during image segmentation, when certain cell types or cells with specific features are preferentially identified and segmented by the model. Alternatively, systematic biases can arise through biochemical or mechanical means during preparation of the smear, such as the systematic destruction of certain delicate cells or intra-cellular structures due to biochemical or mechanical stress, or the sequestration of large cell types in a thicker end of the smear which is more difficult to image. In the case of systematic biases, the error rate can either be derived theoretically or measured empirically through repeated testing. Additionally, sampling variations can arise from insufficient statistical power, especially if computational or time resources are limited. This error can advantageously be estimated by modelling the selection of a bag of instances, i.e. cells, as a series of Bernoulli trials without replacement, with a minimum probability equal to the detection limit of the ground truth, and by asserting a maximum tolerable Type-ll error rate, thereby constraining the probability of a false negative prediction. At the theoretical detection limit of the model, i.e. at least one indicative cell out of a bag of M cells, the probability of detecting at least one indicative cell is therefore modelled as Pr( / > 1: M, DGT) = >DGT)M~X. Asserting the maximum tolerable Type-ll error rate [3 results in the following constraint: Pr( / > 1: M, DGT) > 1 - ?. In this case, the minimum bag size can be calculated as M

[0018] It is important to note that the image segmentation step could also be accomplished by non-neural methods such as thresholding, region-based, or boundary / edge-based methods. Alternatively, the single-cell images could be acquired by a hematology morphology analyzer, which digitizes part or all of the slide and segments individual cells in a single workflow. The first convolutional neural network was advantageously trained to perform pixel-level segmentation of cells, is fed with image tiles from the whole slide image, and outputs an image patch for each identified object (cell). Furthermore, the first network advantageously outputs information about the location of each identified cell in the WSI.

[0019] The first neural network is advantageously a pre-trained model from the StarDist python library [Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers. Cell Detection with Star-convex Polygons. International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI) , Granada, Spain, September 2018.] made available by QuPath [Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. Scientific Reports (2017).]. The model comprises a U-Net architecture [Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)] followed by Non-Maximum Suppression to identify probable objects. The model was trained on Hematoxylin & Eosin-stained cell images from three sources: 1 . Data Science Bowl 2018 dataset BBBC038 provided by the

[0020] Broad Institute [https: / / bbbc.broadinstitute.org / BBBC038]

[0021] 2. MoNuSeg Grand Challenge 2018 dataset [https: / / monuseg.grand-challenge.org / Data / ]

[0022] 3. TNBC dataset from Naylor, et al. 2018 [https: / / zenodo.org / records / 25791 181.

[0023] Alternatively, the segmentation step can advantageously be performed by means of a custom trained U-Net segmentation model [Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)]. This model can be advantageously trained on images of blood, bone marrow, or other cells in a liquid environment, stained with sample-appropriate solutions, for example: May-Grunwald-Giemsa staining for blood or Papanicolau staining for cytology samples. Single cell masks for training this model can be advantageously generated by a pre-trained foundation model such as Segment Anything Model (SAM) or Medical Segment Anything Model (MedSAM) [https: / / openaccess.thecvf.com / content / ICGV2023 / html / Kirillov_Segme nt_Anything_ICCV_2023_paper.html, https: / / www.nature.com / articles / s41467-024-44824-z]. The model is advantageously trained on images from blood smears of healthy individuals, stained with May-Grunwald-Giemsa and imaged on a CellaVision DM96 hematology analyzer at the Hospital Clinic of Barcelona [https: / / futurebloodtesting.org / open-datasets / ]. In a second step, each single-cell image is embedded into a feature vector by means of a second trained neural network. The second neural network advantageously operates on unannotated single-cell images and was trained on over 49 million single-cell images derived from 63 blood smears from unique patients. Each single-cell image was subjected to a set of pre-defined transformations, in order to train the model to maximize the distance between feature vectors stemming from different cells while minimizing the distance between feature vectors from the same cell (contrastive learning). The second neural network has thus advantageously undergone a self-supervised learning. While a self-supervised learning is favoured, the second neural network can also undergo an unsupervised learning or even supervised learning for an auxiliary task such as cell type classification, extracting a layer near the output layer as a feature vector. The second neural network is furthermore advantageously trained to create a rotationally invariant embedding of the single-cell images, and its output is validated by further dimensionality reduction and visualization in a 2-dimensional plane. It is important to note that, because the second neural network is not trained with a specific purpose in mind (for extracting a specific feature), the extracted feature vectors can be used for a variety of third classifier / regressor models downstream. Furthermore, the feature vectors representing cells can be aggregated to form a representation of all or part of the liquid sample for a variety of tasks downstream, such as efficient comparison of liquid samples across patient populations or monitoring liquid samples from a single patient over time.

[0024] Optionally, a step of instance-level augmentation to increase the diversity of the training data for the training of the second neural network, such as rotation, flipping, zooming, color variation, and synthetic background generation, among others on the single-cell images can be applied. Furthermore, an optional step of pre-processing of the single-cell images, for instance color normalization, Gaussian smoothing, style transfer, or background removal, can be foreseen.

[0025] In step c., at least one bag of feature vectors representative for the sample is defined. The number of bags per sample can be advantageously from 1 to 10 but could also be more than 10. The bag size can be from dozens of cells to millions of cells. The label, indicating genomic status of the patient, is transferred from the whole slide image to the bag.

[0026] Before defining the bag in step c., an optional step of feature vector normalization can be applied. This has the advantage of ensuring features within the vector have a similar scale, as well as reducing the risk of so-called vanishing and exploding gradients during back- propagation. A further optional step of classifying the feature vectors to belong to defined cell types can be foreseen. In this case, a subset of feature vectors can be selected and the at least one bag defined in step c. comprises only feature vectors of this subset.

[0027] The third network can have an output head that is a regressor or a classifier. In case of a classifier, the output of the network is thresholded to produce a binary value, a one or zero as indication of the presence of the one or more genomic aberrations in the bag. In case of a regressor, the output of the network is a continuous value. This value can be compared to a predetermined threshold value for determining the label of the bag, or to a previous value in order to determine growth of the aberration-carrying cell population in the tumor. In both cases, the network is advantageously a neural network with a convolutional or transformer backbone and with an attention mechanism, for example a multi-layer perceptron, that operates on the principle of multipleinstance learning.

[0028] For the training of the third neural network, the identity of each bag can be either immutable or mutable with each epoch (bags can be created freshly each epoch), and one or more bags can be used in each epoch. Important to note is that for the training of the third neural network, the bags do not need to be comprised of feature vectors from a single sample, as long as the label stays known, and the number of positive features vectors stays above the detection limit of the model. For example, if the detection limit of the model is 20% and the ground truth says 80% of cells have a biomarker, up to 80%-20% = 60% of cells could be replaced with cells from a different sample, whether positive or negative in genomic status. This is especially advantageous in some cases in which the ground truth value may have been determined on a separately obtained tissue sample. For example, if next-generation sequencing was performed on a solid tissue biopsy composed of 50% tumor cells and a heterozygous aberration was detected at a variant allele frequency of 30%, it can be estimated that 60% of tumor cells will harbor the mutation. If the image being analyzed contains only 20% tumor cells, it is advantageous to either compose each bag with 50% tumor cells, or to adjust the ground truth training value to 20% x 60% = 12%.

[0029] Finally in step e., the sample is classified as indicative of the one or more defined genomic aberrations when an indicative value dependent on the number of bags of feature vectors labelled as indicative meets or exceeds a predetermined threshold value. Important to note is that the number of bags can be one, in particular when the size of the bag is chosen such that the bag is for sure relevant for the sample; meaning that the bag and the sample can only have the same label.

[0030] Advantageously, the indicative value is the percentage of bags of feature vectors labelled as indicative.

[0031] In a first preferred embodiment of the present invention, attention weights obtained from the third neural network are used to label the at least one bag of feature vectors as indicative or nonindicative for the one or more defined genomic aberrations. The attention weights sum to one, allowing for a reliable prediction of the presence of the one or more generic aberrations independently of the number of instances (feature vectors) in the bag. Advantageously, the attention weights are pooled by a learned pooling function to label the at least one bag of feature vectors as indicative or non-indicative for the one or more defined genomic aberrations. The learned pooling function allows uninformative instances to be weighed less heavily, and the function is designed to be symmetric to be invariant to permutation of instances in the bag.

[0032] In another preferred embodiment of the present invention, the method further comprises a step of labelling at least a part of the singlecell images corresponding to the feature vectors of the bag as indicative or non-indicative of the one or more defined genomic aberrations by the third neural network. This allows prediction on the single-cell level for each cell of the bag and to identify which cell shows the one or more genomic aberration. In a further preferred embodiment of the present invention, the method comprises generating a heat-map of the labelled cells present within the at least part of a whole slide image, said heat-map being produced based on spatial location data for each of said labelled cells, as outputted by the first neural network. This allows for identifying on the original whole slide image the presence and position of cells showing the one or more genomic aberration.

[0033] In yet another preferred embodiment of the present invention, the second neural network has been trained by self-supervised learning on single-cell images derived from whole slide images of smear samples that have undergone a defined set of transformations, wherein the second neural network maximizes the spatial distance between feature vectors stemming from different cells and minimize the spatial distance between feature vectors stemming from the same cell. This allows for an optimal training of the second neural network and downstream an optimal detection of the presence of the one or more genomic aberration in the sample.

[0034] In a further preferred embodiment of the present invention, the method further comprises between step a. and b. or before step a. a pre-processing step for the single-cell images, as for example color normalization or Gaussian smoothing. This has the advantage of reducing uninformative variation present in images due to batch effects during sample preparation or slide digitization. Furthermore, between steps c. and d. the feature vectors of the at least one bag are advantageously normalized. This has the advantage of ensuring features within the vector have a similar scale, as well as reducing the risk of so- called vanishing and exploding gradients during back-propagation. In another preferred embodiment of the present invention, the method comprises between steps b. and c. a step of classifying the single-cell images into cell-types and wherein the bag of feature vectors comprises only feature vectors corresponding to one cell-type. This allows to detect the presence of the one or more genomic aberration on only one type of cells. For instance, in the evaluation of peripheral blood samples it may be desired to analyze only cells having a nucleus, such as leukocytes.

[0035] In yet another preferred embodiment of the present invention, the method comprises a step of generating a sub-cellular heat map for indicating the pixel-level position of any shape or texture changes in the cell resulting from the one or more aberrations in the single-cell images. Advantageously, the value of the heatmap is connecting them to a user interface where an expert can look for patterns in the detected cell morphology, either validating known morphological changes or hypothesizing new ones linked to genomic aberrations.

[0036] In another preferred embodiment, the method further comprises a step of bag-level augmentation. This allows for an optimal training of the third neural network without having to acquire a large number of whole slide images. The bag-level augmentation can comprise for instance bag subsampling (random selection of a subset of instances from the bag to create a new bag, while the label of the new bag remains the same as the original bag), bag merging (combination of instances from two or more bags to create a new bag, while the label of the new bag is typically decided based on some logical operation (e.g., AND, OR) applied to the labels of the original bags), instance transformation (applying instance-level transformations (e.g., flipping, rotating) to every instance in the bag), noise addition (addition of some form of noise to the instances within the bag, as for instance Gaussian noise) or feature-level augmentation (modification of the feature vectors in the bag in a manner consistent across all instances).

[0037] According to a second aspect, the present invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the present invention.

[0038] According to a third aspect, the present invention relates to a device or system comprising means for carrying out the method of the present invention.

[0039] According to a fourth aspect, the present invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the present invention.

[0040] In a further aspect, the present invention relates to a method for training a neural network, advantageously an attention-based Multiple Instance Learning network, for deriving from a bag of feature vectors a bag value, wherein from the bag value the at least one bag of feature vectors can be labelled as indicative or non-indicative for the defined genomic aberration. For the training of this neural network, at least parts of whole slide images of smear samples are segmented into single-cell images, advantageously by means of a first trained neural network, wherein the sample is either positive (comprising one or more of genomic aberration) or negative. Each single-cell image is then embedded into a feature vector by means of a second trained neural network, advantageously a trained convolutional neural network or a trained transformer neural network. Subsequently, bags of feature vectors representative of the samples are defined and input to the third neural network and the network is trained until it can produce a bag value from which the presence of the genomic aberration in the bag can be derived.

[0041] In one preferred embodiment, the computer-implemented method for producing a trained model for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample comprises the following steps: a. receiving a training dataset comprising bags of feature vectors, the feature vectors being the result of embedding single-cell images segmented from at least a part of a whole slide image of a smear sample as well as corresponding target bag values; b. initializing a neural network of an attention-based Multiple Instance Learning model with a plurality of network parameters to compute an attention weight for each feature vector of the bags from which predicted bag values can be generated by the attention-based Multiple Instance Learning model; c. training the neural network by:

[0042] I. processing, by the attention-based Multiple Instance Learning model, the plurality of bags of feature vectors to generate predicted bag values; II. calculating, using a loss function, a prediction error based on a comparison between the predicted bag values and the corresponding target bag values;

[0043] III. updating the plurality of network parameters by optimizing the loss function; d. outputting the trained model with the optimized network parameters for use in generating a bag value based on an input bag of feature vectors representing single-cell images of a sample, wherein from the bag value the sample can be labelled as indicative or nonindicative for the defined one or more defined genomic aberrations; wherein the minimal size of the bags of feature vectors is determined by the detection limit of the ground truth, said detection limit of the ground truth being defined as the minimal percentage threshold of cells featuring the said one or more defined genomic aberrations.

[0044] Advantageously, the minimal size of the bags of feature vectors is determined by the detection limit of the ground truth and the yield of the segmentation of the single-cell images.

[0045] In a further aspect, the present invention relates to a method for training a neural network for embedding single-cell images into feature vectors. The network is trained by self-supervised learning on single-cell images derived from whole slide images of smear samples that have undergone a defined set of transformations, wherein the second neural network maximizes the spatial distance between feature vectors stemming from different cells and minimize the spatial distance between feature vectors stemming from the same cell. In a further aspect, the present invention relates to a method for diagnosing a disease associated to a specific biomarker. In one preferred embodiment, the present invention relates in particular to a method for diagnosing a disease, in particular cancer, based on at least a genomic aberration, wherein at least one disease-associated biomarker is detected in an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample by means of the detection method of the present invention. The method of the present invention for the detection of one or more defined genomic aberrations can thus be used to detect a specific genomic aberration-based biomarker to diagnose a disease associated with the presence of this biomarker.

[0046] In a further aspect, the present invention relates to a method for selecting the therapeutic based on the presence or absence of a specific biomarker. In one preferred embodiment, the present invention relates in particular to a method identifying the therapy that will produce the best response, in particular a cancer therapeutic, based on at least a genomic aberration, wherein at least one disease-associated biomarker is detected in an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample by means of the detection method of the present invention. The method of the present invention for the detection of one or more defined genomic aberrations can thus be used to detect a specific genomic aberration-based biomarker to select a therapeutic whose benefit is associated with the presence of this biomarker. In a further aspect, the present invention relates to a method for monitoring or prognosing the evolution of a disease, wherein the computer-implemented method of the present invention is applied on one or more samples of a patient acquired at different point in times, and a prognostic is derived from either the presence of a particular genomic biomarker or the evolution of the number of cells showing one or more genomic aberrations. In particular and in a preferred embodiment, the present invention relates to a method for prognosing the evolution of a disease, in particular cancer, based on genomic aberrations, wherein the number of cells exhibiting a disease-associated biomarker is determined by means of the detection method of the present invention, said determination being performed on images of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, preferably a blood smear sample, from a patient acquired at different points in time. It is important to note that the prognostic can also be made based on only one sample, wherein the prognostic is derived from the presence of a particular genomic biomarker in the sample.

[0047] Finally, in a further aspect, the present invention relates to a method for creating a virtual sample for a patient that can be analysed again and again to detect different biomarkers and / or to diagnose different disease or pathology. As a whole slide image typically comprises thousands to millions of cells, the virtual sample comprises as many feature vectors that can be filtered according to computer metadata, for instance a predicted cell type, for downstream use. The virtual samples can be used for on-demand virtual molecular testing for any number of other biomarkers, cell counts, etc, as needed in the treatment journey of a patient. This method comprises the steps of segmenting at least a part of a whole slide image of a smear sample of a patient into single-cell images, advantageously by means of a first trained convolutional deep neural network and of embedding each single-cell image into a feature vector by means of a second trained neural network, advantageously a trained convolutional neural network or a trained transformer neural network.

[0048] Brief description of the drawings

[0049] - Figure 1 shows the workflow of a preferred embodiment of the present invention;

[0050] - Figure 2 shows an example of a whole slide image of a blood smear sample;

[0051] - Figure 3 shows examples of segmented single-cell images;

[0052] - Figure 4 shows for a selection of single-cell images their corresponding feature vectors and their attention scores;

[0053] - Figure 5 illustrates pre-defined transformations that were applied to single-cell images;

[0054] - Figure 6 shows the performance of the method on a test set of samples;

[0055] - Figures 7 and 8 illustrate the Receiver Operating Characteristic (ROC) curve for two implementations of the method, and

[0056] - Figure 9 illustrates the effect of the bag size on the method performance.

[0057] Detailed description of a preferred embodiment A preferred embodiment of the method of the present invention is presented in figure 1 and results of the method are presented in the other figures.

[0058] Figure 1 illustrates the workflow of a preferred embodiment 100 of the method of the present invention. Solid arrows illustrate the step comprised in this embodiment while dashed arrows represent optional further steps.

[0059] Method 100 comprises a first step 105 where a whole slide image (WSI) of a blood smear sample is segmented by means of a first trained convolutional neural network into single-cell images. It is important to note that the image segmentation step could also be accomplished by non-neural methods in such as thresholding, regionbased, or boundary / edge-based methods.

[0060] Figure 2 shows an example of an actual blood smear sample and Figure 3 shows two examples of such single-cell images.

[0061] The first convolutional neural network was trained to perform pixel-level segmentation of cells and is fed with image tiles from the whole slide image, and outputs an image patch for each identified object (cell). Furthermore, the first network also advantageously outputs information about the location of each identified cell in the WSI. The whole slide images were created by using a Olympus VS200 slide scanner equipped with an automated oil dispenser. After automated application of oil on to the slide, slides were scanned with a 50x oil immersion lens. Three z layers with a 5 micron spacing between them were recorded. The layer with the highest-quality focus was chosen for training and / or inference. The first neural network is a pre-trained model from the StarDist python library [Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers. Cell Detection with Star-convex Polygons. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Granada, Spain, September 2018.] made available by QuPath [Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. Scientific Reports (2017).]. The model, he_heavy_augment.pb, comprises a U-Net architecture [Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)] followed by NonMaximum Suppression to identify probable objects. The model was trained on Hematoxylin & Eosin-stained cell images from three sources:

[0062] 1 . Data Science Bowl 2018 dataset BBBC038 provided by the Broad Institute [https: / / bbbc.broadinstitute.org / BBBC038]

[0063] 2. MoNuSeg Grand Challenge 2018 dataset [https: / / monuseg.grand-challenge.org / Data / ]

[0064] 3. TNBC dataset from Naylor, et al. 2018 [https: / / zenodo.ora / records / 25791 181.

[0065] Future embodiments of the first neural network could advantageously be trained on images of blood, bone marrow, or other cells in a liquid environment, stained with sample-appropriate solutions, for example: May-Grunwald-Giemsa staining for blood or Papanicolau staining for cytology samples.

[0066] In a second step 120, each single-cell image is embedded into a feature vector by means of a second trained convolutional neural network. Examples of embedded vectors are illustrated in Figure 4 for selected single cell images, wherein only the first five and last five components of the vectors are shown as the vectors have a dimension of 2048. It is important to note that this embedding step can also be performed by a neural network with a transformer backbone. In either case, the second neural network operates on unannotated single-cell images and was trained on single-cell images that have undergone a set of pre-defined transformations, in order to maximize the distance between feature vectors stemming from different cells while minimizing the distance between feature vectors from the same cell (contrastive learning). The second neural network has thus undergone a selfsupervised learning. While a self-supervised learning was favoured in this preferred embodiment, the second neural network could have also undergone an unsupervised learning or supervised learning for an auxiliary task. In the present preferred embodiment, the second neural network was furthermore advantageously trained to promote a rotationally invariant embedding of the single-cell images. Moreover, because the second neural network was not trained with a specific purpose in mind (for extracting a specific feature), the extracted feature vectors can be used for a variety of third classifier / regressor models downstream.

[0067] Figure 5 shows examples of pre-defined transformations that were applied to single-cell images to train the second neural network. In this figure, the columns a, b and c show the same cell image having undergone transformations.

[0068] Optionally, a step 1 10 of instance-level augmentation to increase the diversity of the training data for the training of the second neural network, such as rotation, flipping, zooming, color variation, and synthetic background generation, among others on the single-cell images can be applied. Furthermore, an optional step 1 15 of preprocessing of the single-cell images, for instance color normalization, Gaussian smoothing, or background removal can be foreseen.

[0069] It is important to note, that part or the entirety of the feature vectors resulting of step 120 can be stored for further analysis. Indeed, the feature vectors form a virtual patient sample that can be analysed again and again to detect different biomarker and / or to diagnose different disease or pathology. As a WSI typically comprises thousands to millions of cells, the virtual sample comprises as many feature vectors that can be filtered according to computer metadata, for instance a predicted cell type, for downstream use. The virtual samples can be used for on-demand virtual molecular testing for any number of other biomarkers, cell counts, etc, as needed in the treatment journey of a patient. Since the second neural network was not trained with a specific purpose in mind, the extracted feature vectors can be used for a variety of third classifier / regressor models downstream.

[0070] In step 145, at least one bag of feature vectors representative for the sample is defined. The number of bags per sample can be advantageously from 1 to 10 but could also be more than 10. The bag size can be from dozens of cells to millions of cells. However, it is important that the bag size is chosen large enough to ensure that the bag is representative of the sample.

[0071] The minimum bag size can be calculated from the detection limit of the ground truth, the best possible detection limit of the model, and the yield of the segmentation step, said detection limit of the ground truth being defined as the minimal percentage threshold of cells featuring the said one or more defined genomic aberration in the sample. The model's theoretical best detection limit for a bag size of M feature vectors is 1 / M, and the segmentation yield Ys is the percentage of cells in the sample examined by the model. If a limit of detection for the ground truth D, for instance of 10%, is known, the bag size M that must be examined is found by: 1 / (M*Ys) >= D + E, where E is an error allowing for statistical variations in bag sampling. This yields a minimum bag size of M= 1 / ( (D+E)*YS) . On the other hand, the bag size is maximally bounded by number of features that can fit in VRAM.

[0072] Before defining the bag in step 145, an optional step 130 of feature vector normalization can be applied. This has the advantage of ensuring features within the vector have a similar scale, as well as reducing the risk of so-called vanishing and exploding gradients during back-propagation. A further optional step 135 of classifying the feature vectors to belong to defined cell types can be foreseen. In this case, a subset of feature vectors can be selected in optional step 140 and the at least one bag in step defined in step 145 would only comprise feature vectors of this subset.

[0073] In step 155, an attention-based Multiple Instance Learning (MIL) third trained neural network is used to label the at least one bag of feature vectors defined in step 145 as indicative or non-indicative for the defined genomic aberration. The third network can be a regressor or a classifier. In case of a classifier, the output of the network is thresholded to produce a binary value, a one or zero as indication of the presence of the one or more genomic aberrations in the bag. In case of a regressor, the output of the network is a continuous value that can be compared to a predetermined threshold value for determining the label of the bag, or to o previous value in order to determine growth of the aberrationcarrying cell population in the tumor. In both cases, the network is advantageously a neural network with a convolutional or transformer backbone and with an attention mechanism, for example a multi-layer perceptron, that operates on the principle of multiple-instance learning. The third network can by means of a pooling function produce singlecell predictions by inputting a bag containing only a single-cell, i.e., to make a prediction on the presence of one or more genetic aberration on that cell. The trainable pooling function (attention mechanism) is learned from the data (parametrized by a 2-layer neural network) and is independent of number of feature vectors in the bag (weights sum to one). It is advantageously formulated as a symmetric function to remove sequential dependence (is permutation-invariant with respect to cell order) and includes an activation function that is designed to highlight distinctions between positive and negative feature vectors.

[0074] Advantageously, it includes a gated tanh activation function that is more expressive than a typical tanh activation by introducing nonlinearity (gating function) interpretable by assigning weights to each feature vector in the bag according to how informative that feature vector is. Figure 4 shows in the third column an example of attention scores (weights) for selected single-cell images.

[0075] For the training of the third neural network, the identity of each bag can be either immutable or mutable with each epoch (bags can be created freshly each epoch), and one or more bags can be used in each epoch. Important to note that for the training of the third neural network, the bags do not need to be comprised of feature vectors from a single sample, as long as the label stays known, and the number of positive features vectors stays above the detection limit. For example, in the case of a classifier, if detection limit of the model is 20% and the ground truth says 80% of cells have a biomarker, up to 80%-20% = 60% of cells could be replaced with cells from a different sample, whether positive or negative in genomic status.

[0076] In order to efficiently train the third neural network, an optional step 150 of bag-level augmentation can be foreseen. The bag-level augmentation can comprise for instance bag subsampling (random selection of a subset of instances from the bag to create a new bag, while the label of the new bag remains the same as the original bag), bag merging (combination of instances from two or more bags to create a new bag, while the label of the new bag is typically decided based on some logical operation (e.g., AND, OR) applied to the labels of the original bags), instance transformation (applying instance-level transformations (e.g., flipping, rotating) to every instance in the bag), noise addition (addition of some form of noise to the instances within the bag, as for instance Gaussian noise) or feature-level augmentation (modification of the feature vectors in the bag in a manner consistent across all instances).

[0077] Finally in step 160, the sample is classified as indicative of the one or more defined genomic aberrations when an indicative value dependent on the number of bags of feature vectors labelled as indicative meets or exceeds a predetermined threshold value. Important to note is that the number of bags can be one, in particular when the size of the bag is chosen such that the bag is for sure relevant for the sample; meaning that the bag and the sample can only have the same label. 19

[0078] As mentioned above, the third network can be used to make single-cell predictions on the presence of one or more genomic aberration on each cell. This information combined with information on the localization of each map that can be outputted by the first network allows for the creation of a heat map that overlies the single-cell prediction on the original whole slide image.

[0079] The method described for the present invention was tested for the detection of genomic aberration due to the presence of the TP53+ biomarker that is important to prognosis and the selection of therapeutic strategy for patients with chronic lymphocytic leukaemia. The third neural network was trained on a sample set including 34 TP53-negative and 10 TP53-positive chronic lymphocytic leukemia samples. Figure 6 shows the performance of the method on a test set of samples from patients not seen during model training that included 13 negative and 5 positive samples. Various bag compositions and model architectures were tested, where a label like Attention(2048, 512, 1 ) (-1 / 700k, 10) indicates that a non-gated attention mechanism with 512 neurons in a single hidden layer was used, taking in feature vectors of length 2048; 10 bags were composed without replacement, with the size of each bag being the lesser of 700,000 or the number of cells in the sample divided by the number of bags. The graph shows the performance of this method in avoiding false positive errors, i.e. precision (true positives divided by true positives plus false positives); in identifying patients with the genomic aberration, i.e. recall (true positives divided by true positives plus false negatives); in the rate of correct classification, i.e. accuracy (correct predictions divided by total number of predictions), and in class distribution-adjusted accuracy, i.e. Fl -score (twice the product of precision and recall divided by the sum of precision and recall). As one can see, the present method is better than random classification of the samples and achieves a sensitivity value (number of true positives divided by the sum of the number true positives and false negatives) of 0.8, a specificity value (number of true negatives divided by the sum of number of true negatives plus false positives) of 0.69, an accuracy value (sum of true positives plus true negatives divided by the sum of true positives plus false positives plus true negatives plus false negatives) of 0.72. This means that thanks to the present invention 80% of the positive samples (patients) could be identified while saving state of the art molecular tests for 69% of patients who do not have the aberration.

[0080] Figures 7 and 8 illustrate the Receiver Operating Characteristic (ROC) curve for two implementations of the method. The ROC curve plots the True Positive rate against the False Positive rate. An Area Under the Curve (AUC) greater than 0.5 indicates a model is better than random, and models with an AUC close to 1 are highly performant. The attention mechanism referenced in Figure 7 comprises a linear combination of learned weights subjected to a non-linear activation function such as the hyperbolic tangent (tanh). In Figure 8, the gated attention mechanism includes an additional non-linearity that gates the flow of information. The amount of information flowing through each activation neuron is gated according to the sigmoid distribution bounded by [0,1 ]. This additional nonlinearity is designed to further improve the expressiveness of the attention pooling function due to the linearity of the tanh activation function between the values of 0 and 1 .

[0081] In order to illustrate the importance of a minimum bag size, the performance of the present method was evaluated for bags consisting of 5,000 cells, 94,000 cells, or 517,000 cells on average. In each case, the same model architecture incorporating an attention mechanism was used for inference on a test set of 13 negative and 5 positive samples. As illustrated in Figure 9, the improvement in recall (true positives divided by true positives plus false negatives) as bag sizes increase demonstrates that a larger bag size reduces the rate of false negative predictions, thereby improving model sensitivity. This data makes it clear that calculation of the minimum bag size is an important step to identifying and avoiding false negatives, especially when hardware constraints limit the maximum bag size. The tests were performed on an Nvidia Tesla VI 00 GPU with 16

[0082] GB HBM2 GPU memory.

Claims

Claims1 . Computer-implemented method for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample, comprising the following steps: a. Segment at least a part of a whole slide image of the smear sample into single-cell images, advantageously by means of a first trained convolutional deep neural network; b. Embed each single-cell image into a feature vector by means of a second trained neural network, advantageously a trained convolutional neural network or a trained transformer neural network; c. Define at least one bag of feature vectors representative for the sample; d. Derive from the at least one bag of feature vectors a bag value by means of an attention-based Multiple Instance Learning third trained neural network, wherein from the bag value the at least one bag of feature vectors is labelled as indicative or non-indicative for the defined genomic aberration; e. Classify the sample as indicative of the one or more defined genomic aberrations when an indicative value dependent on the number of bags of feature vectors labelled as indicative meets or exceeds a predetermined threshold value;wherein the minimal size of the at least one bag is determined based on the detection limit of the ground truth, said detection limit of the ground truth being defined as the minimal percentage threshold of cells featuring the said one or more defined genomic aberrations.

2. Computer-implemented method according to claim 1 wherein the minimal size of the at least one bag is determined based on the detection limit of the ground truth and the yield of the segmentation step a.

3. Computer-implemented method according to claim 2 wherein the minimal size of the at least one bag is determined based on the detection limit of the ground truth, the yield of the segmentation step a. and and an error value reflecting the statistical variation in bag sampling.

4. Computer-implemented method according to any one of the claim 1 to 3, wherein attention weights obtained from the third neural network are used to label the at least one bag of feature vectors as indicative or non-indicative for the one or more defined genomic aberrations.

5. Computer-implemented method according to claim 4, wherein the attention weights are pooled by a trained pooling function to label the at least one bag of feature vectors as indicative or nonindicative for the one or more defined genomic aberrations.

6. Computer-implement method according to any one of the preceding claims, further comprising a step of labelling at least a part of the single-cell images corresponding to the feature vectors of the bag asindicative or non-indicative of the one or more defined genomic aberrations by the third neural network.

7. Computer-implemented method according to claim 6, wherein the method comprises generating a heat-map of the labelled cells present within the at least part of a whole slide image, said heatmap being produced based on spatial location data for each of said labelled cells, as outputted by the first neural network.

8. Computer-implemented method according to any one of the preceding claims, wherein the second neural network has been trained by self-supervised learning on single-cell images derived from whole slide images of smear samples that have undergone defined set of transformations, wherein the second neural network maximizes the spatial distance between feature vectors stemming from different cells and minimize the spatial distance between feature vectors stemming from the same cell.

9. Computer-implemented method according to any one of the preceding claims, further comprising between step a. and b. or before step a. a pre-processing step for the single-cell images, as for example color normalization or Gaussian smoothing.

10. Computer-implemented method according to any one of the preceding claims wherein between steps b. and d. the feature vectors of the at least one bag are normalized.1 1 . Computer-implemented method according to any one of the preceding claims, wherein the method comprises between steps b. and c. a step of classifying the single-cell images into cell-types andwherein the bag of feature vectors comprises only feature vectors corresponding to one cell-type.

12. Computer-implemented Method according to any one of the preceding claims wherein the indicative value is the percentage of bags of feature vectors labelled as indicative.

13. Computer-implemented Method according to any of the preceding claims further comprising a step of generating a sub-cellular heat map for indicating the pixel-level position of any shape or texture changes in the cell resulting from the one or more aberrations in the single-cell images.

14. Computer-implemented Method according to any one of the preceding claims further comprising a step of bag-level augmentation.

15. Computer-implemented Method according to any one of the preceding claims, wherein the segmentation step is performed on a whole slide image of the sample.

16. Computer-implemented method for producing a trained model for detecting one or more defined genomic aberrations from an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample comprising: a. receiving a training dataset comprising bags of feature vectors, the feature vectors being the result of embedding single-cellimages segmented from at least a part of a whole slide image of a smear sample as well as corresponding target bag values; b. initializing a neural network of an attention-based Multiple Instance Learning model with a plurality of network parameters to compute an attention weight for each feature vector of the bags from which predicted bag values can be generated by the attention-based Multiple Instance Learning model; c. training the neural network by:I. processing, by the attention-based Multiple Instance Learning model, the plurality of bags of feature vectors to generate predicted bag values;II. calculating, using a loss function, a prediction error based on a comparison between the predicted bag values and the corresponding target bag values;III. updating the plurality of network parameters by optimizing the loss function; d. outputting the trained model with the optimized network parameters for use in generating a bag value based on an input bag of feature vectors representing single-cell images of a sample, wherein from the bag value the sample can be labelled as indicative or nonindicative for the defined one or more defined genomic aberrations; wherein the minimal size of the bags of feature vectors is determined by the detection limit of the ground truth, said detection limitof the ground truth being defined os the minimal percentage threshold of cells featuring the said one or more defined genomic aberrations.

17. Computer-implemented method according to claim 16 wherein the minimal size of the bags is determined by the detection limit of the ground truth and the yield of the segmentation of the single-cell images.

18. Method for diagnosing a disease, in particular cancer, based on at least a genomic aberration, wherein at least one disease- associated biomarker is detected in an image of a body liquid, bone marrow or cytological sample, for instance a cytology smear sample, in particular a blood smear sample by means of the method of any one the claims 1 to 15.

19. Method for prognosing the evolution of a disease, in particular cancer, based on genomic aberrations, wherein the number of cells exhibiting a disease-associated biomarker is determined by means of the method of any one the claims 6 to 15, said determination being performed on images of one or more samples of a body liquid, bone marrow or cytological samples, for instance smear samples, preferably blood smear samples from a patient acquired at different points in time and the prognostic is derived from either the presence of a particular genomic biomarker or the evolution of the number of cells showing one or more genomic aberrations.

20. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer tocarry out the method according to any of the claims 1 to 15 or 16 to 17 or 18 or 19. 21 . A system comprising means for carrying out the method according to any of the claims 1 to 15 or 16 to 17 or 18 or 19.

22. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any of the 1 to 15 or 16 to 17 or 18 or 19.

Images