METHOD AND SYSTEM FOR IDENTIFYING BACTERIA IN POLYCULTURES ON MICROSCOPIC IMAGES USING DEEP LEARNING

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to identification of polyculture bacteria on microscopic images using deep learning.BACKGROUND

[0002] Typical methods for microbiology diagnostics aimed at identifying pathogenic microorganisms may take several days or weeks. A classical diagnostics procedure includes collecting a biological sample and cultivation of the sample on artificial medium with specific substrates (temperature, oxygen level, time of cultivation). Initially, the microorganisms are assessed by determining the cultivated colonies, and also based on Gram stained microscopic samples. In view of visual similarities of various species, it is not possible to identify a particular species of bacteria or yeast-like fungi. Therefore, further procedures must be carried out, such as isolating colonies or microorganisms, separating them on selective and differential agar medium, repeating the incubation and identifying species based on tests based on biochemical reactions and / or serological tests. In that procedure, it is possible that a mix of species is cultivated, therefore the procedure may be even longer.

[0003] There are also known other methods, such as methods based on detection of nucleotide acids (mostly PCR (polymerase chain reaction) or real-time PCR tests) or detection of unique microorganism proteins (Matrix Assisted Laser Desorption / Ionization Time-of-Flight (MALDI-TOF) Mass Spectrometry). These methods are relatively costly and require specific equipment.

[0004] There are also known methods that use machine learning techniques to classify bacteria. For example, such methods are disclosed in the following publications: PALMA, Susana ICJ, et al. Machine learning for the meta-analyses of microbial pathogens' volatile signatures. Scientific reports, 2018, 8.1: 1-15. QU, Kaiyang, et al. Application of machine learning in microbiology. Frontiers in microbiology, 2019, 10: 827. PEIFFER-SMADJA, Nathan, et al. Machine learning in the clinical microbiology laboratory: has the time come for routine practice?. Clinical Microbiology and Infection, 2020, 26.10: 1300-1309. ZHANG, Yang, et al. Deep Learning for Imaging and Detection of Microorganisms. Trends in Microbiology, 2021. ZHENG, Lingling, et al. Machine Learning Algorithms Identify Pathogen-Specific Biomarkers of Clinical and Metabolomic Characteristics in Septic Patients with Bacterial Infections. BioMed research international, 2020, 2020. GOODSWEN, Stephen J., et al. Machine learning and applications in microbiology. FEMS Microbiology Reviews, 2021.

[0005] However, each these prior art methods suffers from at least one of the following disadvantages: requires costly data acquisition processes, determines a limited number of bacteria or does not return an explanation of the prediction result.

[0006] JP2020123044 discloses classification means for classifying a bacterium from an image of the bacterium by learning the relationship between the result of the classification of the bacterium performed by a person based on the image of the bacterium contained in the image of the Gram-stained sample and the characteristics of the image of the bacterium.

[0007] US2021130868A1 discloses a method for deep learning video microscopy-based antimicrobial susceptibility testing of a bacterial strain in a sample by acquiring image sequences of individual bacterial cells of the bacterial strain in a subject sample before, during, and after exposure to each antibiotic at different concentrations.

[0008] CN112861627A discloses a method and system for identifying types of pathogenic bacteria based on a microscopic hyperspectral technology. The identification method comprises the following steps: 1) acquiring original hyperspectral data of a pathogenic bacterium sample to be detected; 2) carrying out whitening treatment; 3) carrying out principal component analysis; 4) inputting the principal component image into a pre-established and trained convolutional neural network model, wherein according to the convolutional neural network model, various pathogenic bacteria samples of known types are taken, the previous steps are referred in sequence, and then a model is established and trained based on a principal component image; and 5) outputting a pathogenic bacterium species identification result through the convolutional neural network model.

[0009] Wahid et al., in their article titled "Performance Analysis of Machine Learning Techniques for Microscopic Bacteria Image Classification," disclose a method for the classification of bacteria in microscopic images. This method employs a convolutional neural network to generate a representations matrix of the microscopic images, which is subsequently flattened into a vector form. This vector serves as the input for species classifiers. It is important to note, however, that the disclosed method only permits the classification of bacterial species in images containing bacteria of a single species.

[0010] Similarly, Borowa et al., in their article titled "Deep Learning Classification of Bacteria Clones Explained by Persistence Homology," disclose a method for the classification of bacterial clones in microscopic images. However, this method only facilitates the identification of different clones of bacteria belonging to a single species.SUMMARY OF THE INVENTION

[0011] In view of the drawbacks of the prior art, there is a need to provide alternative methods for identification of polyculture bacteria.

[0012] The invention is defined by the set of appended claims. The present invention is based on deep multiple instances learning (multi-MIL) to classify and identify particular bacteria on a microscopic image of a specimen containing different bacteria species.

[0013] The object of the invention is a computer-implemented method and system for identifying polyculture bacteria on a microscopic image, according to the appended claims.

[0014] The method of the invention allows to classify bacteria with a high accuracy and allows to shorten the time necessary to identify the microorganisms.

[0015] These and other features, aspects and advantages of the invention will become better understood with reference to the following drawings, descriptions and claims.BRIEF DESCRIPTION OF DRAWINGS

[0016] Various embodiments are herein described, by way of example only, with reference to the accompanying drawings, wherein: Fig. 1 shows schematically components and data of a patches processing module for generating patch features vectors for patches of input image; Fig. 2 shows schematically a method for training the patches processing module for generating patch features vectors for patches of input image; Fig. 3 shows an example input image with many polyculture bacteria; Fig. 4A shows a first example of a patch of the input image; Fig. 4B shows a second example of a patch of the input image; Fig. 4C shows a third example of a patch of the input image, Fig. 5 shows schematically components and data of a module for generating a prediction for an input image; Fig. 6 shows schematically a method for training of the module for generating a prediction for an input image; Fig. 7 shows schematically a method for identifying the polyculture bacteria; Fig. 8 shows a structure of a computer-implemented system for handling the methods presented herein. DETAILED DESCRIPTION

[0017] Fig. 1 shows schematically components and data of a patches processing module 100 for generating patch features vectors for patches of input image and Fig. 2 shows a method for training that module.

[0018] In step 110 an input image 111 (as shown for example in Fig. 3) which may comprise multiple polyculture bacteria is divided into patches 112 (examples 112A, 112B, 112C of different patches are shown in Figs. 4A-4C). For example, each patch may have a size of 250x250 pixels, wherein each pixel is represented by a color value in RGB color space. The patches may partially overlap each other (for example, by 50% of their width and height) or may be adjacent to each other, depending on the desired degree of detail of processing data.

[0019] The step 110 of dividing the input image may further contain pre-processing of the input image, such as reduction of the size of the image.

[0020] Moreover, step 110 may further comprise pre-processing of the patches, for example by rejecting patches that are less useful for classification than others. For example, a standard deviation (std) between average values of colours in each channel of the RGB color space is calculated for each patch. Patches having low std value (for example, lower than 2.25, which indicates a patch which is almost empty, such as a patch 112A as shown in Fig. 4A) or high value (for example, higher than 13.5, which indicates a patch that contains lots of bacteria overlapping each other, wherein individual bacteria are undetectable, such as a patch 112B shown in Fig. 4B). Only patches having moderate std values (for example, between 2.25 and 13.5, which indicates that individual bacteria can be detected, such as a patch 112C shown in Fig. 4C) are selected for processing in further steps. Next, each selected patch may be subject to normalization, for example by means of an average and standard deviation determined for a set of random 10000 patches of other input images that have std values in the range of 2,25<std<13,5. For example, the normalized value can be calculated as ratio of a [difference between the current value and a mean value] and the std value, i.e. (current-mean) / std.

[0021] A set of k patches (wherein k is the number of patches in the set) output from step 110 is input in step 120 to a convolutional layers 121 of a patches processing neural network, in order to generate representation of patches. For example, the patches processing neural network may be ResNet (such as ResNet-50, ResNeXt-50), VGG, DenseNet, InceptioNet or SwinTransformer. The patches processing neural network generates representations of patches in a form of a matrix of representations. The output is represented as a set of k matrices of representations of patches, wherein each matrix has a size of HxWxD, wherein H and W are sizes of representation of patch within the patches processing neural network and D is a number of channels. For example, H and W may have a value of 8, and D can be 512. The values can depend on the architecture of neural network used.

[0022] In step 130 the set of k matrices of representations of patches is input to a patch data aggregator 131 that is configured to generate k patch features vectors h 1 , h 2 , ..., h k (i.e. one patch features vector h per each of the k patches), for example by means of 2D average pooling. Each patch features vector hi, h 2 , ..., h k has a size of D, wherein each value within the patch features vector hi, h 2 , ..., h k represents high-level features of a patch considered as significant. Therefore, the aggregator outputs the patch features vectors h 1 , h 2 , ..., h k (which may be also called a hidden representation for patches).

[0023] The set of k patch features vectors hi, h 2 , ..., h k is then input in step 140 to classification layers 141 of a features vectors processing neural network, in order to generate k sets of N probabilities y 1 , y 2 , ..., y N that a particular patch represents a particular species of bacteria. For example, the features vectors processing neural network may be a ResNet-18 network with a single linear layer that is configured to change a k x 512 vector to a k x number_of_classes vector. The classification layers are preferably linear layers. Therefore, step 140 performs classification of bacteria on the patches.

[0024] During the training, a loss function is used to correct network parameters. For this purpose, the mechanism of back propagation of error is used.

[0025] The procedure is performed for multiple input images containing one or more species of N bacteria to be classified. Namely, some input images used for training may contain only bacteria of a single species, some other input images may contain bacteria of two or more species. Moreover, for each imaged specimen a plurality of images can be prepared, each imaged from a different viewpoint.

[0026] As a result, the module 100 is trained to generate the patch features vectors h 1 , h 2 , ..., h k for patches of input image containing one or more species of bacteria to be classified and corresponding probabilities y 1 , y 2 , ..., y N that a particular patch represents a particular species of bacteria.

[0027] Fig. 5 shows schematically components and data of a module 500 for generating a prediction for an input image and Fig. 6 shows a procedure for training that module.

[0028] In step 510, patch features vectors h 1 , h 2 , ..., h k for patches of an image, as obtained at the output of step 130, are input to selected N features vectors aggregators 511-513. Each of the aggregators 511-513 is configured for a distinct species of all of the N species to be classified. The patch features vectors hi, h 2 , ..., h k for patches are directed to those aggregators 511-513. Each features vectors aggregator 511-513 calculates a species features vector h 1< , h 2< , ..., h N< for bacteria species for the processed image (i.e. each aggregator 511-513 calculates a single vector for the species of bacteria corresponding to that aggregator 511-513), having the same dimension D as the patch features vectors hi, h 2 , ..., h k for patches. It is calculated as a weighted average of the patch features vectors h 1 , h 2 , ..., h k for patches that have been input to that aggregator 511-513. For this purpose, one can use Attention-based Multiple Instance Learning or other methods returning weights for the features vectors h 1 , h 2 , ..., h k .

[0029] A skilled person will realize that step 510 can be also performed by alternative techniques, such as aggregating by average, Attention Pooling, Self-Attention Attention Pooling, Loss Attention, TransMIL, Swin Transformer, or Vision Transformer with Focal Attention.

[0030] In step 520, each of the species features vectors h 1< , h 2< , ..., h N< for bacteria species is input to a corresponding species classifier 521-523 configured to generate a probability y 1< , y 2< , ..., y N< that the image contains the particular species of bacteria. For example, each of the species classifier 521-523 can be realized as a shallow neural network having a low number (e.g. five) of linear hidden layers.

[0031] A loss function is used to correct network parameters during training. For this purpose, the mechanism of back propagation of error is used.

[0032] Fig. 7 shows schematically a method for identifying the polyculture bacteria. First, in step 710, the system is trained with manually-labelled images as described above with respect to methods of Figs. 2 and 6 in order to train the neural network modules 121, 141, 511-513, 521-523.

[0033] Next, in step 720, an input image 111 to be analyzed is provided as input and divided to patches 112, with pre-processing as described with reference to step 110.

[0034] Next, in step 730, the patches 112 are input to convolutional layers 121 of the patches processing neural network to generate the set of k matrices of representations of patches, as described with reference to step 120.

[0035] In step 740, the set of k matrices of representations of patches is input to the patch data aggregator 131 to generate patch features vectors h 1 , h 2 , ..., h k for patches, as described with reference to step 130.

[0036] In step 750, the patch features vectors h 1 , h 2 , ..., h k for patches of the image are input to selected N features vectors aggregators 511-513 in order to calculate species features vectors h 1< , h 2< , ..., h N< corresponding to bacteria species for the whole processed image. The procedure is similar as in 510.

[0037] Next, in step 760, the species features vectors h 1< , h 2< , ..., h N< for bacteria species for the processed image are input to the species classifiers 521-523 in order to generate a probability y 1< , y 2< , ..., y N< that the image contains the particular species of bacteria - which is provided as a result of the identification of the polyculture bacteria within the input image.

[0038] Fig. 8 shows a structure of a computer-implemented system 800 for handling the methods presented herein. The system may include at least one non-transitory processor-readable storage medium 810 that stores at least one of processor-executable instructions 815 or data 816 and at least one processor 820 communicably coupled to the at least one non-transitory processor-readable storage medium 810. The at least one processor 820 is configured to (by executing the instructions 815) perform the steps of any of the methods presented in Figs. 2, 6 and 7 and to emulate modules of Figs. 1 and 5.Example embodiment

[0039] An example system for identifying four types of bacteria has been implemented as follows.

[0040] The system was trained to identify N=4 species of bacteria: Escherichia coli, Staphylococcus aureus, Neisseria gonorrhoeae and Lactobacillus plantarum (these species were selected due to the differences in morphology and classical Gram staining method). In order to simulate a natural environment, wherein a single specimen might contain different species, also belonging to different types, the above-mentioned bacteria species were applied together on a microscopic slide (specimens of 12 combinations of species plus specimens of individual species). Next, the specimens were fixed and stained with a typical Grama stain. 12 combinations of species were prepared (including individual species). Two different specimens of each kind were imaged, each specimen imaged by 10 images taken from different viewpoints. In total, 240 images were obtained for combinations of species and 80 images for individual species.

[0041] The images were then divided into 200 patches each, and each patch was manually described to identify all species which were present on the initial image. Namely for an image containing a single species each patch was described as containing that particular species, and for an image containing two species each patch was described as containing that particular species, regardless of which species were present in that particular patch. As a result, a database of 60400 image patches with descriptions was obtained as a training set. The neural network modules 121, 141, 511-513, 521-523 were then trained by the training set according to the procedures of Figs. 2 and 6.

[0042] Such trained system is ready to process images in order to classify the four bacteria species according to the procedure of Fig. 7.

Claims

1. A computer-implemented method for identifying polyculture bacteria on a microscopic image, the method comprising: - providing (710) a computer system comprising: - a patches processing neural network with convolutional layers (121) configured to receive patches of the microscopic image and generate a representation of patches in a form of a set of three dimensional matrices of representations, wherein each matrix has a size of HxWxD, wherein H and W are sizes of representation of patches within the patches processing neural network and D is a number of channels; - a patch data aggregator (131) configured to receive the representation of patches and to generate patch features vectors (h1, h2, ..., hk) for the patches, wherein each value within the patch features vectors represents high-level features of the patch, wherein patch features vectors have a size D which is equal to number of channels from representation of patch; - receiving (720) the microscopic image (111) to be analyzed and dividing the microscopic image (111) to be analyzed to patches (112); - inputting (730) the patches (112) to the convolutional layers (121) of the patches processing neural network to generate a set of matrices of representations of patches; - inputting (740) the set of matrices of representations of patches to the patch data aggregator (131) to generate the patch features vectors (h1, h2, ..., hk) for patches; characterized in that, - the computer system further comprises: - a plurality of features vectors aggregators (511-513) configured to receive patch features vectors (h1, h2, ..., hk) and to calculate species features vectors (h1, h2, ..., hN) corresponding to bacteria species for the image, wherein each of the features vectors aggregators (511-513) is configured for a distinct species of all of the species to be classified, wherein species features vectors have a size D which is equal to size of patch features vectors, wherein species features vectors are calculated as a weighted average of the patch features vectors; and - a plurality of species classifiers (521-523) configured to receive the species features vectors (h1, h2, ..., hN) corresponding to bacteria species for the image and to generate a probability (y1, y2, ..., yN) that the image contains the particular species of bacteria; and - the method further comprises steps of: inputting (750) the patch features vectors (h1, h2, ..., hk) for patches to the features vectors aggregators (511-513) to calculate species features vectors (h1, h2, ..., hN) corresponding to bacteria species for the image; and - inputting (760) the species features vectors (h1, h2, ..., hN) corresponding to bacteria species for the image to the species classifiers (521-523) to generate a probability (y1, y2, ..., yN) that the image contains the particular species of bacteria.

2. The method according to claim 1, wherein the computer system further comprises a features vectors processing neural network with classification layers (141) configured to receive the patch features vectors (h1, h2, ..., hk) for patches and to generate probabilities (y1, y2, ..., yN) that a particular patch represents a particular species of bacteria; and wherein the system is pre-trained by: - providing (110) a training input microscopic image (111) representing known polyculture bacteria and dividing the training input microscopic image to patches (112); - inputting (120) the patches (112) to the convolutional layers (121) of the patches processing neural network to generate a set of matrices of representations of patches; - inputting (130) the set of matrices of representations of patches to the patch data aggregator (131) to generate patch features vectors (h1, h2, ..., hk) for patches; - inputting (140) the patch features vectors (h1, h2, ..., hk) for patches to classification layers (141) of the features vectors processing neural network to generate probabilities (y1, y2, ..., yN) that a particular patch represents a particular species of bacteria; - inputting (510) the patch features vectors (h1, h2, ..., hk) for patches to the features vectors aggregators (511-513) to calculate the species features vectors (h1, h2, ..., hN ) corresponding to bacteria species for the training input microscopic image; and - inputting (520) the species features vectors (h1, h2, ..., hN ) corresponding to bacteria species for the training input microscopic image to the corresponding species classifier (521-523) to generate a probability (y1, y2, ..., yN) thatthe training input microscopic image contains the particular species of bacteria; - while using a mechanism of back propagation of error during the training process.

3. The method according to any of previous claims, further comprising pre-processing (110, 720) the microscopic image to calculate a standard deviation (std) between average values of colours in each channel of the RGB color space for each patch and rejecting patches having a standard deviation value outside a predefined threshold range.

4. A computer-implemented system (800) configured to perform the method according to any of previous claims.