Method and system for characterizing microorganisms contained in complex samples

Abstract

The disclosed method for classifying microorganisms contained in a sample comprises: preparing a slide of the sample; acquiring at least one digital image of the slide; and implementing by computer a model for predicting a class of microorganism from the acquired image. According to the invention, the image is subdivided into sub-images and each sub-image is subdivided into image patches, and: A. applying to each image patch a microorganism feature extractor constituting a convolutional part of a first convolutional neural network trained on image patches individually annotated by at least one class; B. applying to each sub-image a second neural network connected to the extractor, the second neural network comprising an upstream pooling layer and one or more downstream layers, the downstream layers comprising a layer for predicting at least one class and trained globally on training sub-images; and C. for the acquired image: F.a. computing a feature vector computed for the sub-image; F.b. applying a model for predicting at least one class of microorganism.

Description

[0001] Invention Field This invention relates to the field of microbial analysis, and more particularly to the automated characterization of microorganisms contained in complex samples (e.g., blood). In a non-limiting sense, this invention is applicable to the Gram-related, morphological, and aggregate characterization of bacteria, yeasts, and fungi in infected patients or animals, especially those suspected of having sepsis.

[0002] Existing technology The field of microbial analysis faces numerous challenges whenever it deviates from conventional techniques based on cultures or media (e.g., on petri dishes). The detection of sepsis caused by bloodborne bacterial infections is a well-known example. The presence of pathogenic bacteria in the blood can lead to patient death, making early detection crucial. Unfortunately, in the early stages, the amount of bacteria is not only minimal, but these bacteria are also embedded in a complex matrix containing white blood cells, red blood cells, platelets, and many other elements significantly larger in size and number, making them currently difficult or even impossible to detect. Therefore, for any patient suspected of having sepsis, a blood culture is initiated—the step of growing the bacteria present in the blood by adding a nutrient medium selected to accelerate their development as much as possible. The aim of this growth is to multiply the bacteria as quickly as possible so that they can be detected. Unfortunately, given the severity of the infection, blood culture remains an excessively slow process.

[0003] The complexity of samples also presents challenges for microbial analysis in what would otherwise be routine laboratory applications. This is particularly true for Gram characterization of bacteria, regardless of sample complexity. In fact, Gram characterization of bacteria requires significant human expertise, as it relies on the microscopic analysis of Gram slides by qualified technicians. However, many, if not most, analytical laboratories have experienced a significant loss of this expertise, leading to a decline in quality and even the practical discontinuation of this analysis. Nevertheless, this analysis, performed quickly upon sample receipt, allows clinicians to make sometimes decisive antibiotic treatment decisions very rapidly.

[0004] The spread of machine learning techniques, particularly deep learning based on neural networks, is promising. In fact, these techniques have been successfully applied to complex images, such as lung tumor detection. Similar progress is expected in microbiology: characterizing the microorganisms present in samples based on sample images. However, this prospect faces resource barriers, making it a significant challenge for any organization, regardless of size, to develop automated characterization solutions based on artificial intelligence. Indeed, due to the size of bacteria, their abundance in early stages, and the complexity of samples, imaging of significant surface areas at very high resolution is required. In practice, this is done by combining multiple raw images (typically each with millions of pixels, and not necessarily overlapping or even contiguous) into a single composite image, hereinafter referred to as an "image". The resulting image typically ranges from tens of millions to billions of pixels. Even using convolutional neural network techniques to reduce the problem's dimensionality, the final dimensionality remains so large that learning a traditional architecture would require an enormous number of labeled images—undoubtedly millions. Beyond this astronomical number, annotation cannot be performed by non-experts as is done with models on Google Zoo or by subcontracting the annotation of learned models on a dedicated platform. However, as mentioned above in the case of characterizing Gram, experts are lacking.

[0005] Therefore, to date, developing AI-based automated characterization tools remains a challenge in the field of microbial analysis, particularly in in vitro diagnostics (“IVD”). Summary of the Invention

[0006] The purpose of this invention is to propose a method for automatically characterizing microorganisms present in samples based on artificial intelligence analysis of very high-resolution images. This analysis is implemented by a deep learning architecture trained on a reduced learning set and allows for effective diagnosis, particularly in vitro diagnostics.

[0007] Therefore, the object of the present invention is a method for classifying microorganisms contained in a sample from multiple microbial categories, the method comprising: A. Preparing glass slides, particularly microscope slides, including coating the sample onto the glass slide; B. Acquire at least one digital image of the slide with micrometer or submicrometer resolution; C. A computer-based model used to predict microbial categories based on acquired images.

[0008] According to the present invention, the image is subdivided into multiple sub-images and each sub-image is further subdivided into image patches, and the application of a prediction model includes: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing microorganisms individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers, the downstream layers including layers predicting at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled with at least one category; and F. Regarding the acquired images: Fa calculates the feature vector for a sub-image; Fb applies a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

[0009] In other words, this invention proposes an artificial intelligence architecture with multiple specific stages, characterized by increasingly weaker annotations along each stage. This architecture is trained using only a few hundred high-resolution sub-images, which are subdivided into smaller image patches (e.g., 256 pixels × 256 pixels or 224 pixels × 224 pixels), for which a very small percentage (e.g., less than 5%) is heavily annotated, thereby achieving an overall prediction accuracy of over 90%.

[0010] According to one embodiment, the feature vector is a score distribution computed for sub-images. According to one embodiment, each training image is subdivided into sub-images and each sub-image is further subdivided into image patches, and less than 50% of the image patches in the training images are labeled, the labeled image patches forming the training image patches of the first convolutional network. Specifically, less than 10% of the image patches in the training images are labeled. According to one embodiment, the training of the first and second convolutional networks and the prediction model is configured to obtain a prediction macrospecificity greater than or equal to 90%. According to one embodiment, the prediction model in step F is a "random forest" model. According to one embodiment, an upstream pooling layer is associated with an attention layer configured to apply weights to the output of each extractor. According to one embodiment, the second network is a MIL-CNN network trained via instance batches consisting of training image patches.

[0011] According to one implementation scheme, the feature vector for step Fa includes, for each category: - The maximum score in the sub-image; and / or - The Xth percentile of the score in the sub-image, where X is greater than or equal to 90%, preferably equal to 95%; and / or - The Y-th percentile of the score in the sub-image, where Y is less than or equal to 10%, preferably equal to 5%; and / or - Median score in the sub-image.

[0012] According to one implementation, the feature vector also includes the number of sub-images in the image.

[0013] According to one implementation, the first convolutional network is pre-trained on images that do not contain microorganisms, and then trained on labeled training image patches. Specifically, the first pre-trained convolutional network is a VVG16 or ResNet network.

[0014] According to one embodiment, the microorganisms include bacteria and the categories include at least Gram-positive and Gram-negative, and the preparation of the slides includes the preparation of Gram-positive slides. In particular, the microbial categories also include morphological categories. Notably, the microbial categories include the "non-bacterial, non-yeast" category, the "Gram-negative bacilli" category, the "Gram-positive bacilli" category, the "Gram-positive corynebacteria" category, the "Gram-negative cocci" category, the "Gram-positive streptococci" category, the "Gram-positive cluster cocci" category, and the "yeast" category.

[0015] According to one implementation plan, the samples include blood, and in particular, the samples are positive blood cultures.

[0016] Another object of the present invention is a method for training a model for predicting microbial categories from multiple microbial categories from digital images of a glass slide, the glass slide being coated with a sample that may contain microorganisms, the training method comprising: A. Forming a training database comprising digital images of slides labeled with one or more microbial categories, the slide images being divided into multiple sub-images labeled with one or more microbial categories and the training sub-images being further subdivided into image patches, at least some of which are labeled with one or more microbial categories; B. Train the first convolutional neural network based on the labeled image patches; C. Train a second neural network from a labeled sub-image database, the second network including at least one image patch feature extractor, a stage for pooling image patch features, and a stage for predicting one or more microbial classes present in the sub-images; D. Based on the categories predicted by the second neural network applied to the labeled sub-images, a distribution database of the categories present in the training slides is formed; E. A model for predicting the presence of at least one microbial class in a slide based on the spatial distribution of categories. In this method, the predictor includes a convolutional portion of a first neural network, a second neural network connected downstream of it, and a prediction model connected downstream of the second neural network.

[0017] In a variant of step C, the training method includes training a second neural network that includes an upstream pooling layer and one or more downstream layers for predicting microbial categories. The purpose of the second network is to predict one or more categories of a sub-image based on feature vectors generated by an image patch feature extractor. The training of the second network is performed based on feature vectors of labeled sub-images in a database and feature vectors of image patches of the labeled sub-images.

[0018] Specifically, the first network is trained on the microbial category and an "ambiguous" category is added. For objects in an image patch, when the objects in the image patch are unclear, the labeled image patch is also labeled by this category.

[0019] Specifically, training the second network includes training the MIL-CNN network, and the MIL portion of the trained MIL-CNN network constitutes the second trained neural network.

[0020] Another object of the present invention is a method for predicting the types of microorganisms contained in a sample from multiple microbial categories, the method comprising: C. A model using computer applications to predict the types of microorganisms in a sample coated on a glass slide based on acquired images of the slide.

[0021] According to the present invention, the image is subdivided into sub-images and each sub-image is further subdivided into image patches, and the application of the prediction model includes: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing microorganisms individually labeled by at least one category; E. A second neural network is applied to each sub-image, the second neural network being connected to receive features extracted from the constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled; as well as F. Regarding the acquired images: a. Calculate the feature vector based on the score calculated for the sub-image; b. Applying a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

[0022] Another object of the present invention is a prediction system for predicting the types of microorganisms contained in a sample from multiple microbial categories. The system includes a computer unit configured to implement a model for predicting the types of microorganisms in a sample coated on a glass slide based on acquired images of the slide. According to the invention, the images are subdivided into sub-images, and each sub-image is further subdivided into image patches. The application of the prediction model includes: D. For each image patch, apply a microbial feature extractor, the extractor comprising a convolutional portion of a first convolutional neural network trained on a set of training image patches containing microorganisms individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled with at least one category; and F. Regarding the acquired images: a. Calculate the feature vector for the sub-image; b. Applying a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

[0023] Another object of the present invention is a computer program product comprising a computer memory storing computer-readable instructions for performing steps D to F or steps A to E described above.

[0024] Another object of the present invention is a method for predicting the object categories contained in a digital image from multiple object categories, wherein the digital image is subdivided into multiple sub-images and each sub-image is subdivided into image patches, and wherein: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing objects individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing objects, each training sub-image being globally labeled; and F. Regarding the acquired images: a. Calculate the feature vector for the sub-image; b. Applying a model for predicting at least one category of objects present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images. Brief description of the attached diagram The invention will be better understood by reading the following description, which is provided by way of example only, and by referring to the accompanying drawings, in which: - Figure 1 The types of microorganisms present on Gram slides as predicted by this invention are shown; - Figure 2 This is a flowchart of the laboratory workflow for implementing the present invention; - Figure 3 This is a schematic diagram of a system for acquiring Gram slide images; - Figure 4 These are illustrations of Gram slide images at various scales used in this invention; - Figure 5 This details the three-stage prediction model based on the present invention. Figure 4 Flowcharts for operations at each scale; - Figure 6A -C is a schematic diagram of an image patch feature extractor, including the convolutional part of a convolutional neural network, which in this example is a VGG16 architecture network; - Figure 7 This is a schematic diagram of the prediction model on the sub-image, which in this example is formed in the MIL part of the MIL-CNN network with an attention mechanism; - Figure 8 This is a schematic diagram of an image prediction model, based on a random forest prediction stage that analyzes the distribution of descriptors provided by the sub-image prediction stage. - Figure 9A This is a schematic diagram showing the weights calculated based on the attention mechanism of the prediction model on the sub-image, displaying the inferred region of microorganisms in the Gram slide image or one or more sub-images constituting the image, and showing the predictions of the prediction model operating on the image patch. - Figure 9B Two sub-images analyzed by a convolutional neural network are shown, the feature extractor constituting the image patch of the implemented sub-image is derived from the convolutional neural network, and the predictions of the neural network are shown in the form of a "heatmap"; Figure 10 is a flowchart illustrating the training of a three-stage prediction model according to the present invention; - Figure 11A and 11B The performance of the three-stage prediction model according to the present invention in classifying Gram slide images in RGB imaging is shown; - Figure 12A and 12B The performance of the three-stage prediction model according to the present invention on Gram slide images in holographic imaging is shown; and - Figures 13 to 15 Various computer architectures for implementing the three-stage prediction model according to the invention and the microbiology laboratory workflow using the model are illustrated. Invention Details A. Methods and systems for characterizing bacterial Gram in Gram slides One embodiment of the invention will now be described, namely a microbiological laboratory workflow for IVD characterization, particularly bacterial Gram staining, in patients with suspected sepsis, said workflow being based on the analysis of RGB images of Gram slides prepared from positive blood cultures.

[0027] In particular ( Figure 1 The implementation scheme includes automatically flattening and classifying Gram slide images 10 into eight mutually exclusive categories: slides containing neither bacteria nor yeast, slides containing Gram-negative bacilli, slides containing Gram-positive bacilli, slides containing Gram-positive corynebacteria, slides containing Gram-negative cocci, slides containing Gram-positive streptococci, slides containing Gram-positive cluster cocci, and slides containing yeast.

[0028] refer to Figure 2 and Figure 3 The workflow 20 begins with the production of sample 22 from a blood sample collected from a patient, in this case a positive blood culture, for example, cultured in the BACT / ALERT® VIRTUO® system using the applicant's BACT / ALERT® culture medium bottles. As is known per se, blood culture involves multiplying the number of bacteria and yeast initially contained in the blood sample to make them detectable (identifying the "presence" or "absence" of bacteria or yeast) and facilitating their subsequent characterization due to the higher biomass. For this purpose, the blood is mixed with a culture medium, such as a soybean trypsin casein peptone-based medium supplemented with absorbent polymer beads. Therefore, blood culture samples are complex because they contain elements naturally present in blood (red blood cells, white blood cells, platelets, fibrinogen, etc.) as well as elements specific to blood cultures. Although the number of bacteria and yeast has been multiplied, they may still constitute a very small portion of the sample and be masked or confused with or influenced by heterogeneous elements, particularly in terms of number, shape, size, and colorimetric properties.

[0029] The method continues at step 24, preparing Gram slides from positive blood cultures. This preparation involves coating a portion of the blood culture at step 240 to obtain a coating thickness preferably less than 10 µm, which corresponds to the depth of field of a microscope with a magnification of 1000 subsequently used in the workflow. By adjusting the thickness to said depth of field, only a two-dimensional inspection of the coating is required, thus facilitating the analysis described below. After coating, the sample undergoes Gram staining known per se, for example automated by a PREVI® COLOR GRAM instrument sold by the applicant, the purpose of which is to stain the bacteria differently according to their Gram chromaticity.

[0030] Once dry and covered with a coverslip, the Gram slide was placed in a microscope at position 26 (40). Figure 3 In this microscope, a high-magnification oil immersion objective 42, ranging from 60x to 100x, is coupled to a Köhler-type illumination system 44 (illuminating with incoherent white light at least in the [400nm-900nm] range) and an RGB imaging system 46. System 46 includes, for example, a two-dimensional CMOS photoelectric sensor covered with a Bayer matrix for generating color images in a manner known per se, positioned within the image plane of objective 42. The optical and imaging systems of the microscope are selected and / or controlled such that bacteria and yeast (objects ranging in size from hundreds of nanometers to tens of micrometers) occupy at least 5 pixels, preferably at least 10 pixels, in the acquired image.

[0031] Because of the limited field of view of this objective lens, the Gram slide 48 is advantageously placed on a movable support 50, which can be moved by a stage equipped with a piezoelectric motor 52, allowing the slide to move in a plane (x,y) perpendicular to the optical axis z of the objective lens 42. This stage is connected to a computer unit 54, which is also connected to a sensor 46. The computer unit coordinates the movement of the support 50 and thus the slide 48, as well as the image capture by the sensor 46, to image the entire slide 48, at least the entire surface area of ​​the slide on which the sample 56 is coated. Preferably, the acquired original images of the slide are partially overlapped to avoid edge effects, such as the severing of bacteria, bacterial clusters, or chains.

[0032] The image set can be preserved to form all "sub-images," which correspond to the "images" described below, or even a single image of the slide can be obtained through computer reconstruction known per se, or the image set can be re-divided into "sub-images." According to the invention, three scales are obtained for a Gram slide: an image (composite or non-composite), constituent sub-images of the image, and constituent image patches of the sub-images. This digital image set, and preferably each of these digital images, covers a sufficiently large surface area of ​​the slide to a priori represent the microbial community present in the initial sample, such that one or more microbial classes present on the slide a priori exist somewhere in the image.

[0033] Further reference Figure 2 Therefore, step 28 of the workflow generates a microscopic or submicroscopic resolution image of the slide (in this example, a high-resolution composite image, or "ICHR" image). As an example, for a standard laboratory Gram slide measuring 25 mm × 75 mm, an image with at least tens of millions of pixels is obtained; the inventor's prototype has 150 million pixels, with a lateral pixel size corresponding to 40 nm, and each pixel encoded with 16 bits per color. Specifically, the acquisition system produces several hundred pixels for pairs of approximately one-micrometer-sized bacilli (40, 46).

[0034] Laboratory workflow 20 continues with a computer-implemented automated step 30 for analyzing the generated images to at least characterize the Gram number of bacteria present therein, and more specifically, predicting one or more categories of Gram-positive slides from the aforementioned categories. In particular, a “multi-class” prediction step 300 with a three-stage architecture as described below is implemented. This prediction generates a vector whose components correspond to the probability of the category, at least a score ranging between 0 and 1 and summing to 1, and a confidence index (“CI”) associated with the prediction. The specificity of the prediction is optimized such that if the index IC exceeds a predetermined confidence threshold (step 302), no further tests for characterizing the slide are required. Therefore, the prediction results can be directly forwarded to the clinician (step 32), for example, via a report received via email or a notification via smartphone, upon which the clinician can select appropriate antibiotic treatment to administer to the patient (step 34).

[0035] Advantageously, but optionally, the prediction architecture includes the attention mechanism described below. In addition to enhancing the interpretability of the prediction, this mechanism allows for the identification of regions in a Gram slide image that may contain microorganisms. Therefore, where the mechanism concentrates its attention, i.e., where it generates higher weight, the presumed region of microbial presence is displayed on a computer screen at step 304, overlaid on the Gram slide image, for use by a laboratory technician with expertise in slide interpretation. With the aid of this display, the technician can then manually characterize the slide (step 36). Optionally or additionally, the technician can also directly observe the slide under a microscope and characterize it in a conventional manner. It should be noted that this display can be provided across the entire image or even divided, for example, into sub-images for ease of reading.

[0036] Advantageously, but optionally, the prediction architecture also includes intermediate prediction steps for image patches that constitute the image and its sub-images. Again, predictions of different image patches can be overlaid on the image for use by laboratory technicians, which not only focuses their attention on the presumed regions but also provides presumed elements about the types of microorganisms present in different regions of the image. Again, it is possible to divide the image into sub-images.

[0037] As described below, the prototype developed by the inventors, trained on fewer than 600 slides, exhibited a macro-accuracy of 95% with a slide rejection rate of 17%. This prototype significantly improves and has already enabled the automation of Gram readings for most blood culture samples, while providing information beyond simple Gram readings (bacilli, cocci, clusters, chains, yeast, etc.). This automated time saving effectively compensates for the scarcity of human expertise in this field.

[0038] B. A multi-stage predictive architecture for automated analysis of complex images, especially Gram slides. An embodiment of the hierarchical architecture of the predictive model (or “predictor”) according to the present invention will now be described in more detail. Although the architecture is described in relation to the analysis of RGB images, it can also be used to analyze different types of images, particularly holographic images as described below, and even multispectral or hyperspectral images.

[0039] This invention benefits from the extremely high resolution of ICHR images, which can be divided into dozens, preferably at least one hundred, sub-images. Each sub-image can be further subdivided into dozens of image blocks, each image block having a pixel size sufficient to completely contain the microorganism or morphology (cluster, chain, etc.). This subdivision, as... Figure 4As shown, this describes the subdivision of an image into sub-images and the subdivision of sub-images into image patches. Specifically, in the sub-images and image patches shown, the bacillus corresponds to a dark object and is smaller than the image patch, being completely contained within it. For example, the image patch size is 256 pixels × 256 pixels (or 224 × 224), corresponding to approximately 10 × 10 µm. 2 The actual surface area is considered, while the typical maximum size of bacteria and yeast is approximately in the micrometer range; the sub-image size is 1024 pixels × 1536 pixels. In a preferred variant of the invention, adjacent image patches partially overlap to avoid edge effects when training the predictor on the image patches. Similarly, adjacent sub-images partially overlap for the same reason. As a variant, the sub-images do not cover the entire ICHR image and do not overlap in order to better cover the diversity associated with coating inhomogeneity.

[0040] As described below regarding the various stages of learning in a multi-class predictor, this possible subdivision due to the extremely high resolution of ICHR images is associated with the cumbersome annotation process that gradually lessens from image patches to ICHR images. Specifically, this applies to ICHR learning image databases that are subdivided into sub-images, and each sub-image is further subdivided into image patches: i. For each sub-image, the image patch is heavily labeled, at least by one or more microbial categories it contains. In particular, the labeled image patch is advantageously, but optionally, associated with dual labeling: a first overall label corresponds to one or more microbial categories it contains, and a second label encodes the presence or absence of a microorganism for each pixel of it; ii. Each sub-image is globally labeled with one or more categories of existing microorganisms; iii. Each image is globally labeled with one or more categories of existing microorganisms.

[0041] According to the invention, the model extracts image patch features by means of a model for predicting categories on image patches, the image patch features being transferred to a second stage, the second stage extracting sub-image features by means of a model for predicting categories on sub-images, the sub-image features being transferred to a third stage, the third stage performing the prediction of the final category of microorganisms present in the image and thus in the Gram slide.

[0042] In the following text, the ICHR image is subdivided into a two-dimensional sub-image matrix identified by index (i, j), and each sub-image is further subdivided into a two-dimensional image block matrix identified by index (k, l).

[0043] refer to Figure 5 A particular implementation of predictor 60 includes: A. Analysis stream 62 for each sub-image of the ICHR image, each stream produces a score vector, one score per category (represented as "Class_scores (i, j)" for the sub-image at coordinates (i, j)). Analysis stream 62 is executed independently and is identical in terms of computation modules; B. Global analysis stream 64, receiving the feature vector of each stream 62, in Figure 5 The preferred variant shown is a score vector, which generates a score vector for the ICHR image, with one score for each category, denoted as "Class_image", and a confidence index for the prediction, denoted as "Conf_index".

[0044] The analysis stream 62 for each sub-image includes a feature extractor 66, preferably composed of the convolutional part of a convolutional neural network. Figure 6A As a variant, the extractor includes a convolutional portion of a neural network, followed by a flattening layer. Figure 6B As a variant, the extractor comprises a convolutional portion of a neural network, followed by a flattening layer, and then one or more fully connected layers. Figure 6C Optionally, these variants are supplemented downstream with one or more fully connected neuron layers, with the feature vector corresponding to the output of the last connected layer.

[0045] The purpose of this extractor is to extract features from each image patch that makes up a sub-image, denoted as "Emb", which summarizes the information contained in the image patch, particularly regarding the presence or absence of microorganisms.

[0046] refer to Figure 6AThe convolutional neural network 66_CNN is trained on a database of labeled training image patches 66_BDD. The purpose of network 66_CNN is to predict one or more microbial classes present in image patches. In a preferred embodiment of the invention, neural network 66_CNN is a network of type VGG16, RESNET, MobileNetV2, or EfficientNet, preferably pre-trained on a public database, such as those available at the URL https: / / www.image-net.org, particularly a VGG16 network trained on the “ImageNet Large Scale Visual Recognition Challenge (ILSVRC)” (https: / / www.image-net.org / challenges / LSVRC / index.php). Although a blank network, i.e., initialized with random weights and trained from scratch with labeled image patches, can be used, using a pre-trained network improves the macro accuracy of predicting one or more classes of ICHR images by several percentage points. Among dozens of pre-trained networks tested by the inventors, VGG16 is the most effective, achieving an improvement in macro accuracy of approximately 5% compared to networks trained only with image patches from slide images. Because this is a network with a fixed architecture, the image patch sizes derived from ICHR images were chosen to be the same. This size satisfies the conditions described above regarding the sizes of microorganisms, bacteria, and yeast.

[0047] In a favorable but optional variant, predictions made by the convolutional neural network 66_CNN can be used to identify regions in a sub-image that may contain regions of a specific class. These results can be presented as a "heatmap" and overlaid on the image. A sub-part 66 of the network 66_CNN and its parameters are then extracted. This sub-part 66 may include convolutional layers of the 66_CNN, as well as some downstream fully connected layers. New fully connected layers may also be optionally added downstream of it. This element 66 acts as an extractor to extract features from image patches that are correlated with predictions of the relevant class.

[0048] Once features are extracted from each image patch, the corresponding feature vector is sent to a second prediction stage 68, which, based on a neural network, performs multi-class predictions of one or more microbial categories present in the sub-image composed of image patches. More specifically, as Figure 7 As shown, the second stage 68 is the "MIL" ("Multiple Instance Learning") part of the MIL-CNN type network, whose feature extractor part is derived from model 66_CNN, which is advantageously the extractor 66.

[0049] The second predictor 68 includes: i. Upstream stage 70: Implement a gating attention mechanism. In this stage 70, for each feature vector generated from an image patch (represented as Emb for an image patch with coordinates (k, l) in sub-image coordinates (i, j)... i,j (k, l) calculates the coefficient a i,j (k, l), this coefficient measures the weight of the image patch in the sub-image category prediction, and stage 70 multiplies this weight by the corresponding vector to generate a new feature vector a for each image patch. i,j (k, l)×Emb i,j (k, l). In a preferred but not mandatory implementation, this weight is calculated according to the following relationship: in and It is the matrix that constitutes the 68 parameters of the predictor, and Q is the eigenvector Emb. i,j The dimension is (k,l), r is a predetermined positive integer, for example, equal to 512, sigm is the sigmoid nonlinear function, and ⊙ is the term-by-term multiplication operator. This mechanism is specifically described in the paper "Attention-based Deep Multiple Instance Learning" by M. Ilse et al., arXiv:1802.04712V4 [cs.LG], June 28, 2018. Other attention mechanisms are also possible.

[0050] ii. Downstream of or integrated into the attention mechanism stage 70, the pooling stage 72 reduces the overall dimensionality of the feature vectors. For example, stage 72 generates a unique feature vector Z(i,j) for each sub-image (i,j) from K × K vectors derived from the image patch according to the following relation: iii. Downstream of pooling stage 72, stage 74, with a fully connected layer of neurons, receives the vector Z(i,j) as input and produces a predicted score vector Class_scores(i,j) as output. The scores are normalized, ranging between 0 and 1, and conventionally, the higher the score, the higher the probability that the corresponding class exists in the sub-image. In the modality under consideration, this downstream stage can be simplified to a sigmoid-type layer that produces the scores, but other layers can be inserted between the pooling stage and the score acquisition stage if necessary.

[0051] In a preferred embodiment, model 66 and its downstream portion 68 are thus integrated into a new predictor that operates on the sub-image. Figure 5The one implemented in workflow 62. As described in the paper by Ilse et al., this new predictor can learn on sub-image examples, specifically using conventional backpropagation techniques. Preferably, but optionally, the parameters of model part 66 are allowed to evolve freely during this learning phase, so that their values ​​no longer come from the learning of model 66_CNN at the end of the process. In this way, feature extractor 66 and downstream layer 68 are jointly optimized.

[0052] refer to Figure 8 The final stage 64 of predictor 60 includes: i. Upstream stage 80 receives descriptors, such as the predicted score vector Class_scores(i,j) derived from the set of sub-images constituting the Gram's slide image, and calculates the score distribution for each class on the slide based on these vectors. This stage 80 benefits from the large number of sub-images resulting from the high resolution of the ICHR image. More specifically, since the ICHR image consists of dozens or even hundreds or more sub-images, the score distribution of the classes present in the ICHR image can be calculated in a statistically correlated manner. One advantage of calculating the distribution is obtaining a characterization of the ICHR image that is independent of the location of the microorganism in the image while taking into account the entire image. In particular, for sensitive applications such as in vitro diagnosis of patients, especially sepsis, the reliability of results based on a single or limited area of ​​the slide is questionable. In fact, predictions based on a single sub-image may be incorrect or even overly uncertain.

[0053] For example, for each microbial category, phase 80 calculates an approximation of this distribution, involving the extraction of the following statistics: - The maximum score; - 95th percentile; - Median score; - 5th percentile.

[0054] This approximation is calculated quickly and allows for adjustment of the overall predictive performance by each of its components. Notably, the maximum value and the 95th percentile allow for adjustment of the sensitivity level for predicting actual microbial categories (in this case, the 95th percentile is chosen, but other values ​​greater than 50% are also possible depending on the desired sensitivity). The median value allows for mitigation of prediction errors made on sub-images. The 5th percentile allows for adjustment of the sensitivity for predicting the "no microbes" category (in this case, the 95th percentile is chosen, but other values ​​less than 50% are also possible depending on the desired sensitivity). It should be noted that simpler distribution approximations are also within the scope of this invention (e.g., only the maximum score for each category), as well as more complex approximations (e.g., polynomial interpolation of scores, or even approximations by the distribution characterized by its equations, such as Fisher's law, Gauss's law, Bernoulli's law, etc.). In particular, the advantage of obtaining descriptive statistics is the ability to consider application-specific features known to microbiology experts, such as those illustrated by using percentiles to help adjust sensitivity and specificity, two important criteria in IVD diagnostics.

[0055] ii. Downstream stage 82, for each received class distribution and the number of sub-images, R × R, and predicts one or more classes of microorganisms present in the ICHR image, Class_image, and the confidence index Conf_index of this prediction as output. More specifically, the prediction performed in stage 82 is a multi-class prediction by machine learning, and preferably a prediction without neural networks. In fact, the features received in this stage are structured with a defined, fixed number of features and known properties. Furthermore, “conventional” machine learning methods (i.e., non-neural network-based) such as those based on SVM (“Support Vector Machine”), K-nearest neighbors, decision trees, etc., are more suitable in terms of performance, interpretability, and ease of training. Of all these methods, stage 82 preferably performs a “random forest” type prediction, which is particularly effective in avoiding overtraining and providing an interpretable confidence index CI (e.g., equal to the maximum value of the percentage of votes for each class). In fact, as the inventors have noted for the interpretation of complex images, the random forest confidence index takes a high value for accurate predictions (i.e., consistent with the reality of microorganisms present in the slide), otherwise it collapses. Therefore, the choice for step 302 ( Figure 2 The threshold for random forest prediction becomes easy and reliable for use in sensitive applications such as in vitro diagnostics. For example, the random forest prediction model is described in Breiman et al.'s paper "Random Forests", Machine Learning, 45(1), 5-32, 2001.

[0056] It should be noted that the descriptors received by model 80 are not necessarily derived from the prediction scores given by predictor 68. In particular, descriptors derived from layers located further upstream in its architecture can be used.

[0057] The various prediction models described above (image patches, sub-images, or images) are "multi-class" prediction models. As variations, image patch models and / or sub-image models and / or image models can implement "multi-label" prediction in this case, i.e., prediction for the presence / absence type of each considered class, rather than just the presence of one class (the negative class is then treated as an independent class). Multi-label prediction allows for handling certain special cases, such as multimicrobial samples with several Gram types. The advantage of choosing a multi-class prediction model is the reduction in the amount of labeled training data and / or corresponding Gram slides or sample collection activities. Indeed, in the context of microbial infections, such as multimicrobial cases which are mostly a minority compared to single-microbial infections, the relevant data is much less, making training multi-label prediction models more difficult.

[0058] refer to Figure 9A After analyzing the slide image 90, the preferred use of the output from the predictor 60 therefore involves displaying the image 90 on a computer screen 92, the light intensity of which is determined according to the coefficient a. i,j (k,l) Image patch settings. It is worth noting that a higher image patch weight means greater importance in sub-image prediction; the brighter the image patch, as shown by image patch 94, it is much clearer than other areas of image 90. This allows laboratory technicians to examine the content regarding the presence of microorganisms in the image patch as needed and confirm or refute its content. Technicians can therefore display the entire image, or preferably specific sub-images for readability, as shown in the figure. Preferably, one or more predicted categories encoded by the Class_image vector are also displayed, along with the associated confidence index Conf_index, and a signal encoding prediction failure when the index falls below a threshold. Brighter image patches then serve as presumed areas that may contain microorganisms. This assistance then allows technicians to perform analysis more quickly by focusing on these areas first.

[0059] Advantageously, such as Figure 9A The second display and Figure 9B As shown in the image, the sub-images are also analyzed by a convolutional neural network 66_CNN, and its predictions can be displayed on the screen, advantageously as a "heatmap," so that technicians can directly know which areas in the image might correspond to microorganisms and which category they correspond to.

[0060] C. Labeled data and multi-stage prediction architecture C.1. Labeling Data Three annotation scales are generated: the entire image, corresponding to the slide; sub-images; and image patches.

[0061] First, consider the slide level, which is the level relevant from a biological and medical perspective. Each image prior to the slide covers a sufficiently large surface area to reflect the contents of the initial sample. Furthermore, any available knowledge about the sample or slide content can be used to generate annotations on the images.

[0062] Preferably, the Gram slides are from a microbiological analysis laboratory and are produced and labeled under real-world conditions by technicians specializing in Gram analysis.

[0063] However, interpreting complex slides can be difficult, even for experienced technicians. Preferably, a portion of the sample used to produce the slide is characterized in a more in-depth manner. Notably, the actual identity of the microorganisms present in the sample is determined, preferably by MALDI-TOF mass spectrometry using VITEK® MS sold by the applicant.

[0064] The image of a Gram slide first inherits the slide annotations generated in the laboratory, referred to as the "original" annotations. These original annotations are then reviewed by a second expert, who corrects errors and ambiguities using additional characterization of the microorganisms. The final annotations of the slide image are then referred to as the "true values." In addition to correcting the original annotations, the slides undergo initial classification: slides containing multimicrobial mixtures or Campylobacter are discarded from the training set. To improve the overall performance of the predictor according to the invention, Acinetobacter is classified in the "Gram-negative bacilli" category. Other sample characteristics (metadata) are also collected, such as, if applicable, the type of culture medium contained in the sample, the elapsed time to obtain a positive blood culture, the elapsed time to display Gram staining, and even the antimicrobial spectrum of the microorganisms.

[0065] Now, sub-image level annotation is described. Each sub-image of an image receives its "ground truth" annotation by default, but this annotation can be modified by a second expert, for example, in cases where microorganisms present on the slide are not present in the sub-image under consideration. Therefore, to characterize the entire slide image, the second expert traverses each region and thus each sub-image. Preferably and optionally, when the second expert cannot annotate a sub-image in a non-ambiguous manner, the sub-image is discarded from the training set. Preferably and optionally, the "ground truth" annotation of the sub-image is modified and reduced to a single category, so that it can then be restricted to a "multi-class" type model, while showing rare sub-images of two categories being excluded from the training set. Preferably and optionally, for special cases of bacteria exhibiting multiple morphologies (e.g., Gram-variable bacilli such as Bacillus subtilis), Bacillus subtilis(These are Gram-positive bacilli, but often appear as Gram-negative bacilli). The annotations on sub-images can be modified to reflect the actual appearance of the bacteria, allowing sub-image annotations to differ from image annotations. Therefore, annotations are obtained for all or some sub-images, which are typically about one hundred times more numerous than the images in the database. Sub-images are heavily annotated because the designers of the predictor according to the invention incorporate knowledge, and therefore additional prior information, into the sub-image annotations. This annotation will be used to improve predictor training.

[0066] Following the same logic, even heavier annotations can be generated for image patches on at least some datasets. In one embodiment of the invention, this annotation is accomplished by first performing semantic segmentation of the microorganisms, i.e., associating each pixel of the sub-image with a Gram type. Various methods are possible for performing this semantic segmentation step. For example, semi-automatic segmentation can be performed using tools such as Ilastik, described in the paper “ilastik: interactive machine learning for (bio)image analysis” by S. Berg et al., Nature Methods, (2019), available at the URL https: / / www.ilastik.org / , associating each pixel with a microbial or non-microbial state. The sub-image annotations can then be used to associate the microbial pixels with a category. A final manual step performed by an expert ultimately allows for the correction of the obtained segmentation mask and also allows for the highlighting of any blurred areas. Once semantic segmentation is performed, an explicit algorithm allows each image patch of the sub-image to be assigned a category based on the number of pixels of the relevant category present in the image patch.

[0067] Just as sub-image annotations are not necessarily the same as image annotations, image patch annotations are not necessarily the same as sub-image annotations. In particular, when some categories are associated with the organizational state (chain, cluster) of microorganisms, this organization is not always detectable on a reduced-area image patch, and when the number of microbial pixels on the image patch is insufficient, the annotation can be converted to disregard the aggregation state. For example, the category "Gram-positive chain aggregated cocci" would be converted to the category "Gram-positive cocci—aggregation state undetermined" using this method. Finally, new "fuzzy" categories can be assigned to image patches when their constituent pixels are labeled as such by experts, i.e., when the number of pixels associated with the microorganism is very low (this typically corresponds to organisms partially visible due to their location at the edge of the image patch).

[0068] Once implemented, this method yields annotations on image patches for all or some of the available datasets. It should be noted that, in the modalities considered, the number of image patches is typically tens or hundreds of levels per sub-image. The image patches are heavily annotated because the designers of the predictor according to the invention incorporate additional prior information into the annotations of the image patches. This annotation will be used to improve predictor training.

[0069] The slide images and their "truth values" are stored in computer memory for subsequent processing. Similarly, sub-images and their annotations are stored in computer memory. Likewise, image patches and their annotations are stored in computer memory for subsequent processing. This results in three databases.

[0070] Using the method described above, annotations with three progressively stronger levels are obtained: image level, sub-image level, and image patch level. This significantly enhances the overall strength of the annotations on the training set. However, given the number of available images (737 in the considered implementation example) and their dimensionality (over 100 million pixels in the considered example), directly training a predictor using only image-level annotations is most likely to fail.

[0071] C.2. Multi-stage predictor training The method for training the predictor 60 will now be described. According to this method, each stage is trained independently of the other stages, while being based on the same training Gram slide images.

[0072] refer to Figure 10A The method 100 for training the predictor 60 includes constructing a training Gram slide image database in 102, the database being labeled with one or more existing microbial categories, particularly in a database of labeled images, labeled sub-images, and labeled image patches, in the manner described above.

[0073] The method continues by training a convolutional neural network 66_CNN in 104 to predict the category of image patches. This training can begin with an untrained network. However, as mentioned above, it is preferable to select an initial pre-trained network and then retrain it on a database of labeled image patches. In this option, depending on the selected pre-trained network, normalization of the image patch pixel values ​​can be implemented. It is worth noting that some networks accessible in network libraries, particularly those based on the "TensorFlow" software library, are trained on images with normalized pixel values. In this case, upstream of network 66_CNN, normalization, such as standardization (data centered at 0, standard deviation of 1), is provided for the image patches. This normalization stage can then optionally be provided upstream of the feature extractor 66 in the predictor according to the invention.

[0074] The labeled image patch database is divided into training and test sets, ensuring that (the "patch" technique) all image patches originating from a given image are either entirely placed in the training set or entirely placed in the test set. Hyperparameter optimization of Network 66_CNN is performed on the training set using cross-validation techniques, such as using four "folds" formed by "patch" (or "patch cross-validation"), which specifically ensures that all image patches associated with a given image reside in the same "fold". Hyperparameter values ​​are optimized using techniques such as "grid search," "random search," or Bayesian methods.

[0075] In one variant, image patches already labeled "fuzzy" are retained for training and testing, and the learning includes an additional "fuzzy" category that merges image patches in which the labeling experts are unsure which microorganisms are present.

[0076] Once the CNN network 66_CNN is trained, the convolutional parts are extracted, and the last few layers are removed from the network.

[0077] In the first variant of the sub-image level prediction model learning, the extractor is integrated into a MIL-CNN type model via transfer learning. This network is, for example, the network described in the paper by M. Ilse et al. For example, the source code for the network and its training are available on Github at https: / / github.com / AMLab-Amsterdam / AttentionDeepMIL. The method continues training the MIL-CNN network in step 108, dividing the sub-image database into training and test sets, ensuring that the training and test sets have the same image distribution as the image patch model training. The values ​​of its hyperparameters are again optimized through cross-validation, for example using a "grid search" technique, a "random search" technique, or a Bayesian method. Preferably, these hyperparameters include the architecture of the fully connected neuron layer stage 74 (e.g., number of layers, number of neurons per layer, etc.). In this variant, the convolutional and MIL parts of the trained MIL-CNN model can respectively constitute the image patch level extractor 66 and the sub-image level prediction model 68. As a variant, the MIL part is extracted to form model 68, and the extractor is the one trained in the previous step.

[0078] In the second learning variant of the sub-image level prediction model learning, only the MIL part of the MIL-CNN network is trained. For example, all image patches of all sub-images are processed by the trained extractor 66 to obtain the corresponding set of feature vectors, each feature vector being labeled with the annotation of the corresponding sub-image, such as... Figure 10A As shown. These feature vector sets and their labels are stored in database 106. Then, model 68 is trained on this database using the cross-validation technique described above.

[0079] Once MIL stage 68 is trained, each batch of sub-images corresponding to an image is processed in 110 by stage 68 and distribution computation module 80 to generate score distributions, which are stored in the database along with the corresponding image annotations. Then, in 112, the final stage 82, training the random forest-based predictor, divides the distribution database into training and test sets, again with the same slide distribution as before. Hyperparameter optimization is again performed using 4-fold cross-validation. The values ​​of its hyperparameters are optimized, for example, according to a "grid search" technique, with a more limited number of hyperparameters than in the previous predictor.

[0080] Optionally, the method continues at 114, calculating a performance metric, such as overall accuracy, of the stage predictor according to the invention based on the slide rejection rate. It is noteworthy that this calculation involves the predictor processing all slide images and altering step 302 ( Figure 2 A confidence threshold is set. Any slide that does not exceed the threshold is discarded, and the rejection rate is equal to the percentage of discarded slides. The overall accuracy is then calculated for the remaining slides. According to the invention, if the rejection rate is considered too high or the accuracy too low, the method continues to acquire new slides and / or additionally annotate image patches and retrain the predictor as described above to reduce the rejection rate and improve the accuracy.

[0081] Figure 10B An example of the accuracy and rejection rate of a stage predictor prototype designed by the inventors through multiple iterations is shown. In this example, with a target accuracy greater than or equal to 95%, the rejection rate obtained is 17%, which means that this prototype, if designed as is... Figure 3 The image acquisition systems, when used in combination, allow for automated processing of over 80% of Gram slides without additional testing. These performance levels were obtained on an initial dataset of 737 Gram slides, each slide divided into 100 sub-images, and each sub-image into 24 image patches. Of the nearly 1.8 million image patches in total, only 3%, or approximately 50,000, were heavily labeled, while the total number of predictor parameters according to the invention is approximately 15 million.

[0082] Figure 11A Further details of the performance obtained by the predictor prototype according to the invention are shown, which was trained on the aforementioned set of Gram slides, and the performance was obtained on all slides (i.e., the rejection rate was set to 0%). Figure 11B The corresponding confusion matrix is ​​shown.

[0083] D. Application of this invention to signals other than RGB signals The application of this invention to images with pixel values ​​encoded on three color channels (RGB) has already been described. This invention is applicable to other types of signals on one or more wavelengths, particularly digital images with pixel values ​​encoded as holographic information.

[0084] For example, a system for acquiring Gram slide images is the system described in patent applications EP 4307051 and EP 4172699, which is connected to a computational module for reconstructing a refocused image. Independent of the specific variants described in these applications, the principle of holographic imaging involves illuminating a slide with coherent light of one or more wavelengths, recording corresponding intensity images, and generating a holographic digital image of the slide by computer reconstruction (this reconstruction is referred to as “parametric” reconstruction, as described, for example, in these applications, or as “non-parametric” reconstruction, as described, for example, in applications WO 2016 / 075279, WO 2017 / 077238, or WO 17207184), where each pixel is encoded by an intensity value and a phase value for each illumination wavelength (in one embodiment described in both EP 4307051 and EP 4172699, there are eight, thus each pixel is encoded on 16 channels).

[0085] In the first variant, the feature extractor 66 originates from an untrained convolutional network 66_CNN, thus allowing all channels to be used as input. In the second variant, the extractor originates from a pre-trained convolutional network, specifically a network pre-trained on RGB images, as described above. In this option, a dimensionality reduction stage is provided upstream of the extractor 66 if the number of channels / pixels is greater than 3. Advantageously, this stage involves computing the three principal components of the holographic image and injecting these three principal components into the extractor. Advantageously, the principal components used for dimensionality reduction are selected as hyperparameters during the training of the convolutional network 66_CNN.

[0086] The annotation of slide images, sub-images, and image patches, as well as the training of the predictor, are performed in a similar manner to that described above. Figure 12A and 12B The performance levels of the holographic image prototypes developed by the inventors are described, and these performance levels were obtained from an initial database of 737 Gram slides.

[0087] E. Further instructions regarding the detailed implementation plan i. The application of this invention in characterizing microorganisms in samples prepared on Gram slides has already been described. This invention is applicable to any type of digital image including the object to be characterized. For example, still in the field of biology, this technique is suitable for cell imaging of eukaryotic cells, particularly fluorescently labeled cells, whose different types and their different organelles (e.g., the nucleus) can be detected. Another example is hematology, where different graphic elements of blood (red blood cells, white blood cells) can be detected. A third example is urine sample analysis, where graphic elements as well as microorganisms can be detected. A fourth example is histopathology, used to detect cancer cells that may only cover a very small portion of the surface area of ​​a very high-resolution image.

[0088] For some applications where the concept of quantification or counting is important (not so in positive blood cultures), the analysis of the number of positive image patches and / or sub-images of a specific category can serve as a starting point for implementing cell type counting.

[0089] Generally, this invention is applicable to the characterization of any object in composite or non-composite high-resolution images.

[0090] ii. An implementation has been described in which the features provided in the final stage are the predicted scores of the second stage. As a variation, the features are, for example, those generated by the neuronal layers of the MIL part. According to the invention, descriptive statistics are also generated from these features and fed into the final prediction model.

[0091] iii. A sub-image level prediction model implementing an attention mechanism has been described. As a variant, this model does not implement this function; instead, feature vectors derived from image patches are directly grouped using a pooling method, such as applying average pooling or max pooling.

[0092] F. Computer implementation The training phase of the predictor in this invention (especially the reference phase) Figure 10A The training phase and the prediction phase (especially the reference phase) are mentioned. Figure 5 The prediction stage, apart from the steps of preparing the slide and acquiring the light signal corresponding to the image of the slide, is implemented by a computer, i.e., by means of hardware circuitry including computer memory (cache, RAM, ROM, etc.) and one or more microprocessors or processors (CPU and / or GPU), which may or may not be organized as computing nodes, is required to execute computer instructions stored in memory for implementing the stage. It should be understood that any type of computer architecture can be suitable for computational purposes, and the description provided above and below should not be construed as limiting the scope of the invention.

[0093] Several architectures are possible, such as Figure 13-15 As shown.

[0094] First architecture variant ( Figure 13In this process, a first organization 2000, such as the applicant, hosts or controls one or more computing servers 2002 associated with one or more databases 2002 for storing labeled slide images, and the learning phase is implemented by organization 2000 on its one or more servers 2002. A second organization 2006, such as a microbiology laboratory, houses a microscope 2008 as described above, which is connected to or incorporated into a personal computer, desktop computer, or server 2010 and performs Gram slide preparation until a digital image of the slide is obtained by the computer unit 2010. This image is then forwarded via a remote connection network to the first organization 2000 for the remaining portion of the prediction phase to be implemented on server 2002 (or with a different computer unit used for training, e.g., implemented in the cloud as software as a service). The resulting classification report of possible or possible non-existent microorganisms is then forwarded via network 2012 to the second organization 2006, which can take or not take therapeutic measures based on the report.

[0095] Second architecture variant ( Figure 14 The difference from the first architecture variant is that a copy of the software implementing the prediction phase is downloaded to a second organization, which uses computer 2010 or a computing server (not shown) to implement the entire prediction phase. In one variant of this architecture, this download corresponds to a copy of the software in computer 2010, which is supplied by the first organization 2000. In the third variant ( Figure 15 In 2006, all learning and prediction phases were implemented by a single organization.

Claims

1. A method for predicting the types of microorganisms contained in a sample from multiple microbial categories, the method comprising: A. Preparing glass slides, particularly microscope slides, including coating the sample onto the glass slide; B. Acquire at least one digital image of the slide with micrometer or submicrometer resolution; C. Using computer applications to predict microbial categories based on acquired images. The image is characterized in that it is subdivided into multiple sub-images and each sub-image is further subdivided into image patches, and the application of the prediction model includes: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing microorganisms individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers, the downstream layers including layers implementing prediction of at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled; and F. Regarding the acquired images: a. Calculate the feature vector for the sub-image; b. Applying a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

2. The method as described in claim 1, characterized in that, The feature vector is a score distribution calculated for a sub-image.

3. The method as described in any one of claims 1 or 2, characterized in that, Each training image is subdivided into sub-images and each sub-image is further subdivided into image patches, characterized in that less than 50% of the image patches of the training images are labeled, and the labeled image patches form the training image patches of the first convolutional network.

4. The method as described in claim 3, characterized in that, Less than 10% of the image patches in the training images are labeled.

5. The method as described in any of the preceding claims, characterized in that, The training of the first and second convolutional networks and the prediction model was configured to achieve a prediction macrospecificity of greater than or equal to 90%.

6. The method as described in any of the preceding claims, characterized in that, The prediction model for step F is the "random forest" model.

7. The method as described in any of the preceding claims, characterized in that, The upstream pooling layer is associated with an attention layer that is configured to apply weights to the output of each extractor.

8. The method as described in any of the preceding claims, characterized in that, The second network is the MIL-CNN network, which is trained through instance batches, where each instance batch consists of training image patches.

9. The method as described in any one of the preceding claims, characterized in that, The feature vector for step Fa includes the following for each category: - The maximum score in the sub-image; and / or - The Xth percentile of the score in the sub-image, where X is greater than or equal to 90%, preferably equal to 95%; and / or - The Y-th percentile of the score in the sub-image, where Y is less than or equal to 10%, preferably equal to 5%; and / or - Median score in the sub-image.

10. The method as described in claim 8, characterized in that, It also includes the number of sub-images in the image.

11. The method as described in any of the preceding claims, characterized in that, The first convolutional network is pre-trained on images that do not contain microorganisms, and then trained on labeled training image patches.

12. The method as described in claim 10, characterized in that, The first pre-trained convolutional network is either VVG16 or ResNet.

13. The method as described in any of the preceding claims, characterized in that, The microorganisms include bacteria and the categories include at least Gram-positive and Gram-negative, and the preparation of the slides is characterized by the preparation of Gram-positive slides.

14. The method as described in claim 13, characterized in that, Microbial categories also include morphological categories.

15. The method as described in any of the preceding claims, characterized in that, Microbial categories include "non-bacteria, non-yeast", "Gram-negative bacilli", "Gram-positive bacilli", "Gram-positive corynebacteria", "Gram-negative cocci", "Gram-positive streptococci", "Gram-positive cluster cocci" and "yeast".

16. The method as described in any of the preceding claims, characterized in that, The samples included blood.

17. The method as described in any of the preceding claims, characterized in that, The sample was a positive blood culture.

18. A method for training a model to predict microbial categories from multiple microbial categories in digital images of a glass slide, the glass slide being coated with a sample that may contain microorganisms, the training method comprising: A. Forming a training database comprising digital images of slides labeled with one or more microbial categories, the slide images being divided into multiple sub-images labeled with one or more microbial categories and the training sub-images being further subdivided into image patches, at least some of which are labeled with one or more microbial categories; B. Train the first convolutional neural network based on the labeled image patches; C. Train a second neural network from a labeled sub-image database, the second network including at least one image patch feature extractor, a stage for pooling image patch features, and a stage for predicting one or more microbial classes present in the sub-images; D. Based on the categories predicted by the second neural network applied to the labeled sub-images, a distribution database of the categories present in the training slides is formed; E. A model for predicting the presence of at least one microbial class in a slide based on the spatial distribution of categories. In this method, the predictor includes a convolutional portion of a first neural network, a second neural network connected downstream of it, and a prediction model connected downstream of the second neural network.

19. The training method as described in claim 18, characterized in that, The first network is trained on the microbial category and a "fuzzy" category is added. When the objects in an image patch are unclear, the labeled image patch is also labeled by this category.

20. The training method as described in claim 18 or 19, characterized in that, Training the second network includes training the MIL-CNN network, and the MIL portion of the trained MIL-CNN network forms the second training neural network.

21. A method for predicting the types of microorganisms contained in a sample from multiple microbial categories, the method comprising: C. A computer-aided model for predicting the types of microorganisms in a sample coated on a glass slide based on acquired images of the slide. The image is characterized in that it is subdivided into sub-images and each sub-image is further subdivided into image patches, and the applied prediction model includes: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing microorganisms individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled with at least one category; and F. Regarding the acquired images: a. Calculate the feature vector based on the score calculated for the sub-image; b. Applying a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

22. The prediction method as described in claim 21, characterized in that, Steps D to F conform to any one of claims 2 to 17.

23. A prediction system for predicting the types of microorganisms contained in a sample from multiple microbial categories, the system comprising a computer unit configured to implement a model for predicting the types of microorganisms in a sample coated on a glass slide based on acquired images of the slide. Its features are, The image is subdivided into sub-images, and each sub-image is further subdivided into image patches, characterized in that the applied prediction model includes: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing microorganisms individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing microorganisms, each training sub-image being globally labeled with at least one category; and F. Regarding the acquired images: a. Calculate the feature vector for the sub-image; b. Applying a model for predicting at least one class of microorganisms present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

24. The prediction system as described in claim 23, characterized in that, The computer unit is configured to implement steps D to F according to any one of claims 2 to 17.

25. A computer program product comprising a computer memory storing computer-readable instructions for implementing steps D to F as described in any one of claims 2 to 17.

26. A computer program product comprising a computer memory storing computer-readable instructions for implementing steps A to E as described in any one of claims 18 to 20.

27. A method for predicting object categories contained in a digital image from multiple object categories, wherein the digital image is subdivided into multiple sub-images and each sub-image is subdivided into image patches, and wherein: D. Apply a microbial feature extractor to each image patch, the extractor comprising a convolutional portion of a first convolutional neural network trained on a training set of image patches containing objects individually labeled by at least one category; E. Apply a second neural network to each sub-image, the second neural network being connected to receive features extracted from constituent image patches of the sub-image, the second network including an upstream pooling layer and one or more downstream layers for predicting at least one category in the form of scores, the second network being trained on training sub-images containing objects, each training sub-image being globally labeled; and F. Regarding the acquired images: a. Calculate the feature vector for the sub-image; b. Applying a model for predicting at least one class of objects present in a sample based on the feature vector, the prediction model being trained on feature vectors computed from training images.

Images