Method and system for characterising microorganisms contained in a complex sample

Abstract

The disclosed method for classifying microorganisms contained in a sample comprises: preparing a slide of the sample; acquiring at least one digital image of the slide; and implementing, with a computer, a model for predicting the class of the microorganisms depending on the acquired image. According to the invention, said image is subdivided into sub-images and each sub-image is sub-divided into patches, and: A. for each patch, a microorganism feature extractor is applied, the feature extractor forming a convolutional part of a first convolutional neural network trained on patches annotated individually with at least one class; B. for each sub-image, a second neural network connected to the extractor is applied, the second neural network comprising an upstream pooling layer and one or more downstream layers comprising a layer for predicting at least one class, and being trained on training sub-images globally; and C. for the acquired image: F.a. calculating a feature vector calculated for the sub-images; F.b. applying a model for predicting at least one class for the microorganisms.

Description

[0001] METHOD AND SYSTEM FOR CHARACTERIZING MICROORGANISMS CONTAINED IN A COMPLEX SAMPLE

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to the field of microbiological analysis, in particular the automatic characterization of microorganisms contained in a complex sample such as blood. Without being limiting, the invention finds application to the characterization of the Gram, morphology and aggregation of bacteria, yeasts and fungi infecting a patient or an animal, in particular patients suspected of septicemia.

[0004] STATE OF THE ART

[0005] The field of microbiological analysis faces many challenges when it deviates from conventional techniques based on culture on or in a nutrient medium, for example, in Petri dishes. The detection of sepsis triggered by a bacterial infection of the blood is a well-known example. The presence of pathogenic bacteria in the blood can lead to the death of a patient, so their early detection is vital for the patient. Unfortunately, at an early stage, the quantity of bacteria is not only tiny, but they are also embedded in a complex matrix comprising white blood cells, red blood cells, platelets, and many other elements of significantly larger sizes and quantities, making their detection difficult, if not impossible, to date.This is why a blood culture, namely a step of growing the bacteria present in the blood, blood to which is added a nutrient medium chosen to accelerate their development as much as possible, is triggered for any patient suspected of sepsis. This growth aims to multiply the bacteria as quickly as possible in order to make them detectable. Unfortunately, blood culture is a process that is still too slow given the severity of the infection.

[0006] Sample complexity also challenges microbiological analysis in applications that were once routine in laboratories. This is particularly the case for the Gram characterization of bacteria, regardless of the sample, whether complex or not. Indeed, the Gram characterization of bacteria requires significant human expertise, being based on the microscopic analysis of Gram slides by a qualified technician. However, a significant loss of this expertise is observed in a significant number, even the majority, of analysis laboratories, leading to a loss of quality, or even leading to de facto no longer implementing this analysis. However, the latter, conducted very early once a sample is received, allows the clinician to make very rapid decisions on antibiotic therapy, sometimes decisive. The diffusion of machine learning techniques, and in particular deep learning based on neural networks, is a source of hope.Indeed, these techniques have been successfully applied to complex images, such as the detection of lung tumors. Similar progress is expected in microbiology: the characterization of microorganisms present in a sample from an image of the latter. However, this hope faces such a significant resource barrier that even a large organization has great difficulty developing an automatic characterization solution based on artificial intelligence. Indeed, due to the size of the bacteria, their number at an early stage, and the complexity of the sample, it is necessary to image a large surface area with very high resolution.In practice, this is achieved by combining several raw images of typically a few million pixels each, which are not necessarily overlapping or even contiguous, into a single composite image that we will simply call "image" hereinafter. The resulting image typically totals several tens, millions of pixels to several billion pixels. Even if convolutional neural network techniques reducing the dimensionality of the problem were used, the final dimensionality remains so important that learning classical architectures would require a very large number of annotated images, probably millions. In addition to this astronomical quantity, annotation cannot be carried out by non-specialists as is the case, for example, with models present on Google Zoo or models learned by uberization of annotation on dedicated platforms.However, as mentioned above in the case of Gram characterization, experts are lacking.

[0007] To date, therefore, the development of automatic characterization tools based on artificial intelligence in the field of microbial analysis, particularly in the field of In Vitro Diagnostics ("IVD"), remains difficult.

[0008] The aim of the present invention is to propose a method for the automatic characterization of microorganisms present in a sample based on the analysis by artificial intelligence of very high resolution images, this analysis being implemented by a deep learning architecture trained on a reduced learning set and allowing a high-performance diagnosis, in particular a diagnosis which can be qualified as In Vitro Diagnosis.

[0009] To this end, the invention relates to a method for classifying microorganisms contained in a sample, among several classes of microorganisms, the method comprising:

[0010] A. preparing a slide, in particular a microscope slide, comprising spreading the sample on said slide; B. acquiring at least one digital image of the slide with micrometric or sub-micrometric resolution;

[0011] C. the computer-implemented application of a model for predicting the class of microorganisms based on the acquired image.

[0012] According to the invention, said image is subdivided into a plurality of sub-images and each sub-image is subdivided into patches, and the application of the prediction model comprises:

[0013] D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class;

[0014] E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream layers comprising a prediction layer of at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated by at least one class; and

[0015] F. for the acquired image:

[0016] F. a. the calculation of a feature vector calculated for the sub-images;

[0017] F. b. applying a prediction model of at least one class for the microorganisms present in the sample based on the feature vector, said prediction model being trained on feature vectors calculated from training images.

[0018] In other words, the invention proposes an artificial intelligence architecture with several specific stages characterized by increasingly weak annotation along the stages. This architecture is trained with only a few hundred high-resolution sub-images subdivided into patches of reduced dimension (for example 256 pixels by 256 pixels or 224 pixels by 224 pixels), patches of which a tiny quantity (for example less than 5%) are strongly annotated, and this for an overall prediction accuracy of more than 90%.

[0019] According to one embodiment, the feature vector is a distribution of the scores calculated for the sub-images. According to one embodiment, each training image is subdivided into sub-images and each of said sub-images is subdivided into patches, and less than 50% of the patches of the training images are annotated, said annotated patches forming the training patches of the first convolutional network. In particular, less than 10% of the patches of the training images are annotated. According to one embodiment, the training of the first and second convolutional networks and the prediction model is configured to obtain a macro prediction specificity greater than or equal to 90%. According to one embodiment, the prediction model of step F is a “random forest” model. According to one embodiment, the upstream pooling layer is associated with an attention layer configured to apply a weight to the output of each extractor.According to one embodiment, the second network is a MIL-CNN network trained by batches of instances, the batches of instances consisting of the training patches.

[0020] According to one embodiment, the characteristic vector of step F. a comprises for each class:

[0021] - the maximum score among the sub-images; and / or

[0022] - the X ieme percentile of scores among the sub-images, with X greater than or equal to 90%, preferably equal to 95%; and / or

[0023] - the Y eme percentile of scores among the sub-images, with Y less than or equal to 10%, preferably equal to 5%; and / or

[0024] - the median score among the sub-images.

[0025] According to one embodiment, the feature vector further comprises the number of sub-images in the image.

[0026] According to one embodiment, the first convolutional network is pre-trained on images not comprising microorganisms and then trained on annotated training patches. In particular, the first pre-trained convolutional network is a VVG16 or ResNet network.

[0027] In one embodiment, the microorganisms comprise bacteria and the classes comprise at least Gram positive and Gram negative, and preparing the slide comprises preparing a Gram slide. In particular, the classes of microorganisms further comprise classes of morphotypes. In particular, the classes of microorganisms comprise a "neither bacteria nor yeast" class, a "Gram negative bacilli" class, a "Gram positive bacilli" class, a "Gram positive coryneum bacilli" class, a "Gram negative cocci" class, a "Gram positive cocci in chains" class, a "Gram positive cocci in clusters" class and a "yeast" class.

[0028] According to one embodiment, the sample comprises blood, in particular the sample is a positive blood culture. The invention also relates to a method for training a model for predicting a class of microorganisms among several classes of microorganisms from a digital image of a slide on which a sample likely to comprise microorganisms is spread, said training method comprising:

[0029] A. the creation of a training database comprising digital slide images annotated by one or more classes of microorganisms, the slide images being divided into a plurality of sub-images annotated by one or more classes of microorganisms and the training sub-images being subdivided into patches, at least part of the patches being annotated by one or more classes of microorganisms,

[0030] B. training a first convolutional neural network based on the annotated patches;

[0031] C. training a second neural network from the annotated base of sub-images, said second network comprising at least one patch feature extractor, a patch feature pooling stage, and a stage for predicting the class(es) of microorganisms present in the sub-image;

[0032] D. the creation of a database of distributions of the classes present in the training slides based on the classes predicted by the second neural network applied to the annotated sub-images;

[0033] E. of a model for predicting at least one class of microorganisms present in a slide based on spatial distributions of the classes, method according to which the predictor comprises the convolutional part of the first neural network, downstream of which is connected the second neural network, downstream of which is connected the prediction model.

[0034] In a variant of step C, the training method comprises training a second neural network comprising an upstream pooling layer and one or more downstream layers for predicting the microorganism class, the second network having the function of predicting the class(es) of a sub-image as a function of feature vectors produced by a patch feature extractor, the training of the second network being carried out as a function of feature vectors as a function of the annotated sub-images of the database and the feature vectors of the patches of the annotated sub-images

[0035] In particular, the training of the first network is carried out on the classes of microorganisms to which an “ambiguous” class is added, an annotated patch also being annotated by this class when objects in the patch in case of uncertainty about the objects present in the patch. In particular, the training of the second network comprises the training of a MIL-CNN network, the trained MIL portion of the MIL-CNN network constituting the second trained neural network.

[0036] The invention also relates to a method for predicting a class of microorganisms contained in a sample, from among several classes of microorganisms, the method comprising:

[0037] C. the computer-implemented application of a model for predicting the class of microorganisms in a sample spread on a slide based on an image acquired from said slide,

[0038] According to the invention, said image is subdivided into sub-images and each sub-image is subdivided into patches, and the application of the prediction model comprises:

[0039] D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class;

[0040] E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream prediction layers of at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated; and

[0041] F. for the acquired image: a. calculating a feature vector based on the scores calculated for the sub-images; b. applying a prediction model of at least one class for the microorganisms present in the sample based on the feature vector, said prediction model being trained on feature vectors calculated from training images.

[0042] The invention also relates to a system for predicting a class of microorganisms contained in a sample, among several classes of microorganisms, the system comprising a computer unit configured to implement a model for predicting the class of microorganisms in a sample spread on a slide as a function of an image acquired from said slide. According to the invention, said image is subdivided into sub-images and each sub-image is subdivided into patches, and the application of the prediction model comprises:

[0043] D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class;

[0044] E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream prediction layers of at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated; and

[0045] F. for the acquired image: a. the calculation of a characteristic vector calculated for the sub-images; b. the application of a prediction model of at least one class for the microorganisms present in the sample as a function of the characteristic vector, said prediction model being trained on characteristic vectors calculated from training images.

[0046] The invention also relates to computer program products comprising a computer memory storing computer-readable instructions for implementing steps D to F above or steps A to E above.

[0047] The invention also relates to a method for predicting a class of objects contained in a digital image from among several classes of objects, the method according to which the digital image is subdivided into a plurality of sub-images and each sub-image is subdivided into patches, and according to which:

[0048] D. for each patch, applying an object feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising objects, said objects being individually annotated by at least one class;

[0049] E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream prediction layers of at least one class, in the form of a score, said second network being trained on training sub-images comprising objects, each training sub-image being globally annotated; and

[0050] F. for the acquired image: a. the calculation of a characteristic vector calculated for the sub-images; b. the application of a prediction model of at least one class for the objects present in the sample as a function of the characteristic vector, said prediction model being trained on characteristic vectors calculated from training images.

[0051] BRIEF DESCRIPTION OF THE FIGURES

[0052] The invention will be better understood from reading the following description, given solely by way of example, and drawn up in relation to the appended drawings, in which:

[0053] - figure 1 illustrates the classes of microorganisms present on a Gram slide predicted by the invention;

[0054] - Figure 2 is a flowchart of a microbiological analysis laboratory workflow implementing the invention;

[0055] - Figure 3 is a schematic view of a Gram slide image acquisition system;

[0056] - figure 4 is an illustration of the different scales of a Gram slide image used by the invention;

[0057] - Figure 5 is a flowchart detailing the operation of the three-stage prediction model according to the invention based on the different scales of Figure 4;

[0058] - Figures 6A-C are schematic views of patch feature extractors, comprising the convolutional part of a convolutional neural network, in this example a network with VGG16 architecture;

[0059] - Figure 7 is a schematic view of a predictive model at the sub-image level, formed in the example of the MIL part of a MIL-CNN network with attention mechanism;

[0060] - Figure 8 is a schematic view of the predictive model at the image level, based on a Random Forest type prediction stage analyzing a distribution of descriptors provided by the prediction stage at the sub-image level; Figure 9 A is a schematic view illustrating the display of the presumptive microorganism zone in the image of the Gram slide, or in one or more sub-images constituting it as a function of the weights calculated by the attention mechanism of the predictive model at the sub-image level, and the display of the predictions of the predictive model operating at the patch level; - Figure 9B illustrates two sub-images analyzed by the convolutional neural network from which the feature extractor of patches constituting the sub-images is derived, and the predictions of said neural network displayed in the form of "heat maps";

[0061] - Figure 10 is a flowchart illustrating the training of the three-stage predictive model according to the invention;

[0062] - Figures 11A and 11B illustrate the classification performance of Gram slide images in RGB imaging carried out by the three-stage predictive model according to the invention;

[0063] - Figures 12A and 12B illustrate the classification performance of Gram slide images in holographic imaging carried out by the three-stage predictive model according to the invention; and

[0064] - Figures 13 to 15 illustrate different computer architectures for implementing the three-stage predictive model according to the invention and the workflow of a microbiological laboratory using the latter.

[0065] DETAILED DESCRIPTION OF THE INVENTION

[0066] A. METHOD AND SYSTEM FOR CHARACTERIZING THE GRAM OF BACTERIA IN A GRAM SLIDE

[0067] An embodiment of the invention will now be described, namely a microbiological laboratory workflow for the IVD characterization, in particular of the Gram of bacteria in a patient suspected of septicemia, workflow based on the analysis of the RGB image of a Gram slide produced from a positive blood culture.

[0068] In particular (Figure 1), this embodiment comprises the automatic flat classification of a Gram slide image 10 into eight mutually exclusive classes, namely a slide comprising neither bacteria nor yeast, a slide comprising Gram-negative bacilli, a slide comprising Gram-positive bacilli, a slide comprising Gram-positive coryneae bacilli, a slide comprising Gram-negative cocci, a slide comprising Gram-positive cocci in chains, a slide comprising Gram-positive cocci in clusters and a slide comprising yeast.

[0069] Referring to Figures 2 and 3, this workflow 20 begins with the production of a sample 22 from a blood sample from the patient, here a positive blood culture for example carried out using a bottle of BACT / ALERT® medium cultured in the applicant's BACT / ALERT® VIRTUO® system. As is known per se, blood culture consists of multiplying the number of bacteria and yeasts initially contained in a blood sample in order to make them detectable (identification of the state "presence" or "absence" of bacteria or yeast) and to facilitate their subsequent characterization due to a greater biomass. To do this, the blood is mixed with a culture medium for example based on soybean trypticase, supplemented with absorbent polymeric beads. The blood culture sample is therefore complex in that it includes the elements naturally present in the blood (red blood cells, white blood cells, platelets, fibrogens, etc.)) as well as elements specific to blood culture. Although the number of bacteria and yeasts has been multiplied, the latter can still constitute a tiny part of the sample and be masked by, or confused with, heterogeneous elements, particularly in number, shape, size and colorimetry.

[0070] The method continues, at 24, with the production of a Gram slide from the positive blood culture. This production comprises the spreading ("smearing"), at 240, of a fraction of the blood culture so as to obtain a spreading thickness preferably less than 10 μm, this thickness corresponding to the depth of field of a microscope with a magnification of 1000 used subsequently in the workflow. By adjusting the thickness to said depth of field, only a two-dimensional inspection of the spreading is necessary, thereby facilitating the analysis as described below. Once spread, the sample undergoes Gram staining as known per se, for example carried out automatically using the PREVI® COLOR GRAM instrument marketed by the Applicant, this staining having the purpose of staining the bacteria differently depending on their Gram.

[0071] Once dried and covered with a coverslip, the Gram slide is positioned, at 26, in a microscope 40 (figure 3) with a high-magnification oil immersion objective 42, between 60x and 100x, coupled with a Köhler-type illumination system 44 (illumination with white incoherent light in at least the range [400nm-900nm]), and with an RGB imaging system 46. The system 46 comprises, for example, a two-dimensional CMOS photosite sensor, a sensor covered with a Bayer matrix for producing color images in a manner known per se, a sensor placed in the image plane of the objective 42. The optical system of the microscope and the imaging system are chosen and / or controlled so that bacteria and yeasts, objects of a few hundred nanometers to a few tens of micrometers, represent at least 5 pixels, preferably at least 10 pixels in the image obtained.

[0072] The field of vision of such an objective being limited, the Gram slide 48 is advantageously placed on a mobile support 50 movable by means of a piezoelectric motor plate 52, allowing the movement of the slide in the plane (x,y) perpendicular to the optical axis z of the objective 42. This plate is connected to a computer unit 54 also connected to the sensor 46, a computer unit which coordinates the movement of the support 50, and therefore of the slide 48, and the taking of images by the sensor 46, in order to obtain an image of the entire slide 48, at least the entire surface of the slide on which the sample 56 is spread. Preferably, the raw images acquired from the slide partially overlap in order to avoid edge effects, for example bacteria, clusters or chains of cut bacteria.

[0073] The collection of images can be kept to form the set of "sub-images", this collection corresponds to an "image", as described below, or a single image of the slide can be obtained by computer reconstruction in a manner known per se, or this collection of images is re-divided into "sub-images". According to the invention, three scales are obtained for a Gram slide: an image (composite or not), sub-images constituting the image, and patches constituting the sub-images. This collection of digital images, and preferably each of these digital images, covers a sufficiently large surface area of ​​the slide to be a priori representative of the population of microorganisms present in the initial sample, such that the class(es) of microorganism(s) present on the slide are a priori present somewhere on the image.

[0074] Referring again to Figure 2, a microscopic or submicroscopic resolution image of the slide (in this example, a high resolution composite image, or “ICHR” image) is therefore produced at step 28 of the workflow. As an illustration, for Gram slides conventionally used in the laboratory, with a dimension of 25mm by 75mm, this results in an image of at least several tens of millions of pixels, with the inventors’ prototype 150 million pixels, with a lateral dimension of the pixels corresponding to 40nm, each pixel being coded on 16 bits for each color. In particular, the acquisition system 40, 46 produces a few hundred pixels for a bacillus of the order of a micrometer.

[0075] The laboratory workflow 20 continues with an automatic step 30, implemented by computer, of analyzing the image produced in order to characterize at least the Gram of the bacteria present therein, and more specifically to predict the class(es) of the Gram slide among the classes described previously. In particular, a “multi-class” type prediction step 300 having a three-stage architecture as described below is implemented. This prediction generates a vector whose components correspond to the probabilities of the classes, at least scores between 0 and 1, the sum of which is equal to 1, as well as a confidence index (“TC”) associated with the prediction. The specificity of the prediction is optimized so that if the IC index exceeds a predetermined confidence threshold (step 302), no further characterization test of the slide is necessary.The result of the prediction can thus be pushed directly to the clinician (step 32), for example by means of a report received by email or by means of a notification on a smartphone or equivalent, who can on this basis choose an appropriate antibiotic therapy to administer to the patient (step 34).

[0076] Advantageously, but optionally, the prediction architecture includes an attention mechanism described below. In addition to increased interpretability of the prediction, this mechanism makes it possible to identify areas of the image of the Gram slide likely to contain microorganisms. Thus, where the mechanism has focused its attention, i.e. produced a higher weighting, presumptive areas of the presence of microorganisms are displayed, at 304, on a computer screen superimposed on the image of the Gram slide, intended for a laboratory technician expert in slide interpretation. The latter, aided by the display, can then manually characterize the slide (step 36). Optionally, or in addition, the technician can also observe the slide directly through the microscope and characterize it in a conventional manner.Note that this display can be carried out at the level of the entire image, or broken down, for example on each of the sub-images to facilitate reading.

[0077] Advantageously, but optionally, the prediction architecture also includes an intermediate prediction step at the level of the patches that make up the image and its sub-images. Here again, the predictions of the different patches can be presented as an overlay on the image to the laboratory technician, which not only allows his attention to be directed to the presumptive areas, but also provides presumptive elements on the class of microorganisms present in the different areas of the image. Here again, a decomposition of the display into sub-images is possible.

[0078] As described below, the prototype developed by the inventors, trained on fewer than 600 slides, has a macro accuracy of 95% for a slide rejection rate of 17%. This prototype, which can be greatly improved, already makes it possible to automate the Gram reading of a large proportion of blood culture samples, while providing information beyond simple Gram (bacillus, coccus, cluster, chain, yeast, etc.). The time saved with such automation effectively compensates for the scarcity of human expertise in this field.

[0079] B. MULTI-STAGE PREDICTION ARCHITECTURE FOR AUTOMATIC ANALYSIS OF COMPLEX IMAGES, ESPECIALLY GRAM SLIDES

[0080] An embodiment of the multi-stage architecture of a predictive model (or "predictor") according to the invention will now be described in more detail. Although this architecture is described in relation to the analysis of an RGB image, this architecture can also be implemented for the analysis of images of a different nature, in particular holographic as will be described below, or even multispectral or hyperspectral.

[0081] The invention takes advantage of the very high resolution of the ICHR image, the latter being able to be divided into several tens, preferably at least a hundred, of sub-images, each sub-image being able to be subdivided into several tens of patches, each patch having dimensions in pixels sufficient to contain a microorganism or a morphotype (cluster, chain, etc.) in its entirety. Such a subdivision is illustrated in Figure 4 which describes a regular subdivision of the image into sub-images and of the sub-images into patches. In particular on the sub-image and the patch illustrated, bacilli correspond to the dark objects, and are of dimensions smaller than those of the patch, and are contained entirely in the patch. For example, the patches have a dimension of 256 pixels by 256 pixels (or 224 by 224) corresponding to a real surface of approximately 10 x 10 μm 2whereas the typical maximum dimension of bacteria and yeasts is of the order of a micrometer, and the sub-images have a dimension of 1024 pixels by 1536 pixels. In a preferred variant of the invention, neighboring patches partially overlap in order to avoid edge effects when training the predictor at the patch level. Similarly, neighboring sub-images partially overlap for the same reason. Alternatively, the sub-images do not cover the entire ICHR image and do not overlap in order to better cover the diversity linked to the inhomogeneity of the spreading.

[0082] As detailed below in relation to the training of the stages of the multi-class predictor, this subdivision, made possible by the very high resolution of the ICHR image, is associated with an annotation of decreasing strength from the patches towards the ICHR image. In particular, for a database of training ICHR images subdivided into sub-images and each sub-image into patches: i. patches are strongly annotated for each sub-image at least by the class(es) of microorganisms it contains. In particular, an annotated patch is advantageously, but optionally, associated with a double annotation: a first global annotation corresponding to the class(es) of microorganisms it contains as well as a second annotation for each of its pixels coding the presence or absence of microorganism; ii. each sub-image is globally annotated by the class(es) of microorganisms present; iii.each image is globally annotated by the class(es) of microorganisms present. According to the invention, a model extracts characteristics of the patches by a class prediction model at the patch level, patch characteristics which are transmitted to a second stage which extracts characteristics of the sub-images by a class prediction model at the sub-image level, sub-image characteristics which are transmitted to a third stage which makes a final class prediction of the microorganisms present in the image, and therefore in the Gram slide.

[0083] In the following, an ICHR image is subdivided into a two-dimensional matrix of sub-images referenced by the indices (i,j) and each sub-image is subdivided by a two-dimensional matrix of patches referenced by the indices (k, 1).

[0084] Referring to Figure 5, a particular embodiment of the predictor 60 comprises:

[0085] A. an analysis flow 62 for each sub-image of an ICHR image, each flow producing a vector of scores, one score per class, (denoted “Class scores i, j)” for the sub-image of coordinate (i, JJ). The analysis flows 62 are carried out independently and are identical in terms of computational modules;

[0086] B. a global analysis stream 64 receiving the characteristic vectors of each stream 62, in a preferred variant illustrated in FIG. 5 the score vectors, and producing a score vector, a score per class, for the ICHR image, noted “Class image”, as well as a confidence index for this prediction, noted “Conf_index”.

[0087] Each stream 62 for analyzing a sub-image comprises a feature extractor 66 preferably consisting of the convolutional part of a convolutional neural network (Figure 6A). Alternatively, the extractor comprises the convolutional part of a neural network, followed by a flattening layer (Figure 6B). Alternatively, the extractor comprises the convolutional part of a neural network, followed by a flattening layer followed by one or more fully connected layers (Figure 6C). Optionally, these variants are completed downstream by one or more fully connected neural layers, the feature vector corresponding to the output of the last connected layer.

[0088] This extractor has the function of extracting characteristics from each patch constituting the sub-image, noted “Emb”, which summarize the information contained in the patches, notably in terms of the presence or absence of microorganisms.

[0089] Referring to Figure 6A, this convolutional neural network 66_CNN is trained on a database of annotated training patches 66_BDD, the network 66_CNN having the function of predicting the class(es) of microorganisms present in the patches. In a preferred embodiment of the invention, the neural network 66_CNN is a network of the VGG16, ResNet, MobileNetV2, or efficientNet type, preferably pre-trained on public databases such as available at the URL https: / / www.image-net.org, in particular a VGG16 network trained on the basis of ImageNet “ImageNet Large Scale Visual Recognition Challenge (ILSVRC)” (https: / / www.image-net.org / challenges / LSVRC / index.php).Although it is possible to take a blank network, i.e. initialized with random weights, and train it ah initio with annotated patches, the use of a pre-trained network allows an increase in the macro accuracy of the prediction of the class(es) of the ICHR image of several percent. Among the dozens of pre-trained networks tested by the inventors, a VGG16 is the most efficient, leading to an increase in macro accuracy of approximately 5% compared to a network trained only on slide image patches. Since this is a network with a fixed architecture, the dimension of the patches from the ICHR images is chosen to be identical. This dimension satisfies the condition on the dimensions of microorganisms, bacteria and yeasts, as described above.

[0090] In an advantageous but optional variant, the predictions rendered by the convolutional neural network 66_CNN can be used to identify the areas of the sub-images where a particular class is likely to be present. These results can be presented in the form of a "heat map" and superimposed on the image. A sub-part 66 of the network 66_CNN is then extracted, with its parameters. This part 66 can include the convolutional layers of 66_CNN, and possibly some of the fully connected layers located downstream. It is also possible, optionally, to add new fully connected layers downstream. This element 66 functions as an extractor of features of the patches, features relevant for the prediction of the classes concerned.

[0091] Once the characteristics have been extracted from each patch, the corresponding characteristic vectors are communicated to a second prediction stage 68 based on neural networks implementing a multi-class prediction of the class(es) of microorganism present in the sub-image consisting of the patches. More specifically, as illustrated in FIG. 7, this second stage 68 is the “MIL” (for “Multiple Instance Learning”) portion of a MIL-CNN type network whose characteristic extractor part comes from the 66_CNN model, advantageously the extractor 66.

[0092] The second predictor 68 comprises: i. an upstream stage 70 implementing a gated attention mechanism. This stage 70 calculates for each feature vector from the patches (denoted Embi j(k, Z) for the patch with coordinates (fc, Z) in the sub-image with coordinates (Z, j)), a coefficient ai (k, Z) measuring the weight of the patch in the class prediction of the sub-image, weight that the stage 70 multiplies with the corresponding vectors to produce a new feature vector a t j(k, Z) x Embi j(k, l) for each patch. In a preferred, but not mandatory, embodiment, this weight is calculated according to the relationship:

[0093] Where EU IR ÇxR and WE are matrices forming parameters of the predictor 68, Q being the dimension of the feature vectors Embij^k, !), R a predetermined positive integer, for example equal to 512, sigm the sigmoid non-linear function, and O is the term-by-term multiplication operator. Such a mechanism is notably described in the article by M. Use et al., “Attention-based Deep Multiple Instance Learning”, arXivfl 802.04712v4 [cs.LG], 28 Jun 2018. Other attention mechanisms are however possible. ii. downstream of the attention mechanism stage 70, or integrated into the latter, a pooling stage 72 reducing the overall dimensionality of the feature vectors. For example, the stage 72 produces for each sub-image (i, j) a unique feature vector Z(i ) from the K x L vectors from the patches according to the relation: iii. downstream of the pooling stage 72, a stage with layers of fully connected neurons 74, receiving the vector Z(ij) as input and producing as output a vector of prediction scores Class_scores(i,j The scores are normalized, between 0 and 1, and by convention, the higher a score, the higher the probability that the corresponding class is present in the sub-image. In the modality considered, this downstream stage can be reduced to a sigmoid type layer which produces the scores, but if necessary other layers can be inserted between the pooling stage and the stage for obtaining the scores.

[0094] In a preferred embodiment, the model 66 and its downstream part 68 are thus integrated into a new predictor which operates at the sub-image level, and which is the one implemented in the workflow 62 of Figure 5. As described in the article by Use et al., this new predictor can be learned on examples of sub-images, in particular by conventional backpropagation techniques. Preferably, but optionally, the parameters of the part 66 of the model will be left free to evolve during this learning, so that at the end of this process they will no longer have their value resulting from the learning of the model 66_CNN. By this method, the feature extractor 66 and the downstream layers 68 are co-optimized.

[0095] Referring to Figure 8, the last stage 64 of the predictor 60 comprises: i. an upstream stage 80 receiving descriptors, for example from the prediction score vectors Class_scores(i,j) of all the sub-images constituting the image of the Gram slide, and calculating a distribution of the scores on the slide for each of the classes from these vectors. This stage 80 takes advantage of the large number of sub-images constituting the ICHR image due to the high resolution of the latter. More particularly, the ICHR image being made up of several tens, or even a hundred or more, sub-images, it is thus possible to calculate in a statistically relevant manner a distribution of the scores of said classes present in the ICHR image. An advantage of calculating a distribution is to obtain a characterization of the ICHR image independent of the position of the microorganisms in the image while taking into account the entire image.In particular, for applications as sensitive as in vitro patient diagnostics, especially for sepsis, it is questionable to provide a result based on a single or limited area of ​​the slide. Indeed, the prediction based on a single sub-image can be erroneous or too uncertain.

[0096] For example, for each class of microorganism, stage 80 calculates an approximation of this distribution consisting of the extraction of the following statistics: the maximum of the scores, the 95 ieme percentile the median score the 5th ieme percentile

[0097] This approximation is quick to calculate and allows the overall performance of the prediction to be adjusted through each of its components. In particular, the maximum as well as the 95 ieme percentile allows you to adjust the sensitivity level of the prediction of the classes of microorganisms actually present (here the 95 iemepercentile is chosen, but other values ​​above 50% are possible depending on the desired sensitivity). The median value helps to mitigate prediction errors made at the sub-image level. The 5 ieme percentile allows you to adjust the sensitivity of the prediction concerning the “no microorganism” class (here the 95 iemepercentile is chosen, but other values ​​lower than 50% are possible depending on the desired sensitivity). It should be noted that simpler distribution approximations are also within the scope of the invention (for example only the maximum score for each class) as well as more complex approximations (such as for example a polynomial interpolation of the scores or the approximation by a distribution characterized by its equations such as the Fisher, Gauss, Bernoulli law...). In particular, the advantage of obtaining descriptive statistics is to be able to take into account the specificities of the application, known to the microbiology expert, as illustrated by the choice of percentiles to help adjust the sensitivity and the sensitivity which are two important criteria in IVD diagnosis. ii.a downstream stage 82 receiving each of the distributions of the classes and the number of sub-images / x / , and predicting at output the class(es) Class mage of microorganisms present in the ICHR image as well as a confidence index Conf _index of this prediction. More particularly, the prediction implemented by stage 82 is a multi-class prediction by automated learning, and preferably a prediction not implementing a neural network. Indeed, the characteristics received by this stage are structured with a determined, fixed number of characteristics of known nature. Also, the “classic” approaches to automated learning (i.e. not based on a neural network), such as approaches based on SVM (“support vector machine”), K nearest neighbors, decision trees to name a few, are more suitable in terms of performance, interpretability and ease of training.Among all these approaches, stage 82 preferentially implements a “Random Forest” type prediction that is particularly effective in avoiding over-fitting and in delivering an interpretable confidence index IC (for example equal to the maximum percentage of votes for each class). Indeed, as the inventors have noted for the interpretation of complex images, the confidence index of the Random Forest takes a high value for precise predictions (i.e., consistent with the reality of the microorganisms present in the slide) and collapses in the opposite case. The choice of a threshold value such as used in stage 302 (figure 2) for an application as sensitive as in vitro diagnosis is thus facilitated and robust. The Random Forest type predictive model is for example that described in the article by Breiman et al., “Random Forests”, Machine Learning, 45(1), 5-32, 2001.Note that the descriptors received by model 80 are not necessarily derived from the prediction scores returned by predictor 68. In particular, it is possible to exploit descriptors from layers located further upstream in its architecture.

[0098] The different predictive models (patch, sub-image, or image level) described above are “multi-class” predictive models. Alternatively, it is possible for the patch model and / or the sub-image model and / or the image model to implement here a “multi-label” prediction, i.e. of the presence / absence type for each of the classes considered, and not presence of only one of the classes (the negative class then being treated as a separate class). Multi-label prediction makes it possible to deal with certain special cases, such as that of poly-microbial samples presenting several types of Gram. The advantage of choosing multi-class predictive models is to reduce the number of annotated training data and / or the collection campaign of Gram slides or corresponding samples.Indeed, in the context of microbial infections, the poly-microbial case, for example, is largely in the minority compared to mono-microbial infections, so that there is much less associated data, which makes it more difficult to train multi-label predictive models.

[0099] Referring to Figure 9A, a preferred use of the output results of the predictor 60 following the analysis of a slide image 90 by the latter comprises the display on a computer screen 92 of the image 90 whose light intensity is adjusted patch by patch according to the coefficients ai j(k, l). In particular, the higher the weight of a patch, signifying its high importance in the prediction at the sub-image level, the brighter the patch, as illustrated by the patch 94 which is much lighter than the other areas of the image 90. In this way, a laboratory technician can, if he wishes, check the content of this patch in terms of microorganisms present and confirm or deny its content. The technician can thus display the entire image or, preferably for reasons of readability, a particular sub-image as illustrated in this figure.Preferably, the predicted class(es) are also displayed, coded in the Class mage vector, the associated confidence index Conf_index, and a signal coding the failure of the prediction when the index is lower than the threshold. The brighter patches then act as presumptive areas likely to contain microorganisms. This aid then allows the technician to carry out his analysis more quickly by focusing on these areas first.

[0100] Advantageously, as illustrated on the second display screen of Figure 9A and the images of Figure 9B, the sub-images are also analyzed by the convolutional neural network 66_CNN whose predictions can be displayed on a screen, advantageously in the form of “heat maps”, so that the technician can directly know which areas of the image are likely to correspond to microorganisms, and to which class these correspond.

[0101] C. DATA ANNOTATION AND TRAINING OF THE MULTI-STAGE PREDICTION ARCHITECTURE

[0102] C.1. DATA ANNOTATION

[0103] Three annotation scales are performed: at the level of the entire image, to which the blade corresponds; at the level of the sub-image; and at the level of the patch.

[0104] Let us first consider the slide level, which is the relevant level from a biological and medical point of view. Each image a priori covers a sufficiently large area to account for the contents of the slide, and of the initial sample. Therefore, any available knowledge about the contents of the sample or slide can be used to produce an annotation at the image level.

[0105] Preferably, the Gram slides come from microbiological analysis laboratories, produced and annotated under real conditions by technicians specializing in Gram analysis. However, the interpretation of complex slides can be difficult, even for an experienced technician. Preferably, a portion of the sample used to produce a slide is characterized in greater depth. In particular, the actual identity of the microorganisms present in the sample is determined, preferably by MALDI-TOF mass spectrometry, using a VITEK® MS marketed by the Applicant. The image of the Gram slide first inherits the annotation of the slide made in the laboratory, referred to as “raw”. This raw annotation is then verified by a second expert and the annotation errors and ambiguities are corrected thanks to the additional characterizations of the microorganisms.The final annotation of the slide image is then referred to as the “ground truth”. In addition to correcting the raw annotation, a first sorting of the slides is carried out: those comprising a polymicrobial mixture or Campylobacter are removed from the training set. In order to improve the overall performance of the predictor according to the invention, Acinetobacter are classified in the “Gram-Negative Bacillus” class. Other characteristics of the samples (metadata) are also collected, such as, where applicable, the type of culture medium included in the sample, the time elapsed to obtain positivity of the blood culture, the time elapsed for the revelation of the Gram stain, or an antibiogram of the microorganisms. We now describe the annotation at the sub-image level.Each sub-image of an image receives by default the "ground truth" annotation of the latter, but this annotation can be modified by the second expert, for example in the case where the microorganism present on the slide is absent from the sub-image considered. To do this, to characterize the image of the slide in its entirety, the second expert goes through each zone of it and therefore each sub-image. Preferably and optionally, when the second expert is unable to unambiguously annotate a sub-image, the latter is excluded from the training set. Preferably and optionally, the "ground truth" annotation of the sub-images is modified and reduced to a single class, so that we can then restrict ourselves to a "multi-class" type model, the rare sub-images presenting two classes simultaneously being excluded from the training set.Preferably and optionally, in the particular case of bacteria with several morphs (for example, Gram-Variable Bacilli such as Bacillus sublilis, which is taxonomically a Gram-Positive Bacillus, but which often has the appearance of Gram-Negative Bacilli), the annotation at the sub-image level may be modified to reflect the actual appearance of the bacteria, so that the sub-image annotation may be different from the image annotation. This provides an annotation for all or part of the sub-images, which are typically a hundred times more numerous than the number of images in the database. The sub-images are strongly annotated since the designer of the predictor according to the invention injects knowledge, therefore additional a priori information, into the annotation of the sub-image. This annotation will be used to improve the training of the predictor.

[0106] Following the same logic, an even stronger annotation can be produced at the patch level, on at least part of the dataset. In one embodiment of the invention, this annotation is carried out by first performing a semantic segmentation of the microorganisms, i.e. by associating a Gram type with each of the pixels of a sub-image. Various methods are possible to carry out this semantic segmentation step. For example, a semi-automated segmentation can be carried out using a tool such as Ilastik described in the article by S. Berg et al. “ilastik: interactive machine learning for (bio)image analysis”, Nature Methods, (2019) and available at the URL https: / / www.ilastik.org / , associating a microorganism status or not with each pixel. The annotation of the sub-image can then be used to associate a class with the microorganism pixels.A final manual step performed by an expert finally allows to correct the segmentation masks obtained, and also to point out ambiguous areas. Once the semantic segmentation is carried out, an explicit algorithm allows to assign to each of the patches of the sub-image a class, according to the number of pixels of the class in question present on the patch. Just as the sub-image annotation is not necessarily identical to the image annotation, the patch annotation is not necessarily identical to the sub-image annotation. In particular, when certain classes are linked to the state of organization of the microorganisms (chains, clusters), such an organization is not always detectable at the level of a patch of reduced extent, also in the case where the number of pixels of microorganisms is not sufficiently large on a patch, it can convert the annotation in such a way that it no longer reflects the state of aggregation.For example, this process will convert the class "Gram Positive Cocci Aggregated in Chain" to "Gram Positive Cocci - indeterminate aggregation state". Finally, a new "ambiguous" class can be assigned to a patch either when the pixels that compose it have been annotated as such by an expert, or when the number of pixels associated with a microorganism is very low (which generally corresponds to an organism seen partially because it is located at the edge of the patch).

[0107] Once this process is implemented, we obtain a patch-level annotation for all or part of the available dataset. Note that the number of patches, in the modality considered, is typically of the order of a few tens or hundreds per sub-image. The patches are annotated very strongly since the designer of the predictor according to the invention injects additional a priori information into the patch annotation. This annotation will be used to improve the training of the predictor.

[0108] The slide images are stored with their “ground truth” annotation in a computer memory for further processing. Similarly, the sub-images and their annotation are stored in a computer memory. Similarly, the patches with their patch annotations are stored in a computer memory for further processing. Three databases are thus created.

[0109] The process described above produces an annotation with three increasing strength levels: image level, sub-image level and patch level. In doing so, we considerably strengthen the overall strength of our annotation on the training set. However, given the number of images available (737 in the implementation example considered) and their dimensionality (more than 100 million pixels in the example considered), directly training a predictor using only image-level annotation would most likely be doomed to failure.

[0110] C.2. TRAINING THE MULTI-STAGE PREDICTOR

[0111] A method for training the predictor 60 is now described. According to this method, each stage is trained independently of the others while relying on the same training Gram slide images. Referring to FIG. 10A, a method 100 for training the predictor 60 comprises constituting, at 102, a database of training Gram slide images, annotated by the class or classes of microorganisms present, in particular a database of annotated images, annotated sub-images and annotated patches in the manner described above.

[0112] The method continues with the training, at 104, of the convolutional neural network 66_CNN to predict the patch classes. This training can begin with an untrained network. However, as described previously, an initial pre-trained network is preferably selected and then re-trained on the basis of annotated patch data. In this option, depending on the pre-trained network chosen, a normalization of the pixel values ​​of the patches can be implemented. In particular, certain networks accessible in network libraries, in particular those based on the “TensorFlow” software libraries, are trained on images whose pixel values ​​are standardized. In this case, upstream of the network, 66_CNN, a normalization of the patches is provided, for example a standardization (centering of the data on 0, and of the standard deviation at 1).This normalization stage is then optionally provided in the predictor according to the invention, upstream of the characteristic extractor 66.

[0113] The annotated patch database is split into a training set and a test set, ensuring (a technique called "blocking") that all patches from a given image are either all added to the training set or all added to the test set. The optimization of the hyperparameters of the 66_CNN network is carried out on the training set by a cross-validation technique, for example 4 "folds" created using the technique called "blocking" (or "blocked cross-validation"), which guarantees in particular that all patches associated with a given image are positioned in the same "fold". The value of the hyperparameters is for example optimized using a grid search technique, a random search technique, or a Bayesian method.

[0114] In a variant, patches that have been annotated as "ambiguous" are kept for training and testing, and the learning includes an additional "ambiguous" class, this class grouping the patches for which the annotating expert is not certain of the microorganisms present.

[0115] Once the CNN network 66_CNN is trained, the convolutional part is extracted, with the last layers being removed from the network.

[0116] In a first variant of the training of the predictive model of the sub-image level, the extractor is integrated into a MIL-CNN type model by a transfer learning process. This network is for example the one described in the article by M. Use et al. For example, the source code, as well as the training source code, of the network are those accessible on Github at the address https: / / github.com / AMLab-Amsterdam / AttentionDeepMIL. The process continues, in 108, by the training of the MIL-CNN network, the sub-image base being split into a training set and a test set, ensuring the same distribution of images between training and test sets as for training the patch model. The value of its hyperparameters is also optimized by cross-validation, for example using a grid search technique, a random search technique, or a Bayesian method.Preferably, these hyperparameters comprise the architecture of the fully connected neuron layer stage 74 (e.g. number of layers, number of neurons per layer, etc.). In this variant, the convolutional part and the MIL part of the trained MIL-CNN model may respectively constitute the extractor 66 at the patch level and the prediction model 68 at the sub-image level. Alternatively, the MIL portion is extracted to form the model 68 and the extractor is the one trained in the previous step.

[0117] In a second variant of training the predictive model at the sub-image level, only the MIL portion of a MIL-CNN network is trained. For example, all the patches of all the sub-images are processed by the trained extractor 66 so as to obtain sets of corresponding feature vectors, each annotated by the annotation of the corresponding sub-image as illustrated in FIG. 10A. These sets of feature vectors and their annotations are stored in a database 106. The model 68 is then trained on this basis according to a cross-validation technique as described previously.

[0118] Once the MIL stage 68 has been trained, each batch of sub-images corresponding to an image is processed, at 110, by this stage 68 and the distribution calculation module 80 in order to produce score distributions, distributions which are stored with the corresponding image annotations in a database. Then, at 112, the last stage 82 of the Random Forest-based predictor is trained, the distribution base being split into a training set and a test set, again with the same distribution of the blades as previously. The optimization of the hyperparameters is again carried out by cross-validation with 4 folds. The value of its hyperparameters is for example optimized according to a grid search technique, the number of hyperparameters being more limited than in the context of the previous predictors.

[0119] Optionally, the method continues, at 114, by calculating a performance criterion of the stage predictor according to the invention, for example the overall accuracy, as a function of a rejection rate of the slides. In particular, this calculation involves processing all of the slide images by the predictor and varying the confidence threshold of step 302 (figure 2). The slides not passing the threshold are discarded, the rejection rate being equal to the percentage of discarded slides, and the overall accuracy is calculated for the remaining slides. According to the invention, if the rejection rate is deemed too high or the accuracy too low, the method continues by acquiring new slides and / or additional annotation of patches and retraining the predictor as described previously in order to reduce the rejection rate and increase the accuracy.

[0120] Figure 10B illustrates an example of accuracy vs. rejection rate for the prototype of the stage predictor designed by the inventors after several iterations. In this example, by targeting an accuracy greater than or equal to 95%, the rejection rate obtained is 17%, meaning that this prototype, if used as is in combination with the image acquisition system described in Figure 3, can automatically process, without additional testing, more than 80% of the Gram slides. These performances were obtained on an initial data set of 737 Gram slides, each divided into 100 sub-images, each sub-image being divided into 24 patches. Out of a total number of patches of nearly 1.8 million, only 3% of these, or approximately 50,000, were strongly annotated, while the total number of parameters of the predictor according to the invention is approximately 15 million.

[0121] Figure 11A presents in more detail the performances obtained by the prototype of the predictor according to the invention, trained on the set of Gram slides previously described, performances obtained on all the slides (i.e. with a rejection rate set to 0%). Figure 11B presents the corresponding confusion matrix.

[0122] D. APPLICATION OF THE INVENTION TO SIGNALS OTHER THAN RGB SIGNALS

[0123] An application of the invention to images whose pixel values ​​are coded on three color channels (RGB) has been described. The invention applies to other types of signals, in particular digital images whose pixel values ​​code holographic information, on one or more wavelengths.

[0124] For example, the Gram slide image acquisition system is that described in patent applications EP4307051 and EP4172699, connected to a calculation module for the reconstruction of a refocused image.Regardless of the particular variants described in these applications, the principle of holographic imaging is to illuminate the slide with coherent light according to one or more wavelengths, to record the corresponding intensity images and to produce by computer reconstruction (so-called "parametric" reconstruction as for example described in these applications or so-called "non-parametric" reconstruction as described for example in applications WO2016075279, WO20 17077238 or WO 17207184) a digital holographic image of the slide in which each pixel is coded by an intensity value and a phase value for each illumination wavelength (in an embodiment described in the two applications EP4307051 and EP4172699, eight in number, each pixel thus being coded on 16 channels).

[0125] In a first variant, the feature extractor 66 comes from a convolutional network 66_CNN which is not pre-trained, which makes it possible to take all the channels as input. In a second variant, this extractor comes from a pre-trained convolutional network, in particular on RGB images, as described previously. In this option, upstream of the extractor 66, a dimensionality reduction stage is provided if the number of channels per pixel is greater than 3. Advantageously, this stage consists of calculating 3 principal components of the holographic image and injecting these three principal components into the extractor. Advantageously, the principal components chosen for the dimensionality reduction are hyperparameters during the training of the convolutional network 66 CNN.

[0126] The annotations of the slide images, sub-images and patches, as well as the training of the predictor, are carried out in a manner analogous to the manner described above. Figures 12A and 12B describe the performance of the holographic image prototype developed by the inventors, performance obtained on the initial basis of the 737 Gram slides.

[0127] E. EXTENSION OF THE TEACHING OF THE DETAILED EMBODIMENTS i. The application of the invention to the characterization of microorganisms in a sample prepared on a Gram slide has been described. The invention applies to any type of digital image comprising objects to be characterized. For example, remaining in the field of biology, this technique applies to cellular imaging of eukaryotic cells, in particular fluorescently labeled cells whose different types can be detected, as well as their different organelles, such as the cell nucleus. Another example is hematology, where the different formed elements of blood (erythrocytes, leukocytes) can be detected. A third example is the analysis of urine samples, where again the formed elements can be detected together with microorganisms.A fourth example is histopathology, to detect cancer cells that may only cover a very small part of the surface of a very high-resolution image. For some of these applications where the notion of quantification or counting is important (which is not the case for positive blood cultures), analyzing the number of patches and / or sub-images positive for a particular class can be a starting point for implementing a count of the different cell types.

[0128] In general, the invention finds application to the characterization of any object in a high-resolution image, composite or not. ii. An embodiment has been described in which the characteristics provided at the last stage are the prediction scores of the second stage. Alternatively, the characteristics are for example those generated by a layer of neurons of the MIL portion. According to the invention, descriptive statistics are also generated from these characteristics and communicated to the final predictive model. iii. A predictive model at the sub-image level has been described implementing an attention mechanism. Alternatively, this model does not implement this function, the characteristic vectors from the patches being directly grouped via a pooling method such as the application of an average or max pooling.

[0129] F. IT IMPLEMENTATION

[0130] The training phase of the predictor of the invention, in particular that described in relation to Figure 10A and the prediction phase, in particular that described in relation to Figure 5, apart from the steps of preparing the slide and acquiring the optical signal corresponding to the image thereof, are implemented by computer, namely by means of hardware circuits comprising computer memories (cache, RAM, ROM, etc.) and one or more microprocessors or processors (CPU and / or GPU), organized or not in the form of calculation nodes, necessary for the execution of computer instructions stored in the memories for the implementation of said phases. It will be understood that, with regard to calculation, any type of computer architecture may be suitable and that the description above and below should not be understood as limiting the scope of the invention.

[0131] Several architectures are possible, as illustrated in Figures 13-15.

[0132] In a first architectural variant (fig. 13), a first organization 2000, for example the Applicant, hosts or controls one or more computing servers 2002 associated with one or more databases 2002 for storing the annotated slide images and the learning phase is implemented by the organization 2000 on its server(s) 2002. A second organization 2006, for example a microbiological laboratory, hosts a microscope 2008 as described above, connected to, or incorporating, a personal computer, a desktop computer or a server 2010 and implements the preparation of the Gram slide until the acquisition of the digital image of the slide stored by the computer unit 2010.This image is then pushed, through a remote connection network, to the first organization 2000 for implementation on the server 2002 (or a computing unit different from that used for training, for example implemented on a Cloud in the form of Software As a Service) of the remainder of the prediction phase. The report of the classification of microorganisms present, or not, produced is then pushed, through the network 2012, to the second organization 2006 which takes or does not take therapeutic measures depending on the report.

[0133] A second architectural variant (fig. 14) differs from the first in that a copy of the software implementing the prediction phase is downloaded into the second organization which implements the entire prediction phase using the computer 2010 or a computing server (not shown). In a variant of this architecture, this download corresponds to a copy of the software in the computer 2010, a computer provided by the first organization 2000. In a third variant (fig. 15), the entire learning and prediction phases are implemented by a single organization 2006.

Claims

CLAIMS 1. Method for predicting a class of microorganisms contained in a sample, from among several classes of microorganisms, the method comprising: A. the preparation of a slide, in particular a microscope slide, comprising spreading the sample on said slide; B. the acquisition of at least one digital image of the slide with micrometric or sub-micrometric resolution; C. the application, implemented by computer, of a model for predicting the class of microorganisms as a function of the acquired image, characterized in that said image is subdivided into a plurality of sub-images and each sub-image is subdivided into patches, and in that the application of the prediction model comprises: D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class; E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream layers comprising a layer implementing a prediction of at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated; and F. for the acquired image: a. the calculation of a characteristic vector calculated for the sub-images; b. the application of a prediction model of at least one class for the microorganisms present in the sample as a function of the characteristic vector, said prediction model being trained on characteristic vectors calculated from training images.

2. Method according to claim 1, characterized in that the characteristic vector is a distribution of the scores calculated for the sub-images.

3. Method according to one of claims 1 or 2, characterized in that each training image is subdivided into sub-images and each of said sub-images is subdivided into patches, and in that less than 50% of the patches of the training images are annotated, said annotated patches forming the training patches of the first convolutional network.

4. Method according to claim 3, characterized in that less than 10% of the patches of the training images are annotated.

5. Method according to any one of the preceding claims, characterized in that the training of the first and second convolutional networks and of the prediction model is configured to obtain a macro prediction specificity greater than or equal to 90%.

6. Method according to any one of the preceding claims, characterized in that the prediction model of step F is a “random forest” model.

7. Method according to any one of the preceding claims, characterized in that the upstream pooling layer is associated with an attention layer configured to apply a weight to the output of each extractor.

8. Method according to any one of the preceding claims, characterized in that the second network is a MIL-CNN network trained by batches of instances, the batches of instances being made up of the training patches.

9. Method according to any one of the preceding claims, characterized in that the characteristic vector of step F. a comprises for each class: - the maximum score among the sub-images; and / or - the X ieme percentile of scores among the sub-images, with X greater than or equal to 90%, preferably equal to 95%; and / or - the Y eme percentile of scores among the sub-images, with Y less than or equal to 10%, preferably equal to 5%; and / or - the median score among the sub-images.

10. Method according to claim 8, characterized in that it further comprises the number of sub-images in the image.

11. Method according to any one of the preceding claims, characterized in that the first convolutional network is pre-trained on images not comprising microorganisms and then trained on annotated training patches.

12. Method according to claim 10, characterized in that the first pre-trained convolutional network is a VVG16 or ResNet network.

13. A method according to any one of the preceding claims, characterized in that the microorganisms comprise bacteria and the classes comprise at least Gram positive and Gram negative, and in that the preparation of the slide comprises the preparation of a Gram slide.

14. Method according to claim 13, characterized in that the classes of microorganisms further comprise classes of morphotypes.

15. Method according to any one of the preceding claims, characterized in that the classes of microorganisms comprise a “neither bacteria nor yeast” class, a “Gram-negative bacilli” class, a “Gram-positive bacilli” class, a “Gram-positive coryneum bacilli” class, a “Gram-negative cocci” class, a “Gram-positive cocci in chains” class, a “Gram-positive cocci in clusters” class and a “yeasts” class.

16. Method according to any one of the preceding claims, characterized in that the sample comprises blood.

17. Method according to any one of the preceding claims, characterized in that the sample is a positive blood culture.

18. Method for training a model for predicting a class of microorganisms among several classes of microorganisms from a digital image of a slide on which a sample likely to comprise microorganisms is spread, said training method comprising: A. the creation of a training database comprising digital slide images annotated by one or more classes of microorganisms, the slide images being divided into a plurality of sub-images annotated by one or more classes of microorganisms and the training sub-images being subdivided into patches, at least part of the patches being annotated by one or more classes of microorganisms, B. training a first convolutional neural network based on the annotated patches; C. training a second neural network from the annotated base of sub-images, said second network comprising at least one extractor of patch characteristics, a patch characteristics pooling stage, and a prediction stage of the class(es) of microorganisms present in the sub-image; D. the creation of a database of distributions of the classes present in the training slides based on the classes predicted by the second neural network applied to the annotated sub-images; E. of a model for predicting at least one class of microorganisms present in a slide based on spatial distributions of the classes, method according to which the predictor comprises the convolutional part of the first neural network, downstream of which is connected the second neural network, downstream of which is connected the prediction model.

19. Training method according to claim 18, characterized in that the training of the first network is carried out on the classes of microorganisms to which an “ambiguous” class is added, an annotated patch also being annotated by this class when objects in the patch in the event of uncertainty about the objects present in the patch.

20. Training method according to claim 18 or 19, characterized in that the training of the second network comprises the training of a MIL-CNN network, the trained MIL portion of the MIL-CNN network constituting the second trained neural network.

21. Method for predicting a class of microorganisms contained in a sample, from among several classes of microorganisms, the method comprising: C. the computer-implemented application of a model for predicting the class of microorganisms in a sample spread on a slide based on an image acquired from said slide, characterized in that said image is subdivided into sub-images and each sub-image is subdivided into patches, and in that the application of the prediction model comprises: D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class; E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream prediction layers of at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated by at least one class; and F. for the acquired image: a. calculating a feature vector based on the scores calculated for the sub-images; b. applying a prediction model of at least one class for the microorganisms present in the sample based on the feature vector, said prediction model being trained on feature vectors calculated from training images.

22. Prediction method according to claim 21, characterized in that steps D to F are in accordance with any one of claims 2 to 17.

23. System for predicting a class of microorganisms contained in a sample, among several classes of microorganisms, the system comprising a computer unit configured to implement a model for predicting the class of microorganisms in a sample spread on a slide as a function of an image acquired from said slide, characterized in that said image is subdivided into sub-images and each sub-image is subdivided into patches, and in that the application of the prediction model comprises: D. for each patch, applying a microorganism feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising microorganisms, said microorganisms being individually annotated by at least one class; E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream layers for predicting at least one class, in the form of a score, said second network being trained on training sub-images comprising microorganisms, each training sub-image being globally annotated by at least one class; and F. for the acquired image: a. the calculation of a feature vector calculated for the sub-images; b. the application of a prediction model of at least one class for the microorganisms present in the sample based on the feature vector features, said prediction model being trained on feature vectors calculated from training images.

24. Prediction system according to claim 23, characterized in that the computer unit is configured to implement steps D to F which are in accordance with any one of claims 2 to 17.

25. Computer program product comprising a computer memory storing computer-readable instructions for implementing steps D to F according to any one of claims 2 to 17.

26. Computer program product comprising a computer memory storing computer-readable instructions for implementing steps A to E according to any one of claims 18 to 20.

27. Method for predicting a class of objects contained in a digital image from among several classes of objects, the method according to which the digital image is subdivided into a plurality of sub-images and each sub-image is subdivided into patches, and according to which: D. for each patch, applying an object feature extractor, said extractor comprising a convolutional part of a first convolutional neural network, said first network being trained on a set of training patches comprising objects, said objects being individually annotated by at least one class; E. for each sub-image, the application of a second neural network, connected to receive the characteristics extracted from the patches constituting said sub-image, said second network comprising an upstream pooling layer and one or more downstream prediction layers of at least one class, in the form of a score, said second network being trained on training sub-images comprising objects, each training sub-image being globally annotated; and F. for the acquired image: a. the calculation of a characteristic vector calculated for the sub-images; b. the application of a prediction model of at least one class for the objects present in the sample as a function of the characteristic vector, said prediction model being trained on characteristic vectors calculated from training images.

Images