Method for determining a region of interest in an image of a biological sample

Abstract

The invention relates to a method for determining a region of interest in an image of a biological sample using a machine learning model trained on training data comprising: first images of biological samples prepared using a first staining technique and information relating to regions of interest in these first images; and N pairs of images, each pair of images comprising an image of a respective biological sample prepared using an nth staining technique, which image is referred to as the n second image, and an image of the same biological sample prepared using the first staining technique, which image is referred to as the n third image, N being an integer greater than or equal to 1 and n being an integer between 2 and N+1; and wherein the calculation and optimisation of a cost function are carried out on the basis of the training images which are segmented using current parameters of the machine learning model.

Description

[0001] METHOD FOR DETERMINING A REGION OF INTEREST IN AN IMAGE OF A BIOLOGICAL SAMPLE

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to the field of digital pathology, in particular the detection of regions of interest in images of biological samples obtained by staining.

[0004] STATE OF THE ART

[0005] Over the last decade, the rise of digital pathology has led to a sharp increase in computer-aided diagnosis (CAD) systems on whole slide scans (WSIs), also known as virtual microscopy images. In particular, the detection of regions of interest (ROIs) in such images is a valuable aid for pathologists as well as for analysis algorithms that need to focus on a particular ROI (e.g., to perform cell counting on a tumor area only, or for tumor detection on a tissue area), allowing for faster final diagnosis. For example, tissue area detection avoids analyzing the background and therefore accelerates algorithms for searching for anomalies or cell counting, which are performed only in the tissue area.In order to extract these regions of interest, it is necessary to have efficient solutions to perform segmentation of an image of a biological tissue sample and, for example, to separate the foreground of biological tissues and the background.

[0006] Traditionally, pathologists use one or more staining techniques on biological tissues. Staining is a well-known technique for highlighting and identifying certain cells or cellular structures in biological tissues by staining them with more or less intense stains and / or different colors. For example, the use of hematoxylin-eosin (HE, or H&E) staining, commonly used for the detection of cancer cells, stains the nuclei blue / purple, the cytoplasm pink, and other basic cellular elements pink / red. Immunohistochemistry (IHC) staining is commonly used to detect certain cancers using specific markers, for example, the Ki-67 antigen or the combination of cytokeratin 8 and 18 antibodies (CK 8 / 18), and generally stains the cell elements in blue / brown tones.

[0007] There are methods for segmenting biological tissue images that rely on adaptive thresholding, but these methods are very sensitive to coloration variations and are not very effective on a wide variety of images with different colorations. More recently, methods based on machine learning models have emerged. For example, some of these methods rely on the use of a machine learning model trained on annotated images, i.e., images of biological samples for which the regions of interest are known (e.g., a binary segmentation mask verified by a pathologist and representing the contours of the tissues in the sample image).However, in these methods, the training images are usually images of biological samples stained with a single dye, and the results are very degraded when the model is applied to detect regions of interest in images of biological samples stained with a different dye than the one used for the training data. An alternative solution is to train the learning model with annotated images stained with a wide range of dyes. However, such a solution is not optimal because it requires a large number of annotated images, which are both time-consuming and expensive to obtain.

[0008] There is thus a need for a method for detecting areas of interest in images of biological tissues that is effective regardless of the coloration used, and that does not require too much training data to train the detection model.

[0009] The present invention improves the situation. SUMMARY

[0010] The invention relates to a method for detecting regions of interest in a digital image of a biological sample stained with one or more stains, using a segmentation algorithm based on a machine learning model trained in a semi-supervised manner using training data, part of which is annotated, and the other part of which is not.

[0011] Annotated data typically includes images of biological samples stained with a first stain, e.g., HE stain, for which the tissue areas have been manually determined by an expert and are known.

[0012] The unannotated data includes pairs of matched images, each pair comprising an image of a sample stained with a second stain, and an image "similar" to an image of the same sample stained with the first stain. In other words, the paired images correspond to two images of the same sample, one stained with the second stain, and the other stained with the first stain. For these images, the tissue areas are not known in advance.

[0013] The annotated data allows the model to be trained to efficiently determine tissue areas in images stained with the same stain (i.e., the first stain). The unannotated data allows the model to be trained on other stains. The resulting model can efficiently detect areas of interest in images of biological samples, regardless of their staining.

[0014] DEFINITIONS

[0015] In the present invention, the terms below are defined as follows:

[0016] By "region of interest" (or "area of ​​interest") of an image of a biological sample, we mean an area of ​​the image that we seek to detect in relation to the rest of the image. For example, the region of interest may be a tissue area, a tumor area, a stroma area, an epithelial area, a lesion, an artifact, etc.; By "determining a region of interest in an image", we mean a determination of a set of pixels of the image as belonging to a region of interest. This determination is made using a segmentation algorithm based on a machine learning model. A segmentation algorithm predicts, for an input data, a class (or label). In the context of the present invention, the segmentation algorithm predicts, for each pixel of a received image, a class from two possible categories: "region of interest" and "outside region of interest".In the description, the terminologies “classify”, “segment” and “detect / determine a region of interest” are used without distinction;

[0017] By "training data" is meant data enabling the learning model to be trained during the so-called learning or training phase;

[0018] By "staining technique" is meant a technique for preparing biological samples to stain tissues and increase contrasts between the tissues and the background. Certain areas, such as tissue areas of the biological sample, are then stained with certain colors which depend on the technique used. Such techniques are well known to those skilled in the art;

[0019] By "binary segmentation image" (or "binary segmentation mask") is meant an image associated with a starting image, conventionally having the same size as the starting image and being obtained using a segmentation method applied to the starting image. The binary segmentation image reveals the regions of interest of the starting image. Typically, the pixels corresponding to the regions of interest may be of a first color (for example white), and the pixels not corresponding to regions of interest of a second color (for example black). It is noted that in the description below, the term "binary segmentation image" does not necessarily designate an image displayable on a display device, but any possible result of a segmentation by the algorithm, for example a table of values ​​or classes for the pixels of the image. DETAILED DESCRIPTION

[0020] A first aspect of the present disclosure relates to a computer-implemented method for semi-supervised training of a machine learning model to determine a region of interest in an image of a biological sample, the method comprising: receiving training data comprising:

[0021] ■ first images of biological samples prepared with a first staining technique, and information relating to regions of interest in said first images;

[0022] ■ N pairs of images, each pair of images comprising an image of a respective biological sample prepared with an nth staining technique, called the nth second image and an image similar to an image of the same biological sample prepared with the first staining technique, called the nth third image, N being an integer greater than or equal to 1 and n being an integer between 2 and N+1; and training the machine learning model with the training data;and wherein, said information relating to regions of interest in said first images also comprises segmentation binary images respectively associated with the first biological sample images, each segmentation binary image comprising a first set of pixels corresponding to regions of interest of the corresponding first biological sample image, and a second set of pixels corresponding to areas of the corresponding first biological sample image other than the regions of interest, the segmentation binary images respectively associated with the first images being called verified segmentation binary images, the method also comprising: a) from each first biological sample image, generating a respective first segmentation binary image using current parameters of the machine learning model;b) from each n second biological sample image, generating a respective n second segmentation binary image using current parameters of the machine learning model; c) from each n third similar image, generating a respective n third segmentation binary image using current parameters of the machine learning model; d) calculating a cost function from:;

[0023] ■ a representative value of differences between the first segmentation binary images and the corresponding verified segmentation binary images; and

[0024] ■ of N representative values ​​of differences between the n second binary segmentation images and the n third corresponding binary segmentation images;

[0025] According to further advantageous aspects of the present disclosure, said method for semi-supervised training of a machine learning model comprises one or more of the following features, taken alone or in any technically possible combination: the machine learning model is a deep learning model; each n third image is generated from the respective n second image by means of a colorimetric transformation; the colorimetric transformation uses a residual circular generative adversarial network; the representative value of the differences between the first segmentation binary images and the corresponding verified segmentation binary images is a function of a mean square error between the first segmentation binary images and the corresponding verified segmentation binary images;and the N values ​​representative of the differences between the n second binary segmentation images and the corresponding n third binary segmentation images are functions of a mean square error between the n second binary segmentation images and the corresponding n third binary segmentation images; the first staining technique is a hematoxylin-eosin staining and the other N staining techniques are based on one or more markers of an immunohistochemistry, such as the Ki-67 antigen, antibodies or a combination of cytokeratin 8 and 18 antibodies (CK 8 / 18);

[0026] N is equal to 1; and / or the first staining technique is a hematoxylin-eosin stain and the second staining technique is based on immunohistochemistry, or conversely the first staining technique is based on immunohistochemistry and the second staining technique is a hematoxylin-eosin stain.

[0027] In a particular aspect, the present disclosure relates to a computer-implemented method for semi-supervised training of a machine learning model to determine a region of interest in an image of a biological sample. The method may comprise: receiving training data comprising:

[0028] ■ first images of biological samples prepared with a first staining technique, and information relating to regions of interest in said first images;

[0029] ■ image pairs, each image pair comprising an image of a respective biological sample prepared with a second staining technique and an image similar to an image of the same biological sample prepared with the first staining technique; training the machine learning model with the training data.

[0030] In one or more embodiments, no information relating to regions of interest in the images of the image pairs is received. A biological sample image may typically be a digital image of a slide (also referred to as a virtual slide or a digitized slide). The region of interest to be detected may be any region of the image that is desired to be detected or isolated, for example for use in a diagnostic aid context, such as a tissue area, an epithelial area, a stromal area, an artifact, a tumor area, a cell cluster having a particular structure, etc.

[0031] The information about the regions of interest in the first images can typically correspond to classification information about the pixels in the image, for example according to one or the other of the classes: "region of interest" and "outside a region of interest". For example, the class of each pixel in each first image can be known (typically, it has been determined or validated by an expert) and represents the "ground truth". Thus, the first images correspond to annotated training data. Conversely, for pairs of images, it is not necessary to have information about the regions of interest in these images, such as the classes of the pixels in these images. The pairs of images correspond to unannotated training data. Thus, the learning is done in a semi-supervised manner.

[0032] By "image similar to an image of the same biological sample prepared with the first staining technique" is meant an image whose characteristics (contours, background, relative pixel intensities, etc.) are similar to those of an image of the same biological sample that has been or would have been prepared with the first staining technique. As detailed below, such an image can be obtained either fully automatically, for example by generating or digitally simulating it from the image of the sample, or semi-manually, by applying techniques for re-staining the slide containing the biological sample and scanning the new slide obtained.

[0033] Thus, each image pair includes two images similar to images of the same biological sample prepared by two different staining techniques. In other words, the images in a pair "look alike" but correspond to different colorations (e.g., one image in pink tones, and one image in blue / brown tones).As detailed below, there are several techniques for obtaining these image pairs, including biological recoloring techniques (for example, washing the biological tissue and then recoloring it, or making successive longitudinal sections of the sample and coloring each slice a different color), or digital recoloring techniques - also called colorimetric transformations (each pair includes an image of a colored sample, and from this image, a "matched" image is simulated so as to represent the same sample, but colored with a different coloration).Advantageously, the model training phase is therefore carried out in a semi-supervised manner, and makes it possible to obtain an effective model for detecting regions of interest (for example tissue areas) in virtual slides stained with the first or second staining technique, or by a mixture of the two techniques, or even by a third staining technique which would provide colors similar to those used in the training data. Indeed, even if annotations are only available for a single staining, the use of data not annotated with another staining makes it possible to generalize the detection to the second staining, for which no annotation is however available. In other words, the invention makes it possible to use only annotations for a single type of staining (which considerably reduces the cost and time), and to transpose the learning and detection to other colorimetric domains.

[0034] In one or more embodiments, the machine learning model may be a deep learning model. For example, the machine learning model may be a convolutional neural network.

[0035] Indeed, such models are known to provide good results in image segmentation algorithms.

[0036] In one or more embodiments, each pair of images may include an image of a respective biological sample prepared with the second staining technique and an image simulating an image of the same biological sample prepared with the first staining technique. In one or more embodiments, for each pair of images, the image of the biological sample prepared with the second staining technique may be referred to as the second image and the image similar to an image of the same biological sample prepared with the first staining technique may be referred to as the third image. For each pair of images, each third image may be generated from the respective second image through a colorimetric transformation.

[0037] By "respective second image" is meant the second image from which the third image is generated. Of course, the terms "second image" and "third image" are introduced here for the sake of simplification and do not constitute any technical limitation. By "colorimetric transformation" is meant a digital technique for recoloring an image into one or more other color domains.

[0038] The use of a colorimetric transformation to recolor images advantageously makes it possible to automatically obtain pairs of images corresponding to the same biological sample, colored in two different color domains (corresponding to two different coloring techniques).

[0039] In particular, the colorimetric transformation may use a generative adversarial network (GAN), such as a circular generative adversarial network (or "Cycle GAN"), for example a residual circular generative adversarial network (or "Residual Cycle GAN").

[0040] Other colorimetric transformations can be used, for example transformations using color deconvolutions.

[0041] In one or more embodiments, the information relating to the regions of interest in the first images may comprise segmentation binary images respectively associated with the first biological sample images, each segmentation binary image comprising a first set of pixels corresponding to regions of interest of the corresponding first biological sample image, and a second set of pixels corresponding to areas of the corresponding first biological sample image other than the regions of interest. In particular, the segmentation binary images respectively associated with the first images may be referred to as verified segmentation binary images.The method may comprise: a) from each first biological sample image, generating a respective first segmentation binary image using current parameters of the machine learning model; b) from each second biological sample image, generating a respective second segmentation binary image using current parameters of the machine learning model; c) from each third image, generating a respective third segmentation binary image using current parameters of the machine learning model; d) calculating a cost function from:.

[0042] ■ a representative value of differences between the first segmentation binary images and the corresponding verified segmentation binary images; and

[0043] ■ of a value representative of differences between the second binary segmentation images and the corresponding third binary segmentation images.

[0044] Each of the segmentation binary images may respectively comprise a first set of pixels corresponding to regions of interest of the corresponding biological sample image, and a second set of pixels corresponding to areas of the corresponding biological sample image other than the regions of interest. Steps a) to d) may be repeated so as to minimize the cost function.

[0045] By current parameters of the learning model, we mean the values ​​of the parameters during the current iteration. At the end of each iteration, the parameter values ​​are updated to minimize the cost function, and during the next iteration, the model is applied to the images, with the updated parameter values. Thus, advantageously the cost function used for the learning model depends on the one hand on the difference (or "distance") between the segmentation result on the annotated data and the annotations (the "truth"), and on the other hand on the difference between the segmentation results on the unannotated image pairs.

[0046] In particular, the representative value of the differences between the first segmentation binary images and the corresponding verified segmentation binary images may be a function of a mean square error between the first segmentation binary images and the corresponding verified segmentation binary images; and the representative value of the differences between the second segmentation binary images and the corresponding third segmentation binary images is a function of a mean square error between the second segmentation binary images and the corresponding third segmentation binary images.

[0047] Of course, other measures than the mean square error are possible, for example cross entropy, Cohen's kappa, the absolute value of the difference, the Jaccard index, the Dice coefficient, etc.

[0048] In one or more embodiments, one of the staining techniques may be a hematoxylin-eosin staining and the other staining technique may be based on immunohistochemistry. Of course, other staining techniques may be used, such as Sirius red, PAS (Periodic Acid Schiff) staining, etc. Furthermore, the two staining techniques may correspond to two different immunohistochemistry techniques, for example by Ki-67 and by CK8-18.

[0049] Another aspect of the present disclosure relates to a computer-implemented method for determining a region of interest in an image of a biological sample prepared with a staining technique. The method may comprise: receiving the image of the biological sample; and determining a region of interest of the image using a segmentation algorithm using a machine learning model trained according to one of the previously described embodiments. Here, the image of a biological sample prepared with a staining technique is no longer training data, but an image in which it is desired to detect a region of interest.The "coloring technique" used for the image to be segmented can be the first coloring technique, the second coloring technique, a mixture of the two techniques, or a coloring technique that colors the cells in colors similar to those of the first or second coloring technique.

[0050] In one or more embodiments, the region of interest may be a tissue area, a tumor area, a stromal area, an epithelial area, an area corresponding to a lesion, or an area corresponding to an artifact.

[0051] Another aspect of the present disclosure relates to a computing device comprising a circuit configured to implement a method of training the machine learning model and / or a method of determining a region of interest in an image of a biological sample prepared with a staining technique.

[0052] Another aspect of the present disclosure relates to a computer program product comprising instructions for implementing steps of the method of training the machine learning model and / or a method of determining a region of interest in an image of a biological sample prepared with a staining technique, when this program is executed by a processor.

[0053] DESCRIPTION OF FIGURES

[0054] Figure 1 is an example flowchart representing steps of a method of detecting regions of interest in an image of a biological sample according to one or more embodiments;

[0055] Figure 2 represents an example flowchart representing training of the learning model according to one or more embodiments; Figure 3 represents an example of determining the part of the cost function corresponding to the first set of training images according to one or more embodiments;

[0056] Figure 4 represents an example of determining the part of the cost function corresponding to the second set of training images according to one or more embodiments;

[0057] Figure 5 represents an example flowchart representing the generalization of the training of the learning model of Figure 2 according to one or more embodiments;

[0058] Figure 6 illustrates a device for implementing methods of training the machine learning model and / or detecting regions of interest according to one or more embodiments.

[0059] ILLUSTRATIVE EMBODIMENTS OF THE INVENTION

[0060] Figure 1 is an example flowchart depicting steps of a method for detecting regions of interest in an image of a biological sample according to one or more embodiments.

[0061] Training data (or “learning data”) of the machine learning model are received at a step 110. This training data comprises, on the one hand, annotated images of biological samples stained with a first staining technique (received at a step 112). An “annotated” (or “labeled” or “tagged”) image is, for example, an image in which each pixel of the image is associated with a label. For example, the label may be either “region of interest” or “outside region of interest”. These labels may be determined or validated by experts, such as pathologists or biologists, and represent the “truth”.

[0062] The training data also includes unannotated images of biological samples stained with a second staining technique, and simulated images of these same samples, if they had been stained with the first staining technique (received at a step 114). For these images, the pixel labels are not known.

[0063] Thus, the training data includes on the one hand images of biological samples in which the regions of interest are colored according to certain hues and images of biological samples in which the regions of interest are colored according to other hues (for example images annotated in red-pink-purple tones, and unannotated images in blue to brown tones).

[0064] For the sake of simplification, it is assumed in the following that the annotated images (received in step 112) are images of biological samples stained by HE and are hereinafter referred to as “HE images” or “annotated HE images”, and the unannotated images (received in step 114) are images of biological samples stained by IHC and are hereinafter referred to as “IHC images” or “unannotated IHC images”. The simulated images (also received in step 114) are referred to as “simulated HE images”. These images are obtained from the IHC images by a digital processing technique and simulate images of the same biological samples, if the latter had been stained by HE and not by IHC. Of course, the staining techniques for the annotated data and the unannotated data are interchangeable, and other staining techniques may be used.In particular, the two staining techniques may even correspond to two immunohistochemistry techniques with different markers. Furthermore, steps 112 and 114 may be implemented in this order or another, or in parallel.

[0065] There are several techniques for simulating HE images from corresponding IHC images. For example, it is possible to wash and re-stain slides with HE, and to recover digital images of these samples. However, this method is technically difficult and very time-consuming to implement. It is also possible to make successive longitudinal sections of a biological sample in order to obtain two slides, which can then be stained with two different staining techniques. In addition to requiring additional technical gestures, a disadvantage of this method is that, even if they are quite similar, the biological tissues are not exactly the same in the two slides, and it is often necessary to apply registration between the two images obtained, which complicates the procedure. In one or more embodiments, it is possible to use color deconvolution techniques.In particularly advantageous embodiments, the simulated HE images are obtained using machine learning models, such as generative adversarial networks (GANs), which have shown good results in image generation. For example, it is possible to use a residual circular generative adversarial network (residual Cycle-GAN) as described in the article “Residual cyclegan for robust domain transformation of histopathological tissue slides” by Thomas de Bel et al., Medical Image Analysis, Volume 70, May 2020, 102004. Such a method makes it possible to obtain domain transformation functions, for transforming an image in the IHC-stained domain into an image in the HE-stained domain.The residual circular generative adversarial network has the same architecture as a classic cycle-GAN comprising two sets of generators and discriminators, but in the residual circular generative adversarial network, the input image has a direct connection with the generated output image. In this way, the generator does not need to reconstruct the image from a set of filter outputs, but only needs to add a residual, i.e., a color change in the input image to make it look like an image generated in the target domain. Since it reduces the computational load on the generator, this approach requires less data and converges faster than a classic circular generative adversarial network.

[0066] A machine learning model is trained (step 120) from the received training data. It is recalled that training a learning model consists of determining a set of model parameters and / or teaching the model a prediction function from the training data, so that the model can then predict the label of a new received data (in the present application, predict the labels of the pixels of a new received sample image). Since only a portion of the training data is annotated (or labeled, or tagged), the training is therefore carried out in a semi-supervised manner. Embodiments of training the learning model are detailed with reference to Figure 2. In one or more embodiments, the learning model may be a deep learning model, for example a convolutional neural network.

[0067] In step 130, an image of a biological sample is received to be segmented (i.e., to determine the regions of interest). In step 140, a segmentation of the received image is performed using the learning model trained in step 120. During this segmentation, the model predicts the labels of the pixels of the received image from the parameters and / or the prediction function determined during the training. At the end of this segmentation, the pixels are therefore labeled, either as “region of interest” or as “outside the region of interest” for example. Thus, the pixels corresponding to regions of interest are identified as such. The use of a machine learning model to perform image segmentation is known to those skilled in the art and is not further detailed here.For example, the U-Net architecture, also called a “fully convolutional network,” is well suited to biomedical image segmentation.

[0068] According to certain embodiments, the method described with reference to Figure 1 may also comprise the preparation of the samples and the recovery of the associated images. The preparation of the samples conventionally comprises a succession of steps known to the person skilled in the art, in particular: fixation (to immobilize and preserve the sample over time in a state close to that of life, it is generally carried out by a fixative liquid or by freezing), inclusion (to stiffen the sample with an inclusion medium), sectioning (generally by microtomy or cryotomy), staining or immunolabeling (to accentuate the contrasts between the constituent elements of the biological material and to highlight specific cellular and tissue constituents), and finally mounting (to obtain slides ready to be observed).The slides can then be viewed under a microscope, and an overview of the sample can be digitally reconstructed. The method illustrated in Figure 1 may also include annotating the images of biological samples stained with the first staining technique (which are then received at step 112).

[0069] Figure 2 shows an example flowchart representing training of the learning model according to one or more embodiments. As mentioned above, in step 110, automatic model training data is received, and includes for example annotated HE image data (received in step 112), as well as unannotated IHC images with their corresponding simulated HE images (received in step 114).

[0070] As is known, during the training (or "learning") phase of the learning model, the model parameters are estimated iteratively in order to minimize an objective function linked to a cost function L, which represents the prediction error rate that the model achieves on the training data.

[0071] In one or more embodiments, at each iteration, the current parameters of the model are applied to training images to determine regions of interest in these images. Thus, in step 121, the current model is used on all or part of the HE images to determine areas of interest in these HE images. During this step, the annotations are not used. In step 123, the regions of interest determined in step 121 are compared to the “truth”, i.e. to the regions of interest of the annotations, and a first part of the cost function is determined from this comparison.

[0072] In particular, Figure 3 represents an example of determining this first part of the cost function according to one or more embodiments. Figure 3 represents an image 310, for example an HE image, and a corresponding image 320, which corresponds to the annotation of the image 310, i.e. to the regions of interest validated by an expert. The image 310 typically comprises different areas, for example: areas 311 corresponding to the background (generally little or not tinted), and tissue areas 312, 313. In Figure 3, the areas 311 corresponding to the background are represented in white. The tissue areas (i.e., in this example, the regions of interest) are represented by hatched portions 312, 313. The area 313 comprises hatchings closer together than the area 312, to represent more or less intense colorations of the regions of interest in the image 310.For example, in the image 310 of the biological sample stained by HE, the areas 312 represented with more spaced hatching may correspond to the pink areas of the sample, and the areas 313 with the closer hatching may correspond to purple or dark red areas. The verified image 320 may be a binary image (also called a “verified segmentation binary image”) showing the validated regions of interest 322. Thus, in the image 320, the pixels corresponding to the background 321 may have a first value (for example, the value 0, corresponding to the color black) and the pixels of the regions of interest 322 may have a second value distinct from the first value (for example, the value 255 corresponding to the color white).

[0073] By applying a segmentation algorithm using the current parameters of the model to the HE image 310, a binary segmentation image 330 can be determined. This binary segmentation image 330 can typically show the regions of interest 332 determined as such by the model. In the image 330, the pixels corresponding to the background 331 can thus have a first value (for example, the value 0, corresponding to the color black) and the pixels of the regions of interest 332 can have a second value distinct from the first value (for example, the value 255 corresponding to the color white).

[0074] In the example of Figure 3, the training data 310, 320 and the output data 330 are images of the same size. Of course, it is not necessary to provide a representation of the pixel classification in the form of an image: instead of the binary images 320, 330, it is possible for example to store the values ​​or labels of the pixels in a matrix.

[0075] From the verified segmentation binary image 320 and the determined segmentation binary image 330, it is possible to determine the difference between the corresponding pixels of these two images and to define a first part Li of the cost function associated with the model from this difference.

[0076] For example, this first part Li of the cost function can be a function of the mean square error between the verified segmentation binary image 320 and the determined segmentation binary image 330: Yes HEis image HE 310, J : I HE ') is the result of the segmentation algorithm applied to image I HE 310 (i.e. the determined segmentation binary image), M HE is the verified segmentation binary image, and || . ||2 represents the Euclidean norm (or 2 norm). It is recalled that the mean square error between two images A and B having respective sets of pixels j and j is proportional to:

[0077] H is noted that the term || T ( HE ) — M HE||2 typically corresponds to the cost function used to train the model in existing approaches based on machine learning from annotated data only. As detailed below, in the present invention, a second term is added, to teach the model variability in the colorations of biological samples and enable it to segment images of biological samples stained with other segmentation techniques.

[0078] Referring again to Figure 2, in step 122, the current model is used on all or part of the IHC images and on the corresponding simulated HE images, to determine regions of interest in these images. In step 124, the regions of interest determined in step 122 in the IHC images are compared to the regions of interest determined in step 122 in the corresponding simulated HE images, and a second part of the cost function is determined from this comparison.

[0079] In particular, Figure 4 represents an example of determining this second part of the cost function according to one or more embodiments. Figure 4 represents an image 410, for example an IHC image, and a corresponding simulated HE image 420. The IHC image 410 comprises different areas, for example: areas 411 corresponding to the background (generally little or not tinted), and tissue areas 412, 413, which are in this example the regions of interest. In Figure 4, the areas corresponding to the background are the white areas 411 of the image 410. The areas of interest are represented by dot parts 412, 413. The density of the dots in the area 413 is greater than in the area 412, to represent more or less intense colorations of the regions of interest in the image 410.For example, in image 410 of the IHC-stained biological sample, areas 412 depicted with lower dot density may correspond to light brown areas of the sample, and areas 413 with higher dot density may correspond to dark brown areas.

[0080] Similarly, the corresponding simulated HE image 420 comprises different areas, for example: areas 421 corresponding to the background (shown in white in the figure) and areas of interest 422, 423, for example biological tissues. These areas of interest are represented by hatched portions 422, 423. Area 423 comprises hatchings that are closer together than area 422, to represent more or less intense colorations of the regions of interest in the image 420. For example, in the simulated HE image 420, areas 422 shown with more spaced hatchings may correspond to the light pink areas of the sample, and areas 423 with the closer hatchings may correspond to purple and / or dark pink areas.

[0081] By applying a segmentation algorithm using the current parameters of the model to the IHC image 410, a binary segmentation image 430 can be determined. This binary segmentation image 430 can typically show the regions of interest 432 determined as such by the model. On the image 430, the pixels corresponding to the background 431 can thus have a first value (for example, the value 0, corresponding to the color black) and the pixels of the regions of interest 432 can have a second value distinct from the first value (for example, the value 255 corresponding to the color white).

[0082] Similarly, by applying the segmentation algorithm using the current parameters of the model to the simulated HE image 420, another segmentation binary image 440 can be determined. This segmentation binary image 440 can typically show the regions of interest 442 determined as such by the model. In the image 440, the pixels corresponding to the background 441 can thus have a first value (for example, the value 0, corresponding to the color black) and the pixels of the regions of interest 442 can have a second value distinct from the first value (for example, the value 255 corresponding to the color white). In the example of Figure 4, the training data 410, 420 and the output data 330, 440 are images of the same size.Of course, it is not necessary to provide a representation of the pixel classification in the form of images: instead of the binary images 430, 440, it is possible, for example, to store the values ​​or labels of the pixels in a matrix.

[0083] From the two binary segmentation images 430 and 440 obtained, it is possible to determine the difference between the corresponding pixels of these two images and to define a second part L2 of the cost function associated with the model from this difference.

[0084] For example, this first part L2 of the cost function may be a function of the mean square error between the binary segmentation image 430 obtained from an IHC image and the binary segmentation image 440 obtained from the corresponding simulated HE image: Yes IHC is the IHC 410 image and its associated simulated image 420.

[0085] Referring again to Figure 2, the cost function associated with the learning model can then be determined (step 125) from the two parts determined in steps 123 and 124. For example, the cost function may be of the form: where A is a real regularization parameter between 0 and 1. The value of A can advantageously be variable during the training phase, for example to gradually increase between 0 and 1. Thus, the weight of the second part of the cost function in the “total” cost function increases during training. This allows the model to first learn to predict regions of interest in a domain in which the truth is known (i.e. on annotated HE images) before being extended to another domain in which the truth is unknown (i.e. unannotated IHC images). Of course, other cost functions are possible (in particular a weighted sum of mean square errors with other weightings than 1 and A), as long as they are functions of the differences between the segmentation results in the two groups of training data.Additionally, other measures of image difference can be used instead of the mean square error, such as Cohen's kappa (K), cross-entropy, absolute value of the difference, Jaccard index, Dice coefficient, etc.

[0086] Thus, the cost function advantageously takes into account on the one hand the segmentation error on annotated data in a single domain, and on the other hand the segmentation error on unannotated data from different domains. This allows for more robust segmentation on images that do not necessarily come from the same domain (i.e., which are not from samples prepared with the same staining technique), without requiring too much annotated data (especially in several domains, which is costly and time-consuming to obtain).Schematically, training the machine learning model using annotated HE images allows to efficiently segment new HE images, and training the learning model on corresponding IHC / HE image pairs allows to extend the segmentation to other types of staining (in particular to segment new IHC images, even if the training data does not include any annotated IHC images). Thus, the obtained model allows to efficiently segment images of biological samples, regardless of the staining techniques used to obtain them. In particular, the model is effective in determining regions of interest on images stained by IHC, but also by staining techniques other than HE or IHC. Furthermore, it is known that for the same staining technique, the intensity and contrast of the digital image depends on the type of scanner used.Thus, for two different scanners, images of the same slide stained with the same staining technique may have slightly different colors. The model thus trained is also effective in determining regions of interest in images from different scanners with staining differences. The model parameters can then be updated to minimize this cost function (step 126). If a convergence criterion is reached (test 127, arrow “O”), the training phase stops (step 128). Otherwise (test 127, arrow “N”), all or part of steps 112, 114, 121 to 127 are repeated with the parameters updated in step 126.For example, the convergence criterion can be based on the difference in parameter values ​​between the model parameter estimates between two (or more) successive iterations: as long as this difference is greater than a predefined threshold, the convergence criterion is not met.

[0087] Advantageously, the method of detecting regions of interest in an image of a biological sample and training the learning model described above can be generalized as illustrated in Figure 5.

[0088] In these embodiments, the training data comprises on the one hand annotated images of biological samples stained with a first staining technique Ci (received in step 112 as described previously). This first staining technique Ci can be considered as the reference staining technique. Advantageously, this first staining technique can be a hematoxyline-eosin staining (i.e. Ci = HE) as in the previous embodiments.

[0089] The training data also includes N pairs of unannotated images of biological samples stained with N different stains C2 to CN+I (received at steps 1142 to 114N+I similar to step 114 described previously), where N is an integer greater than or equal to 1. Each pair of unannotated images includes an image of a respective biological sample prepared with an nth staining technique, referred to as the n second image, and an image similar to an image of the same biological sample prepared with the first staining technique, referred to as the n third image, where n is an integer between 2 and N+1.

[0090] Advantageously, these other N C2 to CN+I staining techniques are based on one or more immunohistochemistry markers (i.e. C n= IHC), such as Ki-67 antigen, antibodies or a combination of cytokeratin 8 and 18 antibodies (CK 8 / 18). In the same way as before, the received images can be segmented to determine regions of interest, using the current parameters of the machine learning model.

[0091] Thus, the information relating to regions of interest in the first images also includes segmentation binary images respectively associated with the first biological sample images. The segmentation binary images respectively associated with the first images are called verified segmentation binary images. Respectively, each n second image corresponds to an n second segmentation binary image, and each n third image corresponds to an n third segmentation binary image.

[0092] Additionally, each of these segmentation binary images includes a first set of pixels corresponding to regions of interest of the corresponding biological sample image, and a second set of pixels corresponding to areas of the corresponding biological sample image other than the regions of interest.

[0093] Regarding the different cost functions, these can be written as follows:

[0094] Or :

[0095] Li corresponds to the cost function calculated between the first verified segmentation binary image and the first binary image obtained from the first prepared image with the first coloring Ci, and in the case where L1 is taken as the squared error between the two images;

[0096] I e1 is the first image prepared with the first coloring Ci, y( / ci ) is the result of the segmentation algorithm applied to image Ie1 (i.e. the determined segmentation binary image), M C1 is the verified segmented binary image, and || . ||2 represents the Euclidean norm (or 2-norm); and Ln corresponds to the cost function calculated between the n second segmentation image 7 e " and the third GI segmentation image Cn '), in the special case where Ln is taken as the squared error between these two images.

[0097] We can then define the global cost function associated with the learning model from steps 123 and 1242 to 124 n , as follows: Or n is a real regularization parameter between 0 and 1. The value of ncan advantageously be variable during the learning phase, for example to gradually increase between 0 and 1. Thus, the weight of the second part of the cost function in the “total” cost function increases during learning. This allows the model to first learn to predict regions of interest in a domain in which the truth is known (i.e. on the annotated Ci images) before being extended to another domain in which the truth is unknown (i.e. the C images n unannotated).

[0098] Other cost functions are also possible as described previously.

[0099] Figure 6 illustrates a device for implementing methods of training the machine learning model and / or detecting regions of interest according to one or more embodiments.

[0100] The device 500 may comprise a memory 501 for storing instructions for implementing steps of the methods for training the machine learning model and detecting regions of interest in a biological sample image using this model, the received data, in particular the training data and / or the images of biological samples to be segmented, and temporary data for carrying out all or part of the steps of the methods described above. The device 500 further comprises a control circuit 502, an input interface 503 for receiving data, in particular the training data and / or the images of biological samples to be segmented, and an output interface 504 for providing output data, such as parameters of the learning model or regions of interest of a biological sample image.

[0101] In one or more embodiments, the device 500 may be in the form of a computer having a display 505 and a keyboard 506 to enable easy interaction with a user. The device 500 may be a mobile terminal, a computer, a computer network, an electronic component, or other apparatus comprising a processor operably coupled to a memory, as well as, depending on the embodiment selected, a data storage unit, and other associated hardware such as a network interface and a media reader for reading and writing to a removable storage medium. The removable storage medium may be, for example, a compact disc (CD), a digital video / versatile disc (DVD), a flash drive, a USB flash drive, etc.Depending on the embodiment, the memory, data storage unit, or removable storage medium contains instructions that, when executed by the control circuit 502, cause the control circuit 502 to control the input interface 503, the output interface 504, and the memory 501.

[0102] The control circuit 504 may be a component implementing a processor or a computing unit for training a learning model and / or detecting regions of interest in a biological sample image according to the proposed method and for controlling the units 501, 503 and 504 of the device 500.Furthermore, the device 500 may be implemented in software form (“software” or “firmware”), in which case it takes the form of a program executable by a processor, corresponding for example to an application downloadable and executable on a device such as a smartphone or tablet, as described above, or in hardware form (or “hardware”), such as an application-specific integrated circuit (ASIC), a system on chip (in English “System On Chip”, or SOC), or in the form of a combination of hardware and software elements, for example a software program intended to be loaded and executed on a component of the FPGA type (for “Field Programmable Gate Array” in English). SOCs are embedded systems which integrate all the components of an electronic system into a single chip.An ASIC (Application Specific Integrated Circuit) is a specialized electronic circuit that combines custom features for a given application. ASICs are typically configured during manufacturing and can only be simulated by the user. FPGA-type programmable logic circuits have electronic circuits that can be reconfigured by the user. The device 500 may also use hybrid architectures, for example, architectures based on a CPU+FPGA, a GPU (Graphics Processing Unit) or an MPPA (Multi-Purpose Processor Array).

[0103] Furthermore, the functional diagram presented in Figure 1 is a typical example of a program, some instructions of which can be carried out by the device described. As such, Figure 1 may correspond to the flowchart of the general algorithm of a computer program in a particular embodiment. Of course, the present invention is not limited to the embodiments described above as examples; it extends to other variants. For example, although the method is described on two coloring techniques, it is possible to train the model on unannotated data from several different domains (i.e. unannotated images of biological samples prepared with several different coloring techniques). Thus, the trained model can be used to segment a wide variety of images of biological samples, colored with different coloring techniques.Furthermore, even if the invention is presented here in the context of a binary segmentation (area of ​​interest or outside the area of ​​interest), it can be generalized to a number of classes strictly greater than two, and to other types of classes (for example, non-tissue area, tumor tissue area and non-tumor tissue area).

Claims

CLAIMS A computer-implemented method for semi-supervised training of a machine learning model to determine a region of interest in an image of a biological sample, the method comprising: receiving (110, 112, 114) training data comprising: ■ of the first images (310) of biological samples prepared with a first staining technique, and information (320) relating to regions of interest in said first images; ■ N pairs of images (410, 420), each pair of images comprising an image (410) of a respective biological sample prepared with an nth staining technique, called the nth second image, and an image (420) similar to an image of the same biological sample prepared with the first staining technique, called the nth third image, N being an integer greater than or equal to 1 and n being an integer between 2 and N+1; and train (120) the machine learning model with the training data;and wherein, said information (320) relating to regions of interest in said first images also include binary segmentation images respectively associated with the first images (310) of biological samples, each binary segmentation image comprising a first set of pixels corresponding to regions of interest (322) of the first image of the corresponding biological sample, and a second set of pixels corresponding to areas (321) of the first image of the corresponding biological sample other than the regions of interest, the binary segmentation images respectively associated with the first images being called verified binary segmentation images, the method also comprising:; a) From each first biological sample image, generate a first binary image with respective segmentation (330) using current parameters of the machine learning model; b) From each n second biological sample image (410), generate an n third binary image with respective segmentation (430) using current parameters of the machine learning model; c) From each n third similar image (420), generate an n third binary image with respective segmentation (440) using current parameters of the machine learning model; d) Calculate (125) a cost function from: ■ of a value representative of the differences between the first binary segmentation images (330) and the corresponding verified binary segmentation images (320); and ■ of a value representative of the differences between the N second binary segmentation images (430) and the corresponding N third binary segmentation images (440); wherein each of the binary segmentation images comprises respectively a first set of pixels corresponding to regions of interest (322, 332, 432, 442) of the corresponding biological sample image, and a second set of pixels corresponding to areas (321, 331, 431, 441) of the corresponding biological sample image other than the regions of interest; and wherein steps a) to d) are repeated so as to minimize the cost function. A method according to claim 1, wherein the machine learning model is a deep learning model. A method according to any one of the preceding claims, wherein for each n third image (420) is generated from the respective n second image (410) by means of a colorimetric transformation.A method according to claim 3, wherein the colorimetric transformation uses a residual circular generative antagonistic network. A method according to any one of the preceding claims, wherein the representative value of the differences between the first binary segmentation images and the corresponding verified binary segmentation images is a function of a mean squared error between the first binary segmentation images and the corresponding verified binary segmentation images; and the N representative values ​​of the differences between the n second binary segmentation images and the corresponding n third binary segmentation images are a function of a mean squared error between the n second binary segmentation images and the corresponding n third binary segmentation images.A method according to any one of the preceding claims, wherein the first staining technique is hematoxylin and eosin staining and the other N staining techniques are based on one or more immunohistochemical markers, such as the Ki-67 antigen, antibodies, or a combination of cytokeratin 8 and 18 antibodies (CK 8 / 18). A method according to any one of the preceding claims, wherein N is equal to 1. A method according to claim 7, wherein the first staining technique is hematoxylin and eosin staining and the second staining technique is based on immunohistochemistry, or conversely, the first staining technique is based on immunohistochemistry and the second staining technique is hematoxylin and eosin staining.A computer-implemented method for determining a region of interest in an image of a biological sample prepared with a staining technique, the method comprising: receiving (130) the image of the biological sample; determining (140) a region of interest of the image by means of a segmentation algorithm using a machine learning model trained according to any one of claims 1 to 8. Computer device comprising a circuit configured to implement a process according to any one of claims 1 to 9. Product computer program comprising instructions to implement steps of the process according to any one of claims 1 to 9, when this program is executed by a processor.

Images