Artificial intelligence microscopy image processing

Abstract

Method for constructing a training database to train a convolutional neural network for improving the quality of a microscopy image, comprising a step (S2) of generating a first series of images using a reconstruction and refocusing algorithm, a step (S4) of selecting one or more images from the first series, a step (S3) of generating a second series of images, and a step (S5) of constructing training sets, each comprising one or more input images formed by one of the respective images from the second series and one or more output images corresponding to the image(s) selected from the first series. Training the neural network with such a database and processing a microscopy image using a neural network thus trained. Corresponding hardware and computer program. Figure for the abstract: Fig. 1

Description

Title of the invention: Processing of microscopy images by artificial intelligence Technical field

[0001] The invention relates to the field of microscopy image processing, particularly in the cell biology sector.

[0002] The invention is of particular but in no way limiting interest for the processing of high-resolution, or super-resolution, images, i.e. images whose spatial resolution is typically on the scale of a hundred nanometers. State of the prior art

[0003] Several known microscopy techniques make it possible to image objects such as biological cells. The known techniques have in particular the disadvantages mentioned below.

[0004] In conventional microscopy, high-resolution detection poses difficulties in automation and in the identification and characterization of objects of interest in a specimen.

[0005] Fourier ptychographic microscopy (FPM) improves performance in terms of resolution and field for a complexity of the same order as conventional microscopy. Obtaining a quality image allowing the localization and characterization of objects is however delicate: it becomes essential to address sharpness problems around the optimal setting, for objects of interest such as cellular inclusions of the parasite type in a red blood cell or different cellular compartments in a white blood cell. In addition, this technique requires a separate setting for each of the acquired modalities, in particular for intensity and phase, and for each of the color channels, i.e. in red, green or blue.

[0006] Digital holography allows access to different focusing planes, but for unsatisfactory levels of resolution. It is known to perform digital refocusing by deep learning on bimodal holography images, containing both intensity and phase information. However, this technique does not allow targeting cellular compartments of interest or correcting differences in intensity and phase, in particular for each of the color channels.

[0007] Generally speaking, the use of artificial intelligence models is known in the field of microscopy image processing, but the effectiveness of neural network processing depends on the quality and diversity of the images. used for model training. This results in experimental difficulties and significant human and computational costs. Statement of the invention

[0008] The invention aims to provide a solution for improving the quality of microscopy images in order in particular to facilitate the understanding and monitoring of biological processes in research and clinical practice, or to enable analysis or epidemiological monitoring laboratories to carry out rapid diagnoses with high sensitivity and specificity.

[0009] A particular aim of the invention is to provide a solution suitable for processing images acquired with conventional microscopes, in particular in bright field, dark field or even fluorescence.

[0010] Another object of the invention is to provide a solution compatible with the acquisition of images of biological specimens which can be either chemically marked or colored or devoid of dye or immuno-marker.

[0011] To this end, the invention relates to a method for constituting a learning database for training a convolutional neural network intended to improve the quality of a microscopy image, this method comprising the following steps: - generating a first series of images from one or more raw images of a region of a specimen comprising at least one object, the images of the first series being generated using a reconstruction algorithm such that the images of the first series represent said region of the specimen according to respective planes distant from each other along a reference direction, - selection of one or more images in the first series to form one or more output images, - generation of training sets forming the training database, each of the training sets comprising: - one or more input images formed by one or more respective images of a second series comprising images of said region of the specimen, the images of the second series representing said region of the specimen according to respective planes distant from each other along said reference direction, - the output image(s).

[0012] It is thus proposed to generate a diversity of images in a computational manner, which makes it possible to constitute the learning database from a limited number of previously acquired raw images.

[0013] The constitution of the training database requires a single selection of the output image(s), which makes it possible to reduce the corresponding human and / or calculation time.

[0014] Furthermore, the constitution of the learning database can be carried out on the basis of raw images acquired using various microscopy techniques, including, but not limited to, Fourier ptychographic microscopy and digital holography, to image objects contained in biological specimens which then no longer need to be stored.

[0015] Said first series of images, which serves at least to constitute the output image(s) used in the training database, makes it possible to represent said region of the specimen according to different planes along the reference direction, that is to say according to different positions of the focal plane relative to the specimen.

[0016] The notion of image quality is linked to sharpness, or perceptibility or more generally to the possibility of extracting information from a given object of interest taking into account the position of the focal plane of the microscope object relative to the specimen, on an image reconstructed by calculation.

[0017] Each of the training sets can therefore comprise several input images and several output images.

[0018] In a preferred embodiment, for each of the training sets, the input images comprise a first input image comprising intensity information and a second input image comprising phase information, said output image comprising a first output image comprising intensity information and a second output image comprising phase information.

[0019] In other words, the invention is preferably implemented with bimodal imaging, in particular in intensity and in phase.

[0020] Alternatively or additionally, for each of the training sets, the input images may comprise images that each correspond to a respective color channel and the output images may similarly comprise images that each correspond to a respective color channel.

[0021] Of course, one or more of the training sets may each comprise input and output images comprising bimodal information relating to a single color channel or conversely comprising unimodal information, for example intensity or phase, for several color channels. Numerous combinations may be envisaged without departing from the scope of the invention.

[0022] In one embodiment, the second series of images comprises images from the first series.

[0023] This makes it possible to increase the diversity of input images in the training database, without increasing the number of raw image acquisitions.

[0024] Of course, the second series of images may include both images from the first series and other images otherwise acquired or reconstructed by computation.

[0025] For example, the second series may comprise images generated from one or more raw images of said region of the specimen, the images of the second series being generated using a reconstruction algorithm such that the images of the second series represent said region of the specimen in respective planes distant from each other along a reference direction.

[0026] The raw images used to generate the images of the first series may comprise the raw images used to generate the images of the second series. In particular, the raw images used to generate the images of the second series may be a portion of the raw images used to generate the images of the first series. Alternatively, the raw images used to generate the images of the first series may be different from those used to generate the images of the second series.

[0027] The invention also relates to a method as described above and further comprising a step of acquiring said raw images.

[0028] According to a first variant, the selection step is carried out by a human operator.

[0029] The selection of the image used as the output image in the training database by a human operator makes it possible to rely on the skills of a biology expert in order to identify an image representing the object(s) of interest in the most appropriate manner within a series of images.

[0030] According to a second variant, the selection step is carried out by a computer system.

[0031] These variants can be combined, for example and in a non-limiting manner by using the computer system to carry out a preselection and by involving the human operator to select the output image from a series of images thus preselected in the first series of images.

[0032] The invention also relates to a method for training a convolutional neural network intended to improve the quality of a microscopy image, using the learning database constituted by a method as described above.

[0033] The training is preferably performed using a training algorithm such as an error gradient backpropagation algorithm.

[0034] The invention also relates to a method for processing a microscopy image using a convolutional neural network trained with such a method.

[0035] The invention thus makes it possible to generate high-quality, informative, manipulable digital data that does not require the conservation of biological specimens.

[0036] According to another aspect, the invention relates to a device configured to implement a method for constituting a learning database as described above.

[0037] This device preferably comprises a processing unit for generating said first series of images and the training sets and for executing the reconstruction algorithm.

[0038] In one embodiment, in particular when the device is intended to allow a human operator to carry out the selection step, the device comprises means for selecting the output image(s).

[0039] By way of example, these selection means may take the form of a selection interface and / or pointing means.

[0040] According to yet another aspect, the invention relates to a computer program comprising executable instructions which, when executed by computer, implement the steps of a method for constituting a learning database as described above, of a training method as described above and / or of an image processing method as described above.

[0041] The invention makes it possible to obtain numerous advantages compared to the state of the prior art, among which: - an improvement in robustness, in particular by enabling good image correction in intensity and / or phase over a large mechanical adjustment range covering various focal planes, while reducing calculation times; - an improvement in detection sensitivity without degradation of specificity; - a reduction in error density in out-of-focus areas; - a decrease in the density of errors in the sharp focus area; - compatibility with the use of unstained or chemically labeled slides and with traditional slide scanners and microscopes; - a possibility of reducing the mechanical complexity of microscopes; - a medical-economic gain, likely to promote personalized medicine.

[0042] Other advantages and characteristics of the invention will appear on reading the detailed, non-limiting description which follows. Brief description of the drawings

[0043] The following detailed description refers to the accompanying drawings in which:

[0044] [Fig. 1] schematically represents a method of constituting a database of training data in accordance with a non-limiting embodiment of the invention, this method comprising a step of acquiring raw images by transmission microscopy, a step of generating a first series of images from raw images thus acquired, a step of selecting reference images from among the images of the first series, a step of generating a second series images from said raw images and a step of constituting the learning database using the reference images and the images of the second series;

[0045] [Fig.2] illustrates the raw image acquisition step of the method of [Fig.l];

[0046] [Fig.3] illustrates the step of generating the first series of images of the process of the [Fig.l];

[0047] [Fig.4] illustrates the reference image selection step of the method of [Fig.l];

[0048] [Fig.5] illustrates the step of generating the second series of images of the process of the [Fig.l]. Detailed description of embodiments

[0049] [Fig.l] illustrates steps S1-S5 of a method for constituting a training database for a convolutional neural network in accordance with the invention.

[0050] Generally, this method comprises a step S1 of acquiring raw images, a step S2 of generating a first series of images, a step S3 of generating a second series of images, a step S4 of selecting images and a step S5 of constituting the learning database.

[0051] In this example, the raw images are acquired in step SI using a microscope equipped with a light source comprising a matrix of NI = 9 light-emitting diodes and a camera making it possible to acquire images of 3584 x 2688 pixels.

[0052] The following document describes an example of acquisition with a matrix light source that can be used within the framework of the invention: “Konda et al., Fourier pty-chography: current applications and future promises, Vol. 28, No. 7, 30 March 2020, Optics Express”.

[0053] The imaging is performed on a biological specimen comprising in this example blood cells forming objects of interest.

[0054] Conventionally, the specimen is arranged on a blade in the form of a layer whose thickness extends along a Z direction, called “reference direction” in this document.

[0055] A manual or automatic adjustment of the focal plane is first carried out by positioning the focal plane at a chosen position of the specimen in the Z direction, so as to obtain a relatively clear image of a region of the specimen comprising at least one object of interest, this region being located in a central zone of the field of view. By convention, the reference focus thus obtained corresponds to a position z = 0 of the focal plane relative to the specimen in the Z direction.

[0056] The raw images of the specimen are acquired in step SI by modifying the relative position of the focal plane with respect to the specimen in the Z direction, from z = -6 pm to z =

[0057]

[0058]

[0059]

[0060] +6 pm in steps of 2 pm, i.e. a number of N2 = 7 focal plane positions. Referring to Figure 2, for each of these N2 focal plane positions, a stack of NI raw images IB; _ À. is acquired. In this example, each of the raw jpf = 1 images thus acquired is preprocessed so as to retain only a central region of the field of view of dimension 256 x 256 pixels. In this example, said first and second series of images are generated from the raw images preprocessed according to a ptychographic reconstruction technique, using a reconstruction algorithm of the “PIE” type (from the English “Ptychographic Iterative Engine”). This type of algorithm is well known in the transmission microscopy sector. The general principles of such an algorithm are for example described in the following document: “Maiden et al., An improved pty-chographicalphase retrieval algorithm, Ultramicroscopy 109, 2009, 1256-1262”. Concerning step S2 (see figures 1 and 3), from the stack of preprocessed raw images / / j4 corresponding to the reference position of the focal plane, i.e. z = 0, a first series of images / ^1,2} is generated using a reconstruction and focus change algorithm. In this example, the algorithm used, called "EPRY", is described in the following document: "Ou et al., Embeddedpupil function recovery for Fourier ptychographic microscopy, Optical Society of America, 2014, D01:10.1364 / OE.22.004960". The calculated position of the focal plane varies in this example from z = -3 pm to z = +3 pm in steps of 0.1 pm, corresponding to N3 = 60 generated images. The images 02 je |a first series have in this example a size of 512 x 512 pixels.

[0061] For each of these focal plane positions, the algorithm is in this example configured to generate a first image] including information intensity and a second image = 1. : N3 including phase information. In other words, the first series of images N3 includes a first sub- series of intensity images j j1 and a second sub-series of phase images N3'

[0062] The images of each of these sub-series thus represent said region of the specimen and thus the at least one object of interest that it contains according to respective planes distant from each other along the Z direction, in a more or less clear, informative or at least usable manner by a biological expert, it being understood that, for each of these sub-series, the most usable or useful image for a biological expert does not necessarily correspond to the image associated with a focal plane position of z = 0.

[0063] In this example, with reference to Figures 1 and 4, the selection step S4 is carried out by a biology expert who selects one of the images of intensity jp and one of the phase images / [2 within the first series of images j 12 1: NI . In this description, the intensity images jq1 and phase / 12 thus selected are respect tively called “first output image” and “second output image” because they will be used to constitute output images of the training database (see further below).

[0064] Concerning step S3 (see figures 1 and 5), from the N2 stacks of preprocessed raw images, in this example a second series of images r?1-2 is generated in using the aforementioned EPRY algorithm. Of course, another reconstruction algorithm can be used. The 77 h2 images of this second series have in this example a size of 512 x 512 pixels.

[0065] In a manner similar to the reconstruction of the first series of images, the algorithm is in this example configured to generate, for each of the N2 positions of the focal plane, a first image 72f ] comprising intensity information and a second image comprising phase information. In other words, the second series of images / 2-'^ V2 also includes a first sub-series of images of intensity 771 f and a second sub-series of images of phase I22 lzi=\ :N2

[0066] Thus, the images of each of the sub-series of the second series represent said region of the specimen and thus the at least one object of interest that it contains according to respective planes distant from each other along the Z direction, in a more or less clear, informative or at least usable manner by a biological expert, it being understood that, for each of these sub-series, the most usable or useful image for a biological expert does not necessarily correspond to the image associated with a position of the focal plane of z = 0.

[0067] In this example, the images j^1'2 of the second series are used to constitute input images of the training database.

[0068] More precisely, step S5 of constituting the learning database comprises constituting several learning sets which each comprise a first input image formed by a respective one of said first images / 21 of the second series, a second input image formed by a respective one of said second images 722 of the second series, said first output image j| ' and said second output image / ]2.

[0069] In this embodiment, the convolutional neural network implemented is a model known as "U-NET". In a manner known per se, this model has an architecture comprising, on the one hand, an encoder provided with an assembly of convolution layers and layers known as "max-pooling" and, on the other hand, a decoder provided with an assembly of convolution layers and upsampling layers known as "upsampling / transposed convolution". The encoder and the decoder are connected by elements known as "skip connections". The following document describes such a model: "Ronneberger et al., U-net: Convolutional networks for biomedical image segmentation, In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241, Springer, Cham, 2015".

[0070] The neural network is trained with the training database constituted in the manner described above, using a training algorithm, in this case an error gradient backpropagation algorithm.

[0071] In an alternative embodiment, the training is carried out with a learning database comprising both the sets described above, i.e. sets in which the input images are images from the second series, and further comprising sets in which the input images are images from the first series.

[0072] Other variants may be implemented within the scope of the invention to form the training database. For example, the input images of the training sets may comprise only images from the first series. In other words, said second series may not be generated in the manner described above but simply be constituted from the images of the first series.

[0073] A neural network trained with such a database makes it possible to reconstruct bimodal, intensity and phase imaging, improving the quality of microscopy images by modifying the position of the focal plane selectively for different objects of interest.

[0074] A neural network thus trained makes it possible in particular to obtain an image having several objects of interest which appear clearly on the basis of a microscopy image on which all or part of these objects appear blurred, including when two of said objects of interest are superimposed in the reference direction.

[0075] The preceding examples are intended to illustrate the principle of the invention which can be implemented in different ways and / or by combining features and variants described here. For example, before acquiring the raw images, a automated pre-identification of one or more objects of interest.

[0076] In an alternative embodiment, the selection of the output images can be carried out automatically by a computer system, or both with such a computer system configured for example so as to preselect certain images and then by a human operator carrying out the selection on the basis of the images thus preselected.

[0077] In a variant, the input images of the training database, or some of them, are respective parts of images reconstructed on the basis of the raw images, whether for example parts of images of the first series and / or of the second series, the size of which may be 128 x 128 pixels.

[0078] Of course, the invention described above can be implemented in a similar manner not on bimodal imagery but on monomodal imagery, for example either in intensity or in phase, that is to say in particular by using a single input image and a single output image in each of the training sets and / or during the inference phase. Similarly, the training can be carried out by using several monomodal or bimodal input and output images which each correspond to a respective color channel.

Claims

Claims

1. Method for constituting a learning database for training a convolutional neural network intended to improve the quality of a microscopy image, this method comprising the following steps: - generation (S2) of a first series of images ^2) from one or more raw images Go7 = 1;N2) of a region of a *■1 =1: NI specimen comprising at least one object, the images of the first series being generated using a reconstruction algorithm such that the images of the first series represent said region of the specimen according to respective planes distant from each other along a reference direction, - selection (S4) of one or more images Qq !, jp) in the first series to form one or more output images, - generation (S5) of training sets forming the training database, each of the training sets comprising: - one or more input images ( / 212) formed by one or more respective images of a second series ( / 21,2 { ^1 comprising images of said region of the specimen, the images of the second series representing said region of the specimen according to respective planes distant from each other along said reference direction, - the output image(s) Qq1, / J2)-xy;

2. The method of claim 1, wherein, for each of the training sets, the input images (jp^2) comprise a first input image ( / 21) comprising intensity information and a second input image (jp2) comprising phase information, the output images comprising a first output image ( / p) comprising intensity information and a second output image (Qq2) comprising phase information.

3. The method of claim 1 or 2, wherein the second series of images comprises images from the first series.

4. A method according to any one of claims 1 to 3, wherein the second series ( / p1 '2 ) comprises images generated from of one or more raw images Qg7 of said region of the specimen, the images of the second series Qp1,2 ) being generated (S3) using a reconstruction algorithm so that the images of the second series Q21 '2 i ) represent said region of the specimen according to respective planes distant from each other along a reference direction.

5. Method according to any one of claims 1 to 5, comprising a step of acquiring (SI) said raw images:^2)-

6. Method according to any one of claims 1 to 5, wherein the selection step (S4) is carried out by a human operator and / or by a computer system.

7. A method of training a convolutional neural network for improving the quality of a microscopy image, using the training database formed by a method according to any one of claims 1 to 6, using a training algorithm which is preferably an error gradient backpropagation algorithm.

8. A method of processing a microscopy image using a convolutional neural network trained with a method according to claim 7.

9. Device configured to implement a method according to any one of claims 1 to 6, comprising a processing unit for generating said first series of images ( / j1 '2 ;VJ ct the training sets and for executing the reconstruction algorithm, and means for selecting the output image(s).

10. A computer program comprising executable instructions which, when executed by computer, implement the steps of a method for building a learning database according to any one of claims 1 to 4, a training method according to claim 7 and / or a processing method according to claim 8.

Images