Method for classifying an input sequence of images representative of particles in a sample over time

Abstract

The invention relates to a method for classifying an input sequence of images representing target particles (11a-11f) in a sample (12) over time, the method being characterized in that it comprises the following steps performed by a data processing device (20) of a client (2), namely: (b) concatenating the input images of the sequence into a three-dimensional stack; (c) directly classifying the three-dimensional stack using a convolutional neural network, CNN.

Description

Technical Field

[0001] This invention relates to the field of optical acquisition of biological particles. These biological particles can be microorganisms, such as bacteria, fungi, or yeast. They can also include cells, multicellular organisms, or any other type of contaminant particle, including dust.

[0002] This invention is particularly advantageous for analyzing the state of biological particles, for example, to determine the metabolic state of bacteria after antibiotic application. This invention allows for, for example, the production of an antibacterial spectrum by bacteria. Background Technology

[0003] An antibacterial spectrum is a laboratory technique designed to test the phenotype of bacterial strains in relation to one or more antibiotics. Conventionally, an antibacterial spectrum is generated by culturing a sample containing both bacteria and antibiotics.

[0004] European Patent Application No. 2603601 describes a method for generating an antimicrobial spectrum by visualizing the state of bacteria after an incubation period in the presence of antibiotics. To visualize the bacteria, they are labeled with fluorescent markers, revealing their structure. The fluorescence of the markers is then measured, allowing determination of whether the antibiotics are effectively acting on the bacteria.

[0005] The routine procedure for determining the effectiveness of antibiotics against bacterial strains involves obtaining a sample containing the strain (e.g., from a patient, animal, food batch, etc.) and then sending the sample to an analytical center. Upon receiving the sample, the analytical center first culturees the bacterial strain to obtain at least one colony for 24 to 72 hours. Several samples containing various antibiotics and / or various concentrations of antibiotics are then prepared from this colony and incubated again. After a new incubation period of 24 to 72 hours, each sample is manually analyzed to determine the effectiveness of the antibiotic. The results are then returned to the physician for the application of antibiotics and / or the most effective antibiotic concentration.

[0006] However, the labeling process is particularly lengthy and complex, and these chemical markers are cytotoxic to bacteria. Therefore, this visualization method does not allow for multiple observations of bacteria during culture, necessitating a sufficiently long incubation period, approximately 24 to 72 hours, to ensure the reliability of the measurements. Other methods for visualizing biological particles utilize microscopy, allowing for non-destructive measurements of the sample.

[0007] Digital holographic microscopy (DHM) is an imaging technique that overcomes the depth-of-field limitations of conventional optical microscopy. Illustratively, it involves recording a hologram formed by the interference between light waves diffracted from the observed object and a spatially coherent reference wave. This technique is described in Myung K. Kim's commentary entitled "Principles and techniques of digital holographymicroscopy," published in the January 2010 issue of SPIE Reviews, Volume 1, Issue 1.

[0008] Recently, the use of digital holographic microscopy for automated identification of microorganisms has been proposed. Therefore, international application WO2017 / 207184 describes a method for acquiring particles that integrates simple acquisition without the need for focusing associated with digital reconstruction of the focus, allowing for the observation of biological particles while limiting acquisition time.

[0009] Typically, this solution allows for the detection of bacterial structural modifications after only about 10 minutes of incubation in the presence of antibiotics, and the assessment of sensitivity (detecting the presence or absence of division or indicating the pattern of division) at the end of two hours, unlike the conventional procedures described above which may take several days. In fact, because the measurements are non-destructive, analysis can be performed very early in the culture process without the risk of damaging the sample and thus prolonging the analysis time.

[0010] It is even possible to track particles across multiple consecutive images, thus creating a film representing the evolution of the particles over time (since the particles do not change after the first analysis), in order to visualize their behavior, such as their speed of movement or their cell division process.

[0011] Therefore, it is understandable that visualization methods produce excellent results. The difficulty lies in interpreting these images or the film itself, if the goal is, for example, to draw conclusions about the susceptibility of bacteria to antibiotics present in the sample, especially in an automated manner.

[0012] Various techniques have been proposed, ranging from simply counting bacteria over time to so-called morphological analysis aimed at detecting specific “configurations” through image analysis. For example, when bacteria are preparing to divide, two poles appear in the distribution, long before the division itself, resulting in two distinct parts of the distribution.

[0013] The 2014 paper by Choi J., Yoo J., Lee M., et al., entitled “A rapid antimicrobial susceptibility test based on single-cell morphological analysis”, Science Translational Medicine, 6(267), https: / / doi.org / 10.1126 / scitranslmed.3009650, proposed combining these two techniques to assess antibiotic efficacy. However, as the authors emphasize, their method requires highly accurate calibration of a number of thresholds that are strongly dependent on the nature of the morphological changes induced by the antibiotic.

[0014] The article titled "Phenotypic Antimicrobial Susceptibility Testing with Deep Learning Video Microscopy," published in 2018 by Yu H., Jing W., Iriya R., et al., Analytical Chemistry, 90(10), 6314-6322, https: / / doi.org / 10.1021 / acs.analchem.8b01128, describes a deep learning-based approach. The authors propose using a convolutional neural network (CNN) to extract morphological features and features related to bacterial movement. However, this solution has proven to be highly computationally intensive, requiring a large training image library to train the CNN.

[0015] Therefore, the objective technical problem of the present invention is to provide a more efficient and less resource-intensive solution for classifying images of biological particles. Summary of the Invention

[0016] According to a first aspect, the present invention relates to a method for classifying an input image sequence of target particles in a representative sample, characterized in that it includes the following steps performed by a data processing device on a client side:

[0017] (b) Concatenate the input images in the sequence in the form of a three-dimensional stack;

[0018] (c) Direct classification of the three-dimensional stack using a convolutional neural network (CNN), wherein the CNN consists of a series of convolutional blocks consisting of: a 3D convolutional layer that applies a four-dimensional filter to a four-dimensional input feature map to generate a four-dimensional output feature map, an activation layer and a 3D pooling layer, then a flattening layer, and finally one or more fully connected layers.

[0019] Based on the advantageous but non-restrictive features:

[0020] The particle is displayed in a uniform manner in each input image, specifically centered and aligned in a predetermined direction.

[0021] The method includes step (a): extracting an overall image of the sample from each input image in order to display the target particle in the uniform manner.

[0022] Step (a) includes: for each input image, segmenting the overall image to detect the target particles in the samples, and then re-cropping the input image according to the detected target particles.

[0023] Step (a) includes: acquiring the overall image from the intensity image of the sample acquired by the observation device.

[0024] The three-dimensional stack has two spatial dimensions and one temporal dimension, and the filter and feature map have the spatial dimension and the temporal dimension as the three first dimensions, and semantic depth as the fourth dimension.

[0025] The depth of the filters in the 3D convolutional layer is equal to the depth of the input feature map, and the depth of the output feature map is equal to the number of filters in the 3D convolutional layer.

[0026] The CNN consists of only two convolutional blocks.

[0027] The method includes step (a0): training the parameters of the classifier from a training library of previously classified image sequences of particles in the sample by a data processing device of the server.

[0028] According to a second aspect, a system is provided for classifying an input image sequence representing target particles in a sample over time, the system comprising at least one client including a data processing device, characterized in that the data processing device is configured to implement:

[0029] - The input images in the sequence are strung together in the form of a three-dimensional stack;

[0030] - The three-dimensional stack is directly classified using a convolutional neural network (CNN), wherein the CNN consists of a series of convolutional blocks, each consisting of a 3D convolutional layer that applies a four-dimensional filter to a four-dimensional input feature map to generate a four-dimensional output feature map, an activation layer and a 3D pooling layer, followed by a flattening layer, and finally one or more fully connected layers.

[0031] According to an advantageous but not limiting feature, the system also includes: a device for observing the target particles in a sample.

[0032] According to the third and fourth aspects, a computer program product is proposed, including code instructions for executing the method according to the first aspect for classifying an input image sequence of target particles in a representative sample; and a storage device readable by a computer device is proposed, on which the computer program product includes code instructions for executing the method according to the first aspect for classifying an input image sequence of target particles in a representative sample. Attached Figure Description

[0033] Other features and advantages of the invention will become apparent from the following description of preferred embodiments. This description will be provided with reference to the accompanying drawings, in which:

[0034] - Figure 1 This is a diagram of an architecture for implementing the method according to the invention;

[0035] - Figure 2 This illustrates a preferred embodiment of the method according to the invention for observing particles in a sample.

[0036] Examples of devices for children;

[0037] - Figure 3a The acquisition of an input image is illustrated in an embodiment of the method according to the present invention;

[0038] - Figure 3b The acquisition of an input image is shown in a preferred embodiment of the method according to the present invention;

[0039] - Figure 4 The steps of a preferred embodiment of the method according to the present invention are shown;

[0040] - Figure 5 An example of a convolutional neural network architecture used in a preferred embodiment of the method according to the present invention is shown. Detailed Implementation

[0041] Architecture

[0042] This invention relates to a method for classifying an input image sequence representing particles 11a-11f (referred to as target particles) present in a sample 12. It should be noted that this method can be implemented simultaneously for all or some of the particles 11a-11f present in the sample 12, with each particle being sequentially considered a target particle.

[0043] As will be seen, this approach includes machine learning components, particularly convolutional neural networks, CNNs.

[0044] The input or learning data is of image type and represents target particles 11a-11f in sample 12 (in other words, it involves images of samples in which the target particles are visible). The sequence consists of multiple input images of the same target particles 11a-11f over time. As will be seen, if applicable, multiple image sequences of particles 11a-11f in sample 12 can be used as input, if several particles are taken into account.

[0045] Sample 12 consists of a liquid, such as water, buffer solution, culture medium or reaction medium (which may or may not include antibiotics), in which the particles to be observed 11a-11f are located.

[0046] As an alternative embodiment, sample 12 may be in the form of a solid medium, preferably translucent, such as agar, in which particles 11a-11f are located. Sample 12 may also be a gaseous medium. Particles 11a-11f may be located inside the medium or even on the surface of sample 12.

[0047] Particles 11a-11f can be microorganisms such as bacteria, fungi, or yeast. This can also include cellular, multicellular organisms, or any other type of contaminant particle, including dust. Throughout the remainder of the description, preferred examples where the particles are bacteria (and as will be seen, sample 12 includes antibiotics) will be used. The observed particle sizes 11a-11f vary between 500 nanometers and several hundred micrometers, or even a few millimeters.

[0048] The “classification” of the input image sequence involves determining at least one category from a set of possible descriptive categories of the image. For example, in the case of bacterial type particles, it is possible to have a binary classification, that is, two possible action categories, namely “division” or “no division,” indicating resistance to antibiotics or no resistance to antibiotics, respectively. The present invention will not be limited to any particular classification, even though examples of binary classification describing the effects of antibiotics on said target particles 11a-11f will be primarily described.

[0049] This method utilizes server 1 and client 2 in, for example Figure 1The architecture shown is implemented as follows. Server 1 is the learning device (implementing the learning method), and client 2 is the operating device (implementing the classification method), such as a doctor's or hospital terminal.

[0050] These two devices 1 and 2 can be combined, but preferably server 1 is a remote device and client 2 is a consumer device, particularly a desktop computer, laptop computer, etc. Client device 2 is advantageously connected to observation device 10 so as to be able to directly acquire the input image (or, as will be seen below, “raw” acquisition data such as the overall image of sample 12 or even the electromagnetic matrix), typically for direct processing. Alternatively, the input image will be loaded onto client device 2.

[0051] In all cases, for the exchange of data, each of devices 1 and 2 is typically a remote computer device linked to a local area network (LAN) or wide area network (WAN) such as the Internet. Each device includes a processor-type data processing device 3, 20 and a data storage device 4, 21 such as computer memory, for example, flash memory or a hard disk. Client 2 typically includes a user interface 22, such as a screen for interaction.

[0052] Server 1 advantageously stores a learning database, i.e., a collection of image sequences of particles 11a-11f that have been classified under various conditions (e.g., associated with labels indicating "with division" or "without division" indicating sensitivity or resistance to antibiotics) (see below). It should be noted that the learning data can be associated with labels defining test conditions, such as "strain," "antibiotic condition," "time," etc., for bacterial cultures.

[0053] collection

[0054] As explained, this method can directly use any image of the target particles 11a-11f obtained in any way as input. The method preferably begins at step (a), acquiring the input image from the data provided by the observation device 10.

[0055] In a known manner, those skilled in the art will be able to use digital holographic microscopy (DHM) techniques, particularly as described in International Application WO2017 / 207184. Specifically, an intensity image of sample 12, referred to as a hologram, can be acquired that is not focused on the target particle (referred to as an "out-of-focus" image), and can be processed by a data processing device (integrated into device 10 or those devices 20 of client 2, e.g., see below). It will be understood that the hologram in some way "represents" all particles 11a-11f of the sample.

[0056] Figure 2An example of a device 10 for observing particles 11a-11f present in sample 12 is shown. Sample 12 is positioned between a spatially and temporally coherent (e.g., laser) or pseudo-coherent (e.g., light-emitting diode, laser diode) light source 15 and a digital sensor 16 sensitive to the spectral range of the light source. Preferably, light source 15 has a narrow spectral width, for example, narrower than 200 nm, narrower than 100 nm, or even narrower than 25 nm. Throughout the remainder of the description, the central emission wavelength of the light source refers to that in the visible region. Light source 15 emits a coherent signal Sn directed onto a first surface 13 of the sample, transmitted, for example, by a waveguide such as an optical fiber.

[0057] Sample 12 (typically a culture medium, as explained) is contained in an analytical chamber vertically defined by a download slide and a top slide (e.g., a conventional microscope slide). The analytical chamber is laterally defined by adhesive or any other sealing material. The download and top slides are transparent to the wavelength of light source 15, and the sample and analytical chamber allow, for example, more than 50% of the light source wavelength to pass through on the download slide under normal incidence.

[0058] Preferably, particles 11a-11f are arranged in sample 12 on an upper slide. For this purpose, the underside of the upper slide includes ligands for allowing particle attachment, such as polycations (e.g., poly-L-lysine) in a microbial background. This allows for the inclusion of particles with a thickness equal to or close to the depth of field of the optical system, i.e., less than 1 mm (e.g., a tube lens), preferably less than 100 μm (e.g., a microscope objective). Particles 11a-11f, however, can move within sample 12.

[0059] Preferably, the device includes an optical system 23, which, for example, consists of a microscope objective lens and a tube lens, positioned in the air and maintained at a fixed distance from the sample. The optical system 23 may optionally be equipped with a filter, which may be located in front of the objective lens or between the objective lens and the tube lens. The optical system 23 is characterized by an optical axis; an object plane (also referred to as the focal plane) located at a distance from the objective lens; and an image plane, which is conjugate to the object plane by the optical system. In other words, an object located in the object plane has a clear image of that object in the image plane (also referred to as the focal plane). The optical properties of the system 23 are fixed (e.g., a fixed-focus optics device). The object plane and the image plane are orthogonal to the optical axis.

[0060] The second surface 14 of the image sensor 16, facing the sample, is located in or near the focal plane. The sensor, such as a CCD or CMOS sensor, comprises a periodic two-dimensional array of sensitive primary sites, and proximity electronics that adjust the exposure time and reset the sites in a manner known per se. The output signal of the primary site depends on the amount of radiation incident on the site within a spectral range during the exposure time. This signal is then converted, for example, by the proximity electronics into image points or “pixels” of a digital image. Thus, the sensor produces a digital image in the form of a C-column, L-row matrix. The coordinates (c, l) in the matrix, where each pixel corresponds in a manner known per se to the position of Cartesian coordinates (x(c, l), y(c, l)) in the focal plane of the optical system 23, for example, the position of the center of the rectangular sensitive primary sites.

[0061] The pitch and fill factor of the periodic array are selected to conform to the Nyquist-Shannon criterion regarding the size of the observed particles, so as to define at least two pixels for each particle. Therefore, image sensor 16 acquires a transmission image of the sample across the spectral range of the light source.

[0062] The image acquired by image sensor 16 includes holographic information, in that it is produced by the interference between the waves diffracted by particles 11a-11f and a reference wave that has passed through the sample without interacting with it. It is evident, as stated above, that in the context of a CMOS or CCD sensor, the acquired digital image is an intensity image, and phase information is thus encoded in this intensity image.

[0063] Alternatively, the coherent signal Sn originating from the light source 15 can be divided into two components, for example, by means of a translucent plate. The first component then serves as a reference wave, and the second component is diffracted by the sample 12, wherein the image in the image plane of the optical system 23 is generated by the interference between the diffracted wave and the reference wave.

[0064] Reference Figure 3a In step (a), multiple overall images of sample 12 can be reconstructed from the hologram, and then each input image can be extracted from the overall images of the sample.

[0065] In practice, it can be understood that the target particles 11a-11f must be represented in a uniform manner in each input image, specifically centered and aligned in a predetermined direction (e.g., horizontal). The input images must also have a standardized size (it is also expected that only target particles 11a-11f will be seen in the input images). Therefore, the input images are referred to as "thumbnail" and can be defined, for example, to a size of 250×250 pixels. For the desired sequence of input images, for example, one image is taken every minute over a 120-minute time interval, resulting in a sequence of 120 input images.

[0066] As explained, the reconstruction of each overall image is performed by the data processing device of device 10 or by the devices 20 of client 2.

[0067] Typically, (for the acquisition time) a series of complex matrices called “electromagnetic matrices” are constructed to model the wavefront of the light wave propagating along the optical axis based on the intensity image (hologram) of sample 12 for multiple deviations relative to the focal plane of optical system 23 (especially deviations located in the sample).

[0068] These matrices can be projected into real space (e.g., via the Hermitian standard) to form a stack of overall images at various focal lengths.

[0069] From the above, the average focal length can be determined (and the corresponding overall image is selected, or it can be recalculated from the hologram), or the optimal focal length for the target particle can be determined (and the corresponding overall image is selected again, or it can be recalculated from the hologram).

[0070] In all cases, refer to Figure 3b Step (a) advantageously includes segmenting the overall image to detect the target particles in the samples, followed by cropping. Specifically, each input image can be extracted from one of the overall images of the samples to represent the target particles in the uniform manner.

[0071] Generally, segmentation allows for the detection of all particles of interest by removing artifacts such as filaments or microcolonies to improve one or more overall images. Then, one of the detected particles is selected as the target particle, and the corresponding thumbnail is extracted. As explained, this can be done for all detected particles.

[0072] The partitioning can be implemented in any known manner. Figure 3bIn the example, fine segmentation is first performed to eliminate artifacts, followed by coarse segmentation, in this case for detecting particles 11a-11f. Those skilled in the art can use any known segmentation technique.

[0073] In order to obtain the input image sequence for the target particles 11a-11f, a tracking technique can be implemented to track any movement of the particles from one overall image to the next.

[0074] It should be noted that pooling can be performed on all input images acquired for the sample (for multiple particles or even all particles of sample 12, this is done over time) to form a descriptive library of sample 12 (in other words, a descriptive library of the experiment), such as Figure 3a As can be seen on the right, this descriptive library is specifically copied to storage device 21 of client 2. Refer to the "field" level, which is the counterpart to the "particle" level. For example, if particles 11a-11f are bacteria, and sample 12 contains or does not contain antibiotics, then the descriptive library contains all information about the growth, morphology, internal structure, and optical properties of these bacteria throughout the collection field. As will be seen, this descriptive library can be transferred to server 1 to be included in the learning library.

[0075] stack

[0076] As will be seen, the difference with this method is that it can work directly on the input image sequence without working image by image or extracting feature maps in an intermediate way. Furthermore, it will be seen that a simple and lightweight CNN is sufficient to perform reliable and efficient classification.

[0077] Reference Figure 4 The method includes step (b) of concatenating the sequence of input images in the form of a three-dimensional stack (in other words, a 3D "stack"). More specifically, the input images all have the same size and form a matrix sequence, so they only need to be stacked in the order of the input images to obtain a three-dimensional stack.

[0078] Therefore, this 3D stack can be viewed as an image with as many channels as the acquisition moment, as will be seen below, even though this method processes the stack in a highly original way as a single 3D object with a single channel (e.g., 250×250×120 if there is an input image of size 250×250 and one image is acquired per minute over 120 minutes), just as an RGB image is a 2D object with three channels. The first two dimensions are typically spatial dimensions (i.e., the size of the input image), while the third dimension is the "time" dimension (the acquisition moment).

[0079] Preferably, step (b) includes down-sampling the three-dimensional stack, i.e., reducing the size of the input.

[0080] The downsampling can be implemented in the time dimension and / or the spatial dimension of the stack, preferably in both.

[0081] Specifically:

[0082] - Relative to the time dimension, this can be achieved by dividing the acquisition period into n intervals and selecting n+1 images from the sequence corresponding to the ends of these intervals. For example, by retaining only 5 images from a sequence of 120 images, this is equivalent to taking one image every 120 / 4 = 30 minutes (specifically, images acquired at the ends of 1, 30, 60, 90, and 120 minutes). However, it is still possible to select images in a non-uniform manner (e.g., selecting more images at the beginning of the acquisition period than at the end).

[0083] - Relative to the spatial dimension, any image downsampling technique with a given sampling factor (on each axis in order to maintain scale) such as factor 2 can be used to reduce it.

[0084] Combining the two downsampling methods mentioned above results in a stack with a size of 250×250×120 being converted to a stack with a size of 125×125×5 (the stack size is reduced by almost 100 times).

[0085] It should be noted that this downsampling can actually occur before the stack is generated (by selecting and modifying the input image).

[0086] Classification

[0087] In step (c), the 3D stack is directly classified using a suitable convolutional neural network, referred to as a "3D CNN," because it is capable of handling 3D objects as stacks. In fact, as mentioned, it is important to understand that the 3D stack is processed by the CNN as a single 3D object (with a single channel) rather than a 2D object with multiple channels (as is the case for RGB images, for example).

[0088] The term direct classification or "end-to-end" is understood to mean that at least one feature map of the target particles 11a-11f is not extracted separately: it can be understood that the CNN naturally has internal states in the form of feature maps, but these states are never sent outside the CNN, which has a classification result as the only output.

[0089] As a reminder, CNNs are generally well-suited for visual tasks, more specifically for image classification. Typically, CNNs use multiple convolutional layers, and current 3D CNNs use at least one 3D convolutional layer to model the spatiotemporal dependencies between various input images.

[0090] "3D convolutional layer" is understood to mean a convolutional layer that applies a four-dimensional filter, and thus can work on multiple channels of an already three-dimensional stack, i.e., a four-dimensional feature map. In other words, a 3D convolutional layer applies a four-dimensional filter to a four-dimensional input feature map to generate a four-dimensional output feature map. The fourth and final dimension is semantic depth, just like in any feature map.

[0091] This should be distinguished from convolutional layers, which only work on three-dimensional feature maps representing two-dimensional objects (images) across several channels.

[0092] The concept of 3D convolution may seem counterintuitive, but it encapsulates the concept of a convolutional layer, which simply expects to apply multiple “filters” with a depth equal to the number of input channels (i.e., the depth of the input feature map) by scanning them across all dimensions of the input (which is 2D for an image), where the number of filters defines the output depth.

[0093] Therefore, the 3D convolution referred to in this paper applies a four-dimensional filter with a depth equal to the number of channels in the three-dimensional stack at the input, and scans these filters across the entire volume of the three-dimensional stack, thus not only in two spatial dimensions but also in the temporal dimension, i.e., in 3D (hence the name 3D convolution). Thus, each filter captures a three-dimensional stack, i.e., a four-dimensional feature map. In a regular convolutional layer, using a large number of filters certainly allows for an increase in the semantic depth (number of channels) at the output, but the output will always be a three-dimensional feature map.

[0094] Understandably, 3D convolutional layers are still quite heavy and require significant computational power; however, as explained, a very simple architecture (much simpler than known CNNs such as VGG616) is sufficient. Figure 5 The architecture of a current 3D CNN embodiment is shown.

[0095] Conventionally, this architecture advantageously comprises a series of "convolutional" blocks, consisting of 3D convolutional layers to increase the depth of the feature map, activation layers (e.g., ReLU functions), and 3D pooling layers that allow for a reduction in the size of the feature map (typically by a factor of 2). It is noteworthy that two convolutions are sufficient, making the height ideal; current 3D CNNs only include two convolutional blocks.

[0096] "3D pooling layer" is understood to mean, in terms of 3D convolution, a layer that can work on a four-dimensional feature map and has one or more channels that are already a three-dimensional stack. In other words, the size reduction is in all dimensions of the three-dimensional stack (i.e., the first three dimensions of the four-dimensional feature map).

[0097] In the remainder of this description, a clear distinction will be made between the number of “dimensions” of the feature maps in the geometric direction, that is, the number of independent directions in which these maps extend (e.g., a vector is a 1-dimensional object, an image is a 2-dimensional object, and the current feature map is a 4-dimensional object), and the number of “variables” of these feature maps, in other words, the number of dimensions in each dimension, i.e., the number of independent degrees of freedom (which actually corresponds to the concept of dimension in vector space; more specifically, a set of feature maps with a given number of variables constitutes a vector space with a dimension equal to that number of variables).

[0098] Therefore, in Figure 5 In the example, as explained, the 3D CNN begins with 6 layers distributed across 2 blocks. The first block takes a 3D stack with a single channel as input (thus forming an object of size 125×125×5×1 when advantageously downsampled as described above) and includes a sequence of convolutions + ReLU (first 3D convolutional layers and activation layers with ReLU functions) increasing the depth to 30, followed by max pooling layers (possibly using global average pooling) with a 62×62×2×30 feature map as output (as explained, the 3D pooling layer operates in three dimensions, not just two spatial dimensions; thus involving division by 2, including the temporal dimension).

[0099] In the example shown, the first 3D convolutional layer uses 30 filters with dimensions of 3×3×3×1, and therefore requires ((3*3*3*1)+1)*30=570 parameters.

[0100] The second block has the exact same architecture as the first block, and generates a 62×62×2×60 (double depth) feature map as the output from the new convolution + ReLU set (a second 3D convolutional layer and an activation layer with ReLU functionality), and generates a 12×12×1×60 feature map as the output from the max pooling layer (it should be noted that in this case, the spatial size is reduced by a factor of five, but the temporal size is always reduced by a factor of two).

[0101] In this case, the second 3D convolutional layer uses 60 filters with dimensions of 3×3×3×30, and therefore requires ((3*3*3*30)+1)*60=32460 parameters.

[0102] At the output of the last convolutional block (the second in this case), the 3D CNN advantageously includes a "flattened" layer that transforms the "final" feature map (containing the "deepest" information) at the block's output into a vector (a one-dimensional object). Thus, for example, a 12×12×1×60 feature map is transformed into a vector of size 12*12*1*60 = 8,640. It will be understood that there are no restrictions on the size of any map / filter at any level, and the sizes mentioned above are merely examples.

[0103] Finally, in the conventional way, there are one or more fully connected layers (FC, or "dense" layers) at completion, such as Figure 5 (as shown) and an optional final activation layer, such as softmax. In the example shown, the first FC transforms a vector of size 8640 into a smaller vector of size 100 (which requires (8640+1)*100=864100 parameters), while the second FC transforms a vector of size 8640 into a final vector of size 2 (which requires (100+1)*2=202 parameters).

[0104] Preferably, the 3D CNN consists of (i.e., exactly includes) a sequence of convolutional blocks, followed by flattened layers, and finally one or more fully connected layers.

[0105] The final part of the 3D CNN returns the expected result, in this case, the category of the input image sequence (a vector of size 2 corresponds to the binary result).

[0106] Therefore, it can be seen that the total number of parameters is less than 900,000, which is quite low for a CNN (typically tens of millions of parameters), especially considering that the input data is already a large sequence of images. Therefore, current 3D CNNs can be used by multiple clients², including those with moderate computational resources.

[0107] Preferably, the method may include step (a0): training the parameters of the 3DCNN from a training library by the data processing device 3 of server 1. In practice, this step is typically performed upstream, particularly by a remote server 1. As explained, the training library may include a certain amount of training data, particularly image sequences associated with their categories (e.g., "split" or "non-split" for binary classification).

[0108] Training a 3D CNN can be done in a conventional manner. The learning cost function can be composed of appendices to the regular "cross-entropy" data, minimized via a gradient descent algorithm.

[0109] In all embodiments, if applicable, the learned parameters of the CNN can be stored on the data storage device 21 of the client 2 for use in classification. It should be noted that the same CNN can be embedded on multiple clients 2, requiring only one learning phase.

[0110] Computer program products

[0111] According to the second and third aspects, the present invention relates to a computer program product comprising code instructions for executing (particularly on data processing apparatus 3, 20 of server 1 and / or client 2) a method for classifying at least one input image representing target particles 11a-11f in sample 12, and a computer device (memory 4, 21 of server 1 and / or client 2) readable storage means thereon on which the computer program product is stored.

Claims

1. A method for classifying a sequence of input images representing target particles (11a-11f) in a sample (12) over time, the method being characterized in that it includes the following steps performed by a data processing device (20) of a client (2): (b) The input images in the sequence are concatenated in the form of a three-dimensional stack; (c) The three-dimensional stack is directly classified using a convolutional neural network (CNN), which consists of a series of convolutional blocks, each of which comprises the following: A four-dimensional filter is applied to the four-dimensional input feature map to generate a four-dimensional output feature map. This is achieved through 3D convolutional layers, activation layers, and 3D pooling layers, followed by a flattening layer, and finally one or more fully connected layers. The series of convolutional blocks includes at least two convolutional blocks, namely a first block and a second block. The first block has 30 filters with a 3×3×3×1 dimension in its 3D convolutional layer, and the second block has 60 filters with a 3×3×3×30 dimension in its 3D convolutional layer.

2. The method according to claim 1, wherein, The particles (11a-11f) are displayed in a uniform manner in each input image, specifically centered and aligned in a predetermined direction.

3. The method of claim 2, comprising step (a): extracting an overall image of the sample from each input image to display the target particles (11a-11f) in the uniform manner.

4. The method according to claim 3, wherein, Step (a) includes: for each input image, segmenting the overall image to detect the target particles (11a-11f) in the sample (12), and then re-cropping the input image according to the detected target particles (11a-11f).

5. The method according to any one of claims 3 and 4, wherein, Step (a) includes obtaining the overall image from the intensity image of the sample (12) acquired by the observation device (10).

6. The method according to any one of claims 1 to 4, wherein, The three-dimensional stack has two spatial dimensions and one temporal dimension, and the filter and feature map have the spatial dimension and the temporal dimension as the three first dimensions, and semantic depth as the fourth dimension.

7. The method according to claim 6, wherein, The depth of the filters in the 3D convolutional layer is equal to the depth of the input feature map, and the depth of the output feature map is equal to the number of filters in the 3D convolutional layer.

8. The method according to any one of claims 1 to 4, wherein, The CNN consists of only two convolutional blocks.

9. The method according to any one of claims 1 to 4, comprising the step (a0): training the parameters of the CNN from a training library of classified sequences of images of particles (11a-11f) in the sample (12) by a data processing device (20) of the server (1).

10. A system for classifying a sequence of input images representing target particles (11a-11f) in a sample (12) over time, the system comprising at least one client (2) including a data processing device (20), characterized in that, The data processing device (20) is configured to implement: - The input images in the sequence are concatenated in the form of a three-dimensional stack; - The 3D stack is directly classified using a Convolutional Neural Network (CNN), which consists of a series of convolutional blocks comprising: 3D convolutional layers that apply four-dimensional filters to a four-dimensional input feature map to generate a four-dimensional output feature map, activation layers and 3D pooling layers, followed by flattening layers, and finally one or more fully connected layers. The series of convolutional blocks includes at least two convolutional blocks, namely a first block and a second block. The first block has 30 filters with a 3×3×3×1 dimension in its 3D convolutional layer, and the second block has 60 filters with a 3×3×3×30 dimension in its 3D convolutional layer.

11. The system of claim 10, further comprising: Device (10) for observing the target particles (11a-11f) in the sample (12).

12. A computer program product comprising code instructions for performing, when the program is executed on a computer, a method for classifying an input image sequence representing target particles (11a-11f) in a sample (12) over time, according to any one of claims 1 to 9.

13. A storage device readable by a computer device, wherein a computer program product thereon comprises code instructions for performing a method according to any one of claims 1 to 9 for classifying an input image sequence representing target particles (11a-11f) in a sample (12) over time.

Images