Method for segmenting a digital image, electronic device and application of the method.

Abstract

A method for detecting objects (13) in an image (4) is proposed, comprising scanning the image with a sliding window (7) to successively delimit a plurality of working areas (8, 12). An object detection algorithm is applied to each delimited working area so that, if an object is detected within the working area, a binary mask (9) representing the contour of the detected object is generated. This method is implemented by a convolutional neural network pre-trained in object detection and includes, successively after the generation of each binary mask and prior to delimiting the next working area, a conversion of the binary mask into a sequence of coordinates representing the contour of the detected object of interest. Furthermore, a device adapted for implementing this method and applying it to images representative of a Raman spectroscopy is proposed.Figure for the summary: Fig. 2.

Description

Title of the invention: Method for segmenting a digital image, electronic device and application of the method. technical field

[0001] The present invention relates to the technical field of image processing, and more specifically to the technical field of digital image segmentation, for example in the context of particle detection in representative images of Raman spectroscopies.

[0002] The invention relates more particularly to a computer method for segmenting a digital image, an electronic device configured to implement this method, and an application of the method to a representative image of a Raman spectroscopy. Technological background

[0003] In the field of image processing, segmentation is a method for grouping certain pixels of the image into sets of pixels according to predetermined criteria (intensity, value, etc.). A set of pixels is, for example, representative of an object, which is then distinguished from the rest of the image, thus allowing for specific subsequent processing. For example, segmentation is used to count objects in an image (people in a scene, particles of a specific type in a sample, celestial bodies in the sky, etc.).

[0004] To segment digital images, it is known to implement convolutional neural networks (CNNs). A segmentation algorithm implementing a classical convolutional neural network provides, after image processing, a stack of binary masks, each mask representing an object of interest. For example, a mask representing an object of interest is a matrix the size of the image, where the pixels corresponding to the object of interest have a first value and the pixels corresponding to the rest of the image have a second value.

[0005] That being said, the larger the image and / or the greater the number of objects of interest—and therefore the number of masks—the more significant computing resources will be required to implement the segmentation algorithm and any post-processing operations to be performed on the segmented image. Thus, the resolution and the number of detectable objects are highly dependent on the capabilities of the computer performing the segmentation process.

[0006] In certain applications, particularly the segmentation of representative Raman spectroscopy images, the resolution of the images and the number of objects to be detected make it impossible to perform a segmentation process by reasonably configured computers, or at least within reasonable timeframes.

[0007] For example, the inventors observed that processing a digital image with a resolution of 14000 pixels by 14000 pixels containing 1000 objects to be detected requires a RAM capacity of at least 182 Gigabytes and an execution time on the order of ten minutes.

[0008] There is therefore a need for a digital image segmentation process that is economical in computing resources so as to allow its implementation by less powerful computers. Summary of the invention

[0009] The present invention provides a solution to the aforementioned problem by proposing a digital image segmentation algorithm whose execution by a computer processing unit is economical in terms of computing resources.

[0010] According to one aspect, a method for detecting objects of interest in a digital image is proposed, comprising scanning the digital image with a sliding window so as to successively delimit a plurality of working portions of the image. An object-of-interest detection algorithm is applied to each delimited working portion so as to generate, if an object of interest is detected in the working portion, a binary mask representing a contour of the detected object of interest is generated. This method is implemented by a convolutional neural network pre-trained in object-of-interest detection and comprises, successively after the generation of each binary mask and prior to the delimitation of the next working portion, a conversion of the binary mask into a sequence of coordinates representing the contour of the detected object of interest.

[0011] The use of a convolutional neural network and the conversion of binary masks into a sequence of coordinates as they are generated, rather than at the very end of the process (which would require storing all the masks before converting them), allows for significant savings in memory and computing power. For example, the inventors observed that applying the method according to the invention to a 14,000 x 14,000 pixel digital image containing 1,000 objects, by defining a sliding window of 2,500 x 2,500 pixels, requires only 0.076 gigabytes of memory and an execution time of approximately 45 seconds. The method according to the invention can therefore be implemented by standard computers available to the general public.

[0012] According to one embodiment, the process includes a post-processing step of a set of the obtained coordinate sequences which includes the implementation of an algorithmic geometry algorithm.

[0013] According to one embodiment, the algorithmic geometry algorithm is of the Intersection on Union type.

[0014] According to one embodiment, the post-processing step includes detecting sequences of coordinates representative of the same contour of the same object of interest and removing duplicate sequences so that each contour of each detected object of interest is represented by a single sequence of coordinates.

[0015] According to one embodiment, the post-processing step includes the detection of coordinate sequences, each representing at least a partially distinct portion of a contour of the same detected object of interest, and a fusion of these coordinate sequences so as to obtain a set of coordinates representative of the contours of the object of interest.

[0016] According to one embodiment, the image has a resolution greater than or equal to 2048 pixels by 2048 pixels.

[0017] According to one embodiment, the image comprises a number of objects to be detected greater than or equal to 1000.

[0018] According to another aspect, an electronic device is proposed comprising a computer memory adapted to receive a digital image and a processor coupled to the computer memory and configured to implement the process according to the invention.

[0019] According to another aspect, it is proposed to apply the method according to the invention to the detection of particles in a representative image of a Raman spectroscopy.

[0020] Of course, the various features, variants, and embodiments of the invention can be combined in various ways, provided they are not incompatible or mutually exclusive. Brief description of the figures

[0021] In addition, various other features of the invention become apparent from the attached description made with reference to the drawings which illustrate non-limiting embodiments of the invention and where:

[0022] [Fig-1] is a schematic illustration of a computer device according to a mode of the realization of the invention,

[0023] [Fig.2] is a schematic illustration of the implementation of a process according to an embodiment of the invention by the electronic device of [Fig.1].

[0024] It should be noted that in these figures the structural and / or functional elements common to the different variants may have the same references.

[0025] An electronic device, here a computer, according to a first embodiment of the invention, as schematically represented in [Fig.1] and designated as a whole by the numerical reference 1, is adapted to the implementation of the process according to the invention.

[0026] It comprises a central processing unit 2 (CPU), also called a processor, configured to execute computer program code instructions. The electronic device 1 further comprises a read-only memory 3 (ROM), adapted to store data and computer program code instructions. For example, here, the read-only memory 2 contains data representing a digital image 4 and program code 5 configured to, when executed by the central processing unit 2, implement an image segmentation method according to an embodiment of the invention.

[0027] The computer device 1 also includes a RAM type random access memory 6 ("Random Access Memory", according to the usual Anglo-Saxon acronym), configured to receive from read memory 2 the instructions of the program code 5 which will be executed by the processor as well as the data representing the digital image 4 which will be used by the program code 5. For example, prior to the execution of the program code 5, it is transferred from read memory 3 to the RAM with the data representing the digital image 4.

[0028] In the remainder of the description and for the sake of clarity, no distinction will be made between the terms "digital image" and "data representing the digital image".

[0029] The digital image 4 is here a high-resolution digital image, for example, a square matrix of 14,000 pixels by 14,000 pixels. It is representative of a scene containing several objects of interest, in this case, a liquid sample containing a plurality of particles of interest, for example, plastic microparticles, and a plurality of other particles. For example, the digital image 4 is obtained by Raman spectroscopy. In this example, the scene contains 1,000 objects of interest (1,000 plastic microparticles).

[0030] The method according to the invention, corresponding here to the execution of the program code 5 by the central processing unit 2, implements a segmentation of the digital image 4 in order to detect the particles of interest, that is to say, to distinguish them from the rest of the sample. From a digital point of view, this involves distinguishing the pixels and groups of pixels representative of objects of interest from the pixels or groups of pixels not representative of objects of interest. Such an operation notably allows to analyze the sample, for example to count, measure or sort the particles of interest.

[0031] A segmented image, that is, as obtained after implementation of the process (provided by the program code 5 after its execution), comprises data representative of the scene (here, the sample), which correspond, for example, to the data of the initial digital image 4, to which are associated specific data relating to the particles of interest. Reading the segmented image by an appropriate program code will, for example, allow the image to be displayed on a computer screen and one or more particles of interest to be highlighted, selected, their characteristics displayed, etc.

[0032] A first step of the process according to the embodiment described herein involves applying a processing step to the digital image 4, here a segmentation processing step, or a processing step for detecting objects of interest. Due to the high resolution of the digital image 4, this processing step is not applied all at once to the entire digital image 4, but successively to several portions of the digital image 4, hereinafter referred to as "working portions", by applying a sliding window algorithm.

[0033] For example, a window of fixed size is defined, for example square or rectangular, here with dimensions of 2500 pixels by 2500 pixels, corresponding here to the boundaries of a working area to which the processing will be applied. The window 7 is then moved (or slid) over the digital image 4 so as to successively define several other working areas to which the processing is successively applied, until the entire digital image 4 has been scanned, or traversed, by the window 2 and the processing has been applied to all the pixels of the digital image 4.

[0034] Fig. 2 illustrates the application of this treatment to a first work portion 8, or initial portion, defined by the window, here referenced 7. Since the treatment of each work portion is identical, for the sake of brevity, only the treatment of the first work portion 8 will be described.

[0035] This first working portion 8 is, for example, located in the upper left corner of the digital image 4 (the window 7 is initially located in the upper left corner of the digital image 1). It includes here a particle of interest 13.

[0036] Here, the processing applied to the first work portion 8 involves the implementation of a segmentation algorithm using a convolutional neural network (CNN). For example, the processing implements a pre-trained segmentation model to detect said objects of interest and to generate a binary mask 9 representing the detected object of interest, here the particle of interest 13. The binary mask 9 is a matrix that is representative of the first work portion 8, which has the same dimensions that the first working portion 8, in which the pixels corresponding to the particle of interest 13 have a first value. The other pixels have a second value. At the date of filing of this application, such an algorithm is being developed by the American company Meta® under the name Detectron2®.

[0037] Given the high resolution of the digital image 4 and the potentially high number of particles to be detected, the storage of all the binary masks generated during the processing of each portion of work would require a very high RAM capacity, in particular too high for the process to be implemented by a conventional computer.

[0038] To avoid this problem, the invention proposes an alternative solution to preserving the masks. According to this alternative solution, the processing applied to each work portion, here applied to the first work portion on [Fig. 2], further includes a conversion of the binary mask 9 into a sequence of n coordinates 10 (i.e., into an ordered set of n coordinates). This sequence of coordinates 10 corresponds to the vertices of a polygon representing the contours of the object of interest 13. A coordinate has a first value x, indicating the row of pixels of the binary mask 9 on which the vertex is located, and a second value y, indicating the column of pixels of the binary mask 9 on which the vertex is located. At the end of the conversion of the binary mask 9 into a sequence of coordinates, only the sequence of coordinates is retained. The binary mask 9 is, for example, removed from RAM 6.The number n of coordinates in the coordinate sequence depends on the desired precision for the segmentation. Thus, the higher the number n of coordinates, the more faithful the polygon will be to the contours of the object of interest.

[0039] The coordinate sequence is then converted, or translated, so as to obtain a translated coordinate sequence 11 representing not the position of the vertices in the mask (or in the working portion) but the position of the vertices in the digital image 2. In particular, each coordinate is translated here by a value (xwindow, ywindow) corresponding to the position of the window 7 in the image. In the particular case of the first working portion 8, the translation values ​​xwindow and ywindow are zero.

[0040] After processing the first work portion 8, the window is moved horizontally to the right by a predetermined number of pixels to define the next work portion, here a second work portion 12, to which the processing described previously in relation to the first work portion 8 will be applied. For example, the predetermined number of pixels is less than the width of the window 7, so that the first work portion 8 and the second work portion The working area 12 overlaps. During the process, the window 7 is thus moved several times to the right by a predetermined number of pixels, until it reaches the opposite edge of the digital image 4, here the right edge. With each movement of the window 7, that is, with each new definition of a working area, the processing described above is applied to the working area.

[0041] This horizontal movement of the window 7 from the left edge of the digital image 4 to the right edge is then repeated for different vertical positions of the window 7, each time separated by a second predetermined number of pixels downwards relative to the previous vertical position, until the window 7 reaches the lower right corner of the digital image 4, i.e. until the processing has been applied over the entire surface of the digital image 4. Preferably, the second predetermined number is less than the height of the window 7.

[0042] Given the overlap of the different areas of interest (related to the horizontal and vertical offset of window 7, which is less in pixels than the width and height of window 7, respectively), it is possible that some objects of interest, or parts of objects of interest, may be found in two different working areas, particularly in the common part of the two working areas. In such a case, there will be several sets of coordinates corresponding to the same object of interest.

[0043] To remedy this problem, the process according to this embodiment includes, following the application of the processing to the entire digital image 4, a post-processing step which involves applying an algorithmic geometry algorithm to the coordinate sequences. Such an algorithm makes it possible to detect and merge the coordinate sequences corresponding to partially overlapping polygons or duplicate coordinate sequences by implementing intersection over union (loU) algorithms or a doubly connected edge list (DCEL) type algorithm. Such algorithms are described in particular in the book "Computational Geometry", Mark de Berg, Otfried Cheong, Marc van Kreveld, Mark Overmars, Springer editions, 2008, ISBN: 978-3-540-77973-5.

[0044] Following the application of the algorithmic geometry algorithms, a segmented digital image is obtained, comprising the initial digital image 4 to which are associated all the sets of coordinates determined during the segmentation process, each of which represents an object of interest in the digital image 4. This segmented digital image is, for example, copied from RAM into read-only memory 4. It can then be read by suitable software.

[0045] The invention is not limited to the embodiments and modalities described above in connection with Figures 1 and 2. For example, the application of the method to a digital image representative of a sample and obtained by Raman spectroscopy has been described above. That being said, the invention is compatible with any digital image containing objects of interest, for example, images obtained by spectroscopy or photography. The images may be representative of a sample containing particles or of any other scene, for example, a representative image of the sky in the context of astronomical studies, a representative image of a crowd, a road traffic area, etc.In each case, the convolutional neural network implemented in the segmentation processing and stored in read-only memory within the program code instructions will have been pre-trained specifically to detect the corresponding object of interest (stars, people, cars, particles...).

[0046] Furthermore, the method according to the invention is particularly suited to very high-resolution images. That being said, it is compatible with images of all dimensions, square or rectangular, preferably greater than or equal to 2048 pixels by 2048 pixels.

[0047] Although the method according to the invention advantageously allows processing images containing a high number of objects of interest, for example more than 1000, it is also suitable for processing images containing a lower number of objects of interest, for example a number of objects of interest less than 1000.

[0048] Various other modifications may be made to the invention within the scope of the annexed claims.

Claims

Demands

1. A method for detecting objects of interest in a digital image (4), comprising scanning the digital image (4) with a sliding window (7) so as to successively delimit a plurality of working portions (8, 12) of the digital image (4), an object-of-interest detection algorithm (13) being applied to each delimited working portion (8, 12) so as to generate, upon detection of an object of interest (13) in the working portion, a binary mask (9) representative of a contour of the detected object of interest, the method being characterized in that it is implemented by a convolutional neural network pre-trained in object-of-interest detection (13) and that it comprises, successively after the generation of each binary mask (9) and prior to the delimitation of the next working portion (12), a conversion of the binary mask (8) into a representative coordinate sequence (11) of the outline of the detected object of interest.

2. A method according to claim 1, comprising a post-processing step of a set of the coordinate sequences (11) obtained which includes the implementation of an algorithmic geometry algorithm.

3. Method according to claim 2, wherein the algorithmic geometry algorithm is of the Intersection on Union type.

4. A method according to claim 2 or 3, wherein the post-processing step comprises detecting coordinate sequences representative of the same contour of the same object of interest and removing duplicate sequences so that each contour of each detected object is represented by a single coordinate sequence.

5. A method according to any one of claims 2 to 4, wherein the post-processing step comprises the detection of coordinate sequences, each representing at least a partially distinct portion of a contour of the same detected object of interest, and a fusion of these coordinate sequences so as to obtain a set of coordinates representative of the contours of the object of interest.

6. A method according to any one of claims 1 to 5, wherein the image has a resolution greater than or equal to 2048 by 2048.

7. A method according to any one of claims 1 to 6, wherein the image comprises a number of objects to be detected greater than or equal to 1000.

8. Electronic device comprising a computer memory (6) adapted to receive a digital image and a processor (2) coupled to the computer memory (6) and configured to implement the method according to any one of claims 1 to 7.

9. Application of the method according to any one of claims 1 to 7 to the detection of particles in a representative image of a RAMAN spectroscopy.

Images