Method for segmenting a digital image, electronic device, and application of the method

Abstract

The invention relates to a method for detecting objects (13) in an image (4), the method comprising scanning the image with a sliding window (7) so as to successively define a plurality of working portions (8, 12), an object detection algorithm being applied to each defined working portion such that, in the event of an object being detected in the working portion, a binary mask (9) representative of a contour of the detected object is generated; this method is implemented by a convolutional neural network pretrained for object detection and comprises, following the generation of each binary mask and prior to defining the next working portion, converting the binary mask into a sequence of coordinates representative of the contour of the detected object of interest. The invention further relates to a device suitable for implementing the method and to the application of the method to images representative of RAMAN spectroscopy.

Description

Method for segmenting a digital image, electronic device and application of the method. technical field

[0001] The present invention relates to the technical field of image processing, and more specifically to the technical field of digital image segmentation, for example in the context of particle detection in representative images of Raman spectroscopies.

[0002] The invention relates more particularly to a computer method for segmenting a digital image, an electronic device configured to implement this method, and an application of the method to a representative image of a Raman spectroscopy. Technological background

[0003] In image processing, segmentation is a method for grouping certain pixels in an image into sets of pixels according to predetermined criteria (intensity, value, etc.). A set of pixels, for example, represents an object, which is then distinguished from the rest of the image, thus allowing for specific subsequent processing. For example, segmentation is used to count objects in an image (people in a scene, particles of a specific type in a sample, celestial bodies in the sky, etc.).

[0004] To segment digital images, it is common practice to implement convolutional neural networks (CNNs). A segmentation algorithm using a classic convolutional neural network provides, after image processing, a stack of binary masks, each representing an object of interest. For example, a mask representing an object of interest is a matrix the size of the image, where the pixels corresponding to the object of interest have a first value and the pixels corresponding to the rest of the image have a second value.

[0005] That being said, the larger the image and / or the greater the number of objects of interest – and therefore the number of masks – the more the segmentation algorithm is implemented. Any post-processing operations required on the segmented image will demand significant computing resources. Therefore, the resolution and the number of detectable objects are highly dependent on the capabilities of the computer performing the segmentation process.

[0006] In certain applications, particularly the segmentation of representative Raman spectroscopy images, the image resolution and the number of objects to be detected make it impossible to perform a segmentation process on reasonably configured computers, or at least within reasonable timeframes.

[0007] For example, the inventors observed that processing a digital image with a resolution of 14000 pixels by 14000 pixels containing 1000 objects to be detected requires a RAM capacity of at least 182 gigabytes and an execution time on the order of ten minutes.

[0008] Therefore, there is a need for a digital image segmentation process that is economical in terms of computing resources so as to allow its implementation by less powerful computers. Summary of the invention

[0009] The present invention provides a solution to the aforementioned problem by proposing a digital image segmentation algorithm whose execution by a computer processing unit is economical in terms of computing resources.

[0010] In one aspect, a method for detecting objects of interest in a digital image is proposed. This method involves scanning the digital image with a sliding window to successively delimit a plurality of working areas of the image. An object-of-interest detection algorithm is applied to each delimited working area so that, if an object of interest is detected within the working area, a binary mask representing the contour of the detected object of interest is generated. This method is implemented by a convolutional neural network pre-trained in object-of-interest detection and includes, successively after the generation of each binary mask and prior to the delimitation of the next working area, a conversion of the binary mask into a sequence of coordinates representative of the contour of the detected object of interest.

[0011] The use of a convolutional neural network and the conversion of binary masks into a sequence of coordinates as they are generated, rather than at the very end of the process (which would require storing all the masks before conversion), allows for significant savings in memory and computing power. As an example, the inventors observed that applying the method according to the invention to a 14,000 x 14,000 pixel digital image containing 1,000 objects, by defining a sliding window of 2,500 x 2,500 pixels, requires only 0.076 gigabytes of memory and an execution time of approximately 45 seconds. The method according to the invention can therefore be implemented by standard computers available to the general public.

[0012] According to one implementation method, the process includes a post-processing step of a set of the obtained coordinate sequences which involves the implementation of an algorithmic geometry algorithm.

[0013] According to one implementation method, the algorithmic geometry algorithm is of the Intersection on Union type.

[0014] According to one implementation method, the post-processing step includes detecting sequences of coordinates representative of the same contour of the same object of interest and removing duplicate sequences so that each contour of each detected object of interest is represented by a single sequence of coordinates.

[0015] According to one implementation method, the post-processing step involves detecting coordinate sequences, each representing at least a partially distinct portion of a contour of the same detected object of interest, and merging these coordinate sequences to obtain a set of coordinates representative of the contours of the object of interest.

[0016] According to one implementation method, the image has a resolution greater than or equal to 2048 pixels by 2048 pixels.

[0017] According to one implementation method, the image contains a number of objects to be detected greater than or equal to 1000.

[0018] According to another aspect, an electronic device is proposed comprising a computer memory adapted to receive a digital image and a processor coupled to the computer memory and configured to implement the process according to the invention.

[0019] According to another aspect, it is proposed to apply the method according to the invention to the detection of particles in a representative image of a Raman spectroscopy.

[0020] Of course, the different features, variants and embodiments of the invention can be combined with each other in various ways as long as they are not incompatible or mutually exclusive. Brief description of the figures

[0021] Furthermore, various other features of the invention become apparent from the attached description made with reference to the drawings which illustrate non-limiting embodiments of the invention and where:

[0022] [Fig. 1] is a schematic illustration of a computer device according to one embodiment of the invention,

[0023] [Fig. 2] is a schematic illustration of the implementation of a process according to an embodiment of the invention by the electronic device of figure 1.

[0024] It should be noted that in these figures the structural and / or functional elements common to the different variants may have the same references.

[0025] An electronic device, here a computer, according to a first embodiment of the invention, as schematically represented in Figure 1 and designated as a whole by the numerical reference 1, is adapted to the implementation of the process according to the invention.

[0026] It includes a central processing unit 2 (CPU), also called a processor, configured to execute computer program code instructions. The electronic device 1 also includes a read-only memory 3 (ROM), suitable for storing data and computer program code instructions. For example, here, the read-only memory 2 contains data representing a digital image 4 and program code. 5 configured to, when executed by the central computing unit 2, implement an image segmentation method according to an implementation mode of the invention.

[0027] The computer device 1 also includes a RAM type random access memory 6 ("Random Access Memory", according to the usual Anglo-Saxon acronym), configured to receive from read memory 2 the instructions of the program code 5 which will be executed by the processor as well as the data representing the digital image 4 which will be used by the program code 5. For example, prior to the execution of the program code 5, it is transferred from read memory 3 to the RAM with the data representing the digital image 4.

[0028] In the remainder of the description and for the sake of clarity, no distinction will be made between the terms "digital image" and "data representing the digital image".

[0029] Digital image 4 is a high-resolution digital image, for example, a 14,000 x 14,000 pixel square matrix. It represents a scene containing several objects of interest, in this case, a liquid sample containing a plurality of particles of interest, such as plastic microparticles, and a plurality of other particles. For example, digital image 4 is obtained by Raman spectroscopy. In this example, the scene contains 1,000 objects of interest (1,000 plastic microparticles).

[0030] The method according to the invention, corresponding here to the execution of the program code 5 by the central processing unit 2, implements a segmentation of the digital image 4 in order to detect particles of interest, that is, to distinguish them from the rest of the sample. From a digital point of view, this involves distinguishing the pixels and groups of pixels representative of objects of interest from the pixels or groups of pixels not representative of objects of interest. Such an operation makes it possible, in particular, to analyze the sample, for example, to count, measure, or sort the particles of interest.

[0031] A segmented image, that is, the image obtained after the process has been implemented (provided by program code 5 after its execution), contains data representative of the scene (here, the sample), which corresponds, for example, to the data of the initial digital image 4, to which specific data are associated relating to particles of interest. Reading the segmented image with appropriate program code will, for example, allow the image to be displayed on a computer screen and one or more particles of interest to be highlighted, selected, their characteristics displayed, etc.

[0032] The first step of the process, according to the implementation method described here, involves applying processing to the digital image 4, in this case segmentation processing, or object-of-interest detection processing. Due to the high resolution of the digital image 4, this processing is not applied all at once to the entire digital image 4, but successively to several portions of the digital image 4, referred to hereafter as "working portions," by applying a sliding-window algorithm.

[0033] For example, a fixed-size window is defined, such as a square or rectangle, here measuring 2500 pixels by 2500 pixels, corresponding to the boundaries of a working area to which the processing will be applied. Window 7 is then moved (or dragged) across the digital image 4 to successively define several other working areas to which the processing is applied successively, until the entire digital image 4 has been scanned, or traversed, by window 2 and the processing has been applied to all pixels of the digital image 4.

[0034] Figure 2 illustrates the application of this treatment to a first work portion 8, or initial portion, defined by the window, here referenced 7. Since the treatment of each work portion is identical, for the sake of brevity, only the treatment of the first work portion 8 will be described.

[0035] This first working portion 8 is, for example, located in the upper left corner of digital image 4 (window 7 is initially located in the upper left corner of digital image 1). It contains here a particle of interest 13.

[0036] Here, the processing applied to the first work portion 8 involves implementing a segmentation algorithm using a convolutional neural network (CNN). For example, the processing uses a pre-trained segmentation model to detect the objects of interest and to generate a binary mask 9 representing the detected object of interest, here the particle of interest 13. The binary mask 9 is a matrix that represents the first work portion 8, which has the same dimensions as The first working portion 8, in which the pixels corresponding to the particle of interest 13 have a first value. The other pixels have a second value. At the date of filing of this application, such an algorithm is being developed by the American company Meta® under the name Detectron2®.

[0037] Given the high resolution of the digital image 4 and the potentially large number of particles to be detected, the preservation of all the binary masks generated during the processing of each portion of work would require a very high RAM capacity, notably too large for the process to be implemented by a conventional computer.

[0038] To avoid this problem, the invention proposes an alternative solution to mask preservation. According to this alternative solution, the processing applied to each work package, here applied to the first work package in Figure 2, further includes a conversion of the binary mask 9 into a sequence of n coordinates 10 (that is, into an ordered set of n coordinates). This sequence of coordinates 10 corresponds to the vertices of a polygon representing the contours of the object of interest 13. A coordinate has a first value Xi indicating the row of pixels of the binary mask 9 on which the vertex is located and a second value yi indicating the column of pixels of the binary mask 9 on which the vertex is located. At the end of the conversion of the binary mask 9 into a coordinate sequence, only the coordinate sequence is preserved. The binary mask 9 is, for example, removed from RAM 6.The number n of coordinates in the coordinate sequence depends on the desired precision for the segmentation. Thus, the higher the number n of coordinates, the more closely the polygon will resemble the contours of the object of interest.

[0039] The coordinate sequence is then converted, or translated, to obtain a translated coordinate sequence 11 representing not the position of the vertices in the mask (or in the working portion) but the position of the vertices in the digital image 2. In particular, each coordinate is translated here by a value (x W indow, ywindow) corresponding to the position of window 7 in the image. In the particular case of the first working portion 8, the translation values ​​x W indow and y W indow are null.

[0040] After processing the first working area 8, the window is moved horizontally to the right by a predetermined number of pixels to define the next working area, in this case a second working area 12. The processing described previously in relation to the first working area 8 will then be applied to this second working area. For example, the predetermined number of pixels is less than the width of the window 7, so that the first working area 8 and the second working area 12 overlap. During this process, the window 7 is moved several times to the right by the predetermined number of pixels until it reaches the opposite edge of the digital image 4, in this case the right edge. With each movement of the window 7, that is, with each new definition of a working area, the processing described previously is applied to that working area.

[0041] This horizontal movement of the window 7 from the left edge of the digital image 4 to the right edge is then repeated for different vertical positions of the window 7, each time separated by a second predetermined number of pixels downwards from the previous vertical position, until the window 7 reaches the lower right corner of the digital image 4, i.e. until the processing has been applied over the entire surface of the digital image 4. Preferably, the second predetermined number is less than the height of the window 7.

[0042] Given the overlap of the different areas of interest (related to the horizontal and vertical offset of window 7, which is less than the width and height of window 7 in pixels), it is possible that some objects of interest, or parts of objects of interest, may appear in two different working areas, particularly in the common part of the two working areas. In such a case, there will be several sets of coordinates corresponding to the same object of interest.

[0043] To remedy this problem, the process according to this embodiment includes, following the application of the processing to the entire digital image 4, a post-processing step which involves applying an algorithmic geometry algorithm to the coordinate sequences. Such an algorithm makes it possible to detect and merge coordinate sequences corresponding to partially overlapping polygons or duplicate coordinate sequences by putting them into work of intersection over union (loU) algorithms or doubly connected edge list (DCEL) algorithms. Such algorithms are notably described in the book "Computational Geometry", Mark de Berg, Otfried Cheong, Marc van Kreveld, Mark Overmars, Springer, 2008, ISBN: 978-3-540-77973-5.

[0044] Following the application of the algorithmic geometry, a segmented digital image is obtained, comprising the initial digital image 4 to which are associated all the coordinate sets determined during the segmentation process, each representing an object of interest in the digital image 4. This segmented digital image is then copied from RAM into read-only memory 4. It can then be read by suitable software.

[0045] The invention is not limited to the implementations and embodiments described above in connection with Figures 1 and 2. For example, the application of the method to a digital image representing a sample and obtained by Raman spectroscopy has been described previously. That being said, the invention is compatible with any digital image containing objects of interest, for example, images obtained by spectroscopy or photography. The images may represent a sample containing particles or any other scene, for example, a representative image of the sky in the context of astronomical studies, a representative image of a crowd, a road traffic area, etc.In each case, the convolutional neural network implemented in the segmentation processing and stored in read-only memory in the program code instructions will have been pre-trained specifically to detect the corresponding object of interest (stars, people, cars, particles...).

[0046] Furthermore, the method according to the invention is particularly suited to very high-resolution images. That being said, it is compatible with images of all dimensions, square or rectangular, preferably greater than or equal to 2048 pixels by 2048 pixels.

[0047] Although the method according to the invention advantageously allows processing images containing a high number of objects of interest, for example more than 1000, it is also suitable for processing images containing a lower number of objects of interest, for example a number of objects of interest less than 1000.

[0048] Various other modifications may be made to the invention within the scope of the attached claims.

Claims

Demands 1. A method for detecting objects of interest in a digital image (4), comprising scanning the digital image (4) with a sliding window (7) so as to successively delimit a plurality of working portions (8, 12) of the digital image (4), an object-of-interest detection algorithm (13) being applied to each delimited working portion (8, 12) so as to generate, upon detection of an object of interest (13) in the working portion, a binary mask (9) representing a contour of the detected object of interest, the method being characterized in that it is implemented by a convolutional neural network pre-trained in object-of-interest detection (13) and that it comprises, successively after the generation of each binary mask (9) and prior to the delimitation of the next working portion (12), a conversion of the binary mask (8) into a sequence of coordinates (11) representing the contour of the object of interest detected.

2. A method according to claim 1, comprising a post-processing step of a set of the coordinate sequences (11) obtained, which includes the implementation of an algorithmic geometry algorithm.

3. A method according to claim 2, wherein the algorithmic geometry algorithm is of the Intersection on Union type.

4. A method according to claim 2 or 3, wherein the post-processing step comprises detecting coordinate sequences representative of the same contour of the same object of interest and removing duplicate sequences so that each contour of each detected object is represented by a single coordinate sequence.

5. A method according to any one of claims 2 to 4, wherein the post-processing step comprises the detection of coordinate sequences, each representing at least a partially distinct portion of a contour of the same detected object of interest, and a fusion of these coordinate sequences so as to obtain a set of coordinates representative of the contours of the object of interest.

6. A method according to any one of claims 1 to 5, wherein the image has a resolution greater than or equal to 2048 by 2048.

7. A method according to any one of claims 1 to 6, wherein the image comprises a number of objects to be detected greater than or equal to 1000.

8. An electronic device comprising a computer memory (6) adapted to receive a digital image and a processor (2) coupled to the computer memory (6) and configured to implement the method according to any one of claims 1 to 7.

9. Application of the method according to any one of claims 1 to 7 to the detection of particles in a representative image of a RAMAN spectroscopy.

Images