DEVICE AND METHOD FOR DETECTING AN AREA OF INTEREST IN AN IMAGE

Description

Technical Field

[0001] The present invention relates to a method and system for determining a region of interest in an image and is of interest in many fields. Previous technique

[0002] Several methods exist for detecting and isolating regions of interest (ROIs) in images. A region of interest (ROI) is typically a part of an image with notable properties. This could include an object present in the image, such as an object of interest to a user, an object likely to be of interest to a user, or an object that could be the subject of subsequent automated processing (such as tracking). For example, in autonomous driving systems, it can be useful to detect vehicles, road signs, or people along the route. In logistics systems, it is important to be able to locate objects in warehouses, and these objects can be of various types.There are, in particular, interactive techniques in which a user selects the region of interest themselves, with varying degrees of accuracy or precision. Several methods also exist, based on image processing technologies. With the development of artificial intelligence, and more specifically deep learning technologies, these region of interest detection techniques have improved. However, sometimes the identified region of interest does not perfectly correspond to an object and does not define its exact boundaries. Various techniques have been developed to refine the detection of a region of interest that has been roughly identified by a user or by an initial, at least partially automated, image processing method.For example, techniques exist that use depth maps and model the background with Gaussian mixtures to identify the foreground of the region of interest. These approaches use advanced data structures (mixtures of Gaussians and graphs) and are particularly computationally intensive. Therefore, they can only be implemented on certain computing architectures with very high capacity. Consequently, there is a need for simpler solutions to refine the determination of a region of interest.

[0003] Other methods for determining a region of interest from a depth histogram are disclosed in patent publications US2013 / 215107A1, US2015 / 104101A1 and US2019 / 197719A1. Description of the invention

[0004] The invention proposes to overcome at least one drawback of the prior art by proposing a method comprising: obtaining at least one distribution of pixels in a first area of ​​an image as a function of their depth; detecting at least one region of interest in said first area taking into account, for at least one first peak of said pixel distribution, a relative height of said first peak with respect to the highest minimum of said pixel distribution among at least one local minimum between said first peak and another peak of said distribution higher than said first peak.

[0005] For the sake of simplicity, in this application we will qualify the "relative height" of a peak as the relative height of that peak with respect to that highest local minimum.

[0006] Thus, the present invention proposes, in an innovative way, to apply a notion similar to that, recently introduced in the field of geography, of geographical prominence, to the detection of regions of interest in a digital image.

[0007] According to at least one embodiment, the process comprises: an identification of said first area of ​​the image, a obtaining of a depth map of said first area of ​​the image

[0008] According to at least one embodiment, said detection takes into account the depth of said first peak.

[0009] According to at least one embodiment, the detection of at least one region of interest includes a determination of the relative height of the peaks of said distribution, a sorting of said peaks in descending order of their relative height in order to obtain a second distribution of pixels, a selection of said first peak for which the next peak of relative height has a greater depth

[0010] According to at least one embodiment, the detection of at least one region of interest includes: a determination of the relative height of peaks of said distribution, a sorting of said peaks in an increasing order of their depth, a selection of said first peak for which the next depth peak has a lower relative height.

[0011] According to at least one embodiment, said region of interest is determined by selecting the set of pixels between two minima framing said first peak selected in said second pixel distribution.

[0012] According to at least one embodiment, said distribution is a discrete or continuous distribution.

[0013] The characteristics, presented individually in this application in connection with certain embodiments of the process of this application, can be combined with each other according to other embodiments of this process.

[0014] The invention also relates to a computer program comprising instructions for executing a process comprising: obtaining at least one distribution of pixels in a first area of ​​an image as a function of their depth; detecting at least one region of interest in said first area taking into account, for at least one first peak of said pixel distribution, a relative height of said first peak with respect to the highest minimum of said pixel distribution among at least one local minimum between said first peak and another peak of said distribution higher than said first peak, when said program is executed by a computer.

[0015] The invention also relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for executing the steps of a process comprising: obtaining at least one distribution of pixels in a first area of ​​an image as a function of their depth; detecting at least one region of interest in said first area taking into account, for at least one first peak of said pixel distribution, a relative height of said first peak with respect to the highest minimum of said pixel distribution among at least one local minimum between said first peak and another peak of said distribution higher than said first peak

[0016] The present invention also relates to a device comprising one or more processors configured together or separately for performing the steps of the process according to the invention, according to any one of its embodiments. Thus, the present invention relates to a device comprising one or more processors configured together or separately for: obtain at least one distribution of pixels of a first area of ​​an image as a function of their depth; detect at least one region of interest in said first area taking into account, for at least one first peak of said pixel distribution, a relative height of said first peak with respect to the highest minimum of said pixel distribution among at least one local minimum between said first peak and another peak of said distribution higher than said first peak. Brief description of the drawings

[0017] Other features and advantages of the present invention will become apparent from the description given below, with reference to the accompanying drawings which illustrate an example of an embodiment without any limiting character. [ Fig. 1 ] There figure 1 represents a system implementing certain embodiments of the present invention, [ Fig. 2 ] There figure 2 represents a representation of an area encompassing a region of interest in an image, [ Fig. 3 ] There figure 3 represents an example of a depth map of the encompassing area of ​​the figure 2 , [ Fig. 4 ] There figure 4 represents an embodiment of a process according to the invention, [ Fig. 5 ] There figure 5 represents a first embodiment of a discrete pixel distribution according to the invention, [ Fig. 6 ] There figure 6 represents a second embodiment of a continuous pixel distribution according to the invention, [ Fig. 7 ] There figure 7 represents the concept of relative peak height as defined in this application, [ Fig. 8 ] There figure 8 represents a method of sorting peaks in descending order of relative height, [ Fig. 9 ] There figure 9 represents a method of sorting peaks in ascending order of depth. Description methods of implementation

[0018] This disclosure relates to the detection of a region of interest in an image. In this document, a region of interest is defined as any part of an image that is relevant or beneficial to a given application. For example, in an application related to autonomous vehicle driving, a region of interest could be an obstacle such as another vehicle, an object on the road, roadwork, pedestrians, etc.

[0019] According to some embodiments, a region of interest can correspond to an object of interest to a user, and this for different applications.

[0020] According to some embodiments, this disclosure relates to the general field of computer vision and is applicable to various applications using regions of interest.

[0021] There figure 1 represents a system that can implement this disclosure in certain embodiments.

[0022] A scene is captured by means of at least one image capture. A scene can be representative of an indoor or outdoor environment and include one or more objects, animals, people, background, etc. These capture means can take different forms depending on the implementation and may include, in particular: one or more cameras such as stereographic cameras, and / or LIDAR (English acronym for "light detection and ranging"), and / or RADAR (English acronym for "radio detection and ranging"), and / or plenoptic cameras, and / or sensors using ultrasound known as ToF (English acronym for "time of flight") devices.

[0023] These means for capturing at least one image (20) can be associated with processing means 10. The processing means 10 and the capture means 20 can be contained within a single device 100 or belong to different devices coupled together. The processing means include, in particular, means for obtaining a depth map of at least a portion of the scene whose image was captured by the capture means. The processing means may include stereographic vision devices, radar vision devices, lidar vision devices, or ToF devices.

[0024] As illustrated by the figure 1 The processing means may have the hardware architecture of a computer. Thus, the processing means 10 include, in particular, a processor 1, random access memory 2, read-only memory 3 and non-volatile memory 4. They also include communication means 5, in particular for communicating with the capture device 20 and a user interface 30.

[0025] Read-only memory 3 constitutes a storage medium conforming to at least one embodiment of this disclosure, readable by processor 1, and on which is stored a computer program PROG conforming to at least one embodiment of this disclosure, comprising instructions for executing steps in the process of determining an area of ​​interest according to at least one embodiment of this disclosure. The PROG program defines functional modules of the device.

[0026] The user interface 30 allows a user to interact with the capture and processing means. The user interface can take various forms, including one or more screens, touch-sensitive or not, one or more keyboards or styluses, or a tablet, mobile phone, or computer. On the figure 1 The user selected an area encompassing the area of ​​interest.

[0027] There figure 2 Figure 20 represents an example of a scene filmed using image capture means (such as a camera in this example) and in which an area is delimited by the black rectangle. This area is intended to encompass a region of interest (for example, an object) whose precise location within this area is to be determined by this disclosure. This area may also be referred to as the encompassing area hereafter.

[0028] According to some embodiments, the area is delimited by a user using the user interface 30.

[0029] According to some embodiments, the area is automatically delimited by the processing means 10. This automatic delimitation can, for example, be based on methods based on artificial intelligence.

[0030] In some embodiments, the area is delimited both manually and automatically. A user can, for example, indicate to the processing means that they wish to determine the area(s) containing a certain type of object, a car for example, and the processing means are then responsible for detecting the cars in the image and delimiting the areas encompassing the cars present in the image.

[0031] According to some embodiments, several encompassing areas can be identified, automatically by the processing means or manually by a user.

[0032] There figure 3 represents an example of a depth map of the bounding area illustrated in figure 1 According to some embodiments, the depth map can be obtained by encompassing area, for all or several encompassing areas, or for all or part of the captured image.

[0033] A depth map can be a grayscale representation of the image, where the shade of gray indicates the distance from the camera: the darker an area, the closer it is to the camera, or vice versa. A depth map is a visualization of a matrix (a two-dimensional array, for example) that associates each pixel with its distance (in meters or other units of distance) from the camera.

[0034] There figure 4 represents an embodiment of a process according to this disclosure.

[0035] The process according to at least one embodiment of this disclosure may be implemented in a device such as device 100 shown in figure 1 .

[0036] One or more images of a scene are captured by the image capture means identified in figure 1 The present process can make it possible to identify one or more regions of interest in this image or these images.

[0037] Thus, during step E1, an initial area of ​​the image is identified or selected. For example, a user selects one or more areas within an image that roughly encompass the region(s) of interest, meaning that the encompassing area includes at least a portion of the image that is not part of the region of interest. The encompassing area can be an area whose contours roughly follow the contours of the region of interest and / or be geometrically shaped. For example, when the user selects the area manually (using a stylus or mouse, for instance), the encompassing area can be a rectangle around a region of interest (for example, when the region of interest is itself roughly rectangular) (as shown in figure 2 ), a circle around a region of interest (for example, when the region of interest is itself rather round).... This selection can be made using user interface 30 identified on the figure 1 This selection can be made by surrounding the area of ​​interest, so as to encompass the entire area of ​​interest. As mentioned previously, this selection can also be made automatically or semi-automatically by the processing methods 10, for example, using artificial intelligence-based methods. The processing methods can be configured to identify certain objects, for example, animals in a scene, through machine learning, and to delimit an area encompassing the animals for each animal detected.

[0038] The encompassing area thus approximately delimited, around the region of interest, therefore includes a set of pixels some of which are part of the region of interest to be isolated or identified.

[0039] Once the encompassing areas have been obtained, a depth map is obtained for each or some of the encompassing areas, step E2.

[0040] In some embodiments, an overall depth map of the captured image can be constructed and then this map can be segmented into several depth maps, each representing a portion of the captured image, at least one of the depth maps representing a portion of the captured image and including at least one encompassing area of ​​at least one region of interest that can be used in the remainder of the process described below.

[0041] According to some embodiments, when several encompassing areas are identified, at least some of the steps of the process below can be implemented simultaneously in parallel for different identified encompassing areas.

[0042] As mentioned previously, depth maps can be obtained for example by processing means 10 or by devices coupled to these processing means, by stereographic vision devices, radar vision, lidar vision or "time of flight capture" type devices known by the acronym ToF, English acronym for "time of flight".

[0043] In the following steps, the process makes it possible to determine at least one region of interest in at least one encompassing area from at least one depth map associated with that encompassing area.

[0044] To do this, during an E3 step, we obtain a distribution or a representation of the distribution of pixels of the bounding area as a function of their depth, from the depth map obtained for the bounding area.

[0045] The resulting distribution can be of different types. In at least one embodiment, the distribution is a discrete distribution (for example, in the form of a histogram). In at least one embodiment, the distribution is a continuous distribution.

[0046] There figure 5 illustrates, as an example, a representation in the form of a histogram.

[0047] In this histogram, the x-axis represents the distance to the camera (i.e., the depth), and the y-axis represents the number of pixels. The step size or granularity chosen for the x-axis is 5 cm. The height of the histogram bars represents the number of pixels in each interval, with the intervals on the x-axis being [0, 5 cm]; [5, 10 cm]; [10, 15 cm]... Of course, this granularity can vary; for example, it can depend on the maximum depth—the greater the maximum depth, the greater the granularity can be. It can also depend on, or be indexed to, the size of the area.

[0048] There figure 6 illustrates a continuous representation. The x-axis represents the distance to the camera, therefore the depth, and the y-axis represents the number of pixels.

[0049] During step E4, at least one region of interest is determined in at least one of the previously selected encompassing areas. This determination takes into account, for at least one first peak of the pixel distribution, a relative height of the first peak with respect to at least one second peak of the pixel distribution, the second peak being the highest local minimum located in the pixel distribution between the first peak and another peak of the distribution higher than said first peak.

[0050] In some embodiments, the determination takes into account the depth of the first peak. Thus, it may be possible to select, for example, a peak in the foreground or a peak close to this foreground.

[0051] In some embodiments, step E4 may include obtaining (locating or identifying) E41 peaks in the representation. The histogram peaks are identified on the figure 5 , where the index i represents an interval number on the x-axis, a peak Pi is such that its height H i satisfies: H i − 1 < H i > H i + 1

[0052] According to some embodiments, step E4 may include obtaining (or determining) E42 information or data relating to the height, also called relative height for simplification, of the peaks of the representation.

[0053] There figure 7 illustrates the distribution of relative peak heights (as determined in step E42) of the pixel distribution.

[0054] In the illustrated example, the relative height of a peak is the minimum of the difference between the peak's height and the minimum of the histogram between the peak and the peaks immediately above it to the right and left, if any. The relative height of the highest peak corresponds to its height.

[0055] On the figure 7 The relative height of a peak Pi is calculated as follows We determine the highest peak to the right and the highest peak to the left of the current peak Pi. We determine the minima MinGi and MinDi on each of the intervals defined by MinGi: minima between the current peak and the highest peak to the left, and MinDi: minima between the current peak and the highest peak to the right. MinDi has a height HDi and MinGi has a height HGi. The relative height PRO(Pi) of the current peak Pi is given by: PRO P i = Min H i − HD i H i − HG i

[0056] La The definition of the relative height of a peak given above also applies to a continuous distribution, but the identification of the peaks may differ. Therefore, calculating the relative height of a peak, in the case of a continuous representation, may involve identifying the extrema of the distribution.

[0057] The maximum of a mathematical function satisfies f '( x ) = 0 and f" < 0; while the minimum of a functionf check f '( x ) = 0 and f" > 0; f ' And f" representing respectively the first and second derivatives of the function f .

[0058] The minima and maxima can be identified by solving the equation f' ( x ) = 0

[0059] This equation can be solved: analytically by directly solving the equation or numerically using Newton's method.

[0060] Calculating the second derivative then allows us to distinguish between minima and maxima.

[0061] Given that the minima and maxima are known, the relative height of a maximum can then be obtained by applying the previous definition.

[0062] The region of interest can be determined by selecting the foreground or near-foreground peak with the highest relative height, step E43. The region of interest can be determined from a representation of the relative heights of the peaks in the pixel distribution. To do this, it may be possible to represent the distribution of relative heights (and no longer the heights of the peaks in the pixel distribution as in the figure 6 ).

[0063] On the figures 8 et 9 We have schematically represented 6 peaks labeled Pa, Pb, Pc, Pd, Pe, Pf which were previously identified during step E41. If the figure 5 represents a first distribution or a representation of a first distribution, then the figures 8 et 9 may represent a second, respectively a third, distribution or a representation of a second, respectively a third, distribution.

[0064] Each of these peaks is associated with a depth noted respectively for the peaks Pa, Pb, Pc, Pd, Pe, Pf, Prof(Pa), Prof(Pb), Prof(Pc), Prof(Pa), Prof(Pe), Prof(Pf).

[0065] Each of these peaks is associated with a relative height noted respectively for the peaks P a , P b , P c , P d , P e , P f , HR(P a ) , HR(P b ) , HR(P c ) , HR(P d ) , HR(P e ) , HR(P f ).

[0066] According to this example, we have: HR P a > HR P b = HR P c > HR P d > HR P e > HR P f Prof P b < Prof P a < Prof P d < Prof P C < Prof P f < Prof P e

[0067] According to a first embodiment of step E43, the peaks can be sorted, E431, in descending order of their relative height as illustrated in the figure 8 schematically, in order to facilitate the identification of the foreground peak or one close to this foreground having the highest relative height.

[0068] We traverse the peaks in descending order of relative height, and for each peak, we compare the depth of the current peak with the depth of the peak following it. At step E432, we select the first peak for which the following peak has a greater depth. Thus, the selected peak in this example is peak Pb, because peak Pa has a greater relative height but a shallower depth, and the following peaks all have greater depths. Note that peak Pca has the same relative height but a greater depth, so it is not selected. Therefore, peak Pb represents the foreground peak here.

[0069] According to a second embodiment of step E43, the peaks can be sorted, E431', in ascending order of their depth as illustrated in the figure 9schematically, in order to facilitate the identification of the foreground or near-foreground peak having the highest relative height.

[0070] We traverse the peaks in ascending order of depth, comparing the relative height of the current peak with the relative height of the peak following it. At step E432', we select the first peak for which the following peak has a lower relative height. Thus, the selected peak in this example is peak Pa, because peak Pb, which has a greater depth, has a greater relative height. Note that peak Pc has the same relative height but a greater depth, and is therefore not selected. Therefore, peak Pa is not in the foreground but close to it. For example, this method can be used to filter out unwanted elements present in the foreground.

[0071] We can note in this representation in ascending order of depth, that, in some embodiments, if two peaks have an identical associated depth, then at identical depth, the peak with the greatest relative height is selected.

[0072] The area of ​​interest is determined from the representation obtained, step E43.

[0073] According to some embodiments, the area of ​​interest is obtained by selecting, in step E44, a set of pixels around the peak selected in step E43.

[0074] According to some embodiments, this set of pixels consists of the pixels between the two minima framing the peak selected during step E43.

[0075] Some applications of this disclosure may be of interest in the manufacturing industry, particularly in verifying the conformity of parts produced on assembly lines. This disclosure can indeed help to precisely identify a part and thus determine its shape and size, and verify whether it conforms to expectations or specifications, at least partially automatically (without human intervention, for example). Knowledge of the size and location of objects can also help stabilize and refine the movements of robots handling these objects.

[0076] Other applications may relate to the field of logistics. Certain embodiments of this disclosure may enable the tracking and location of goods in warehouses. Knowing the size of objects can help estimate (for example, optimize) the storage space required for their storage and can therefore contribute to warehouse flow management.

[0077] Other applications may relate to autonomous vehicle driving by enabling the determination of obstacles on the road, for example, a region of interest that may represent an obstacle (other vehicles, objects, works, pedestrians...).

[0078] Other applications may involve the construction of maps of a physical environment to enable the navigation of robots, drones and autonomous vehicles in the presence of obstacles.

Claims

1. Computer-implemented method comprising: - obtaining at least one distribution of the pixels of a first area of an image depending on their depth; - detecting at least one region of interest in said first area taking account, for at least one first peak of said pixel distribution, of a relative height of said first peak of said pixel distribution with respect to the highest minimum among at least one local minimum comprised between said first peak and another peak of said distribution which is higher than said first peak.

2. Device comprising one or more processors configured together or separately to: - obtain at least one distribution of the pixels of a first area of an image depending on their depth; - detect at least one region of interest in said first area taking account, for at least one first peak of said pixel distribution, of a relative height of said first peak of said pixel distribution with respect to the highest minimum among at least one local minimum comprised between said first peak and another peak of said distribution which is higher than said first peak.

3. Method according to Claim 1 comprising, or device according to Claim 2, said one or more processors being configured for: - obtaining a depth map of said first area of the image.

4. Method according to one of Claims 1 and 3 comprising, or device according to one of Claims 2 and 3, said one or more processors being configured for, taking into account said depth of said first peak during said detection.

5. Method according to one of Claims 1 or 3 and 4 comprising, or device according to one of Claims 2 to 4, said one or more processors being configured for, detecting at least one region of interest on the basis of - determining the relative height of peaks of said distribution, - sorting said peaks in decreasing order of their relative height in order to obtain a second pixel distribution, - selecting said first peak for which the following peak in relative height has a greater depth.

6. Method according to one of Claims 1 or 3 and 4 comprising, or device according to one of Claims 2 to 4, said one or more processors being configured for, detecting at least one region of interest on the basis of - determining the relative height of peaks of said distribution, - sorting said peaks in increasing order of their depth, - selecting said first peak for which the following peak in depth has a lesser relative height.

7. Method according to Claim 5 comprising, or device according to Claim 5, said one or more processors being configured for, determining said region of interest by selecting all of the pixels comprised between two minima flanking said first selected peak in said second pixel distribution.

8. Method according to one of the preceding claims or device according to one of the preceding claims, where said distribution is a discrete or continuous distribution.

9. Computer program comprising instructions for executing a method comprising: - obtaining at least one distribution of the pixels of a first area of an image depending on their depth; - detecting at least one region of interest in said first area taking account, for at least one first peak of said pixel distribution, of a relative height of said first peak of said pixel distribution with respect to the highest minimum among at least one local minimum comprised between said first peak and another peak of said distribution which is higher than said first peak, when said program is executed by a computer.

10. Computer-readable storage medium on which is stored a computer program comprising instructions for executing the steps of a method comprising: - obtaining at least one distribution of the pixels of a first area of an image depending on their depth; - detecting at least one region of interest in said first area taking account, for at least one first peak of said pixel distribution, of a relative height of said first peak of said pixel distribution with respect to the highest minimum among at least one local minimum comprised between said first peak and another peak of said distribution which is higher than said first peak.