Method and device for segmenting material objects in the space surrounding an autonomous robot

Abstract

Method and Device for Segmenting Material Objects in the Space Surrounding an Autonomous Robot. The present invention relates to a method and associated device for segmenting material objects in the space surrounding an autonomous robot. In particular, the invention addresses a method based on the analysis of frequency spectra of occupancy grids by dense neural networks to segment vehicles in the environment of an autonomous vehicle. Figure for the abstract: Fig. 2

Description

Title of the invention: Method and device for segmenting material objects in the space surrounding an autonomous robot Scope of the invention

[0001] The invention relates to the technical field of object recognition, and more particularly to a method and device for segmenting material objects in a space surrounding an autonomous robot. State of the art

[0002] The recognition of physical objects in the space surrounding an autonomous robot generally consists of interpreting data provided by various sensors located on the autonomous robot in order to produce a model of the vehicle's immediate environment. For vehicle applications, environmental modeling falls within the domain of perception for the autonomous car.

[0003] The automation of driving in the context of autonomous vehicles is a major research challenge of our time. Such a system must combine, on the one hand, very high precision, as an error can lead to serious accidents, and on the other hand, the system must be able to run in real time on onboard computers with limited computing power. Therefore, the recognition of other vehicles present in the environment of an autonomous vehicle is of particular importance.

[0004] To enable precise obstacle localization, an autonomous vehicle is equipped with range sensors such as LiDARs (Light Detection and Ranging), radars, sonars, or stereoscopic cameras. These sensors are unique in their ability to position obstacles in space by measuring their distance and angular position relative to the sensor.

[0005] Each of these sensor types has its own advantages and limitations; stereoscopic cameras, for example, only function during the day, while LiDARs experience reduced acuity in the rain. To ensure maximum reliability under all circumstances, these sensors are generally used in combination on autonomous vehicles. A data fusion and interpretation process, called perception, allows the measurements from the different sensors to be combined.

[0006] The perception task is performed in real time by an onboard computer in the autonomous vehicle. Constraints in terms of cost, power consumption, and the size occupied by the onboard computer severely limit the available computing resources. Thus, all the calculations performed during the perception task They must be sufficiently lightweight to allow their execution in real time on this highly constrained hardware. The resulting representation of this perception process is a digital representation of the scene in which the autonomous vehicle is located, and it is called an environment model.

[0007] Generic environment models exist that allow the fusion of several heterogeneous sensors into a single representation of the environment. The two-dimensional occupancy grid is one such environment model of interest commonly used in the context of autonomous vehicles and, more generally, mobile robotics.

[0008] The two-dimensional occupancy grid offers the following advantages: the possibility of merging measurements from different sensors into a single grid, the possibility of representing occupancy uncertainty from noisy measurements or from the merging of contradictory data, and finally a low computational cost compatible with the significant material constraints of the autonomous vehicle.

[0009] An occupancy grid is a computer model of the environment commonly used in robotics and autonomous driving. It is a matrix representation of the environment seen from above, probabilistically representing the occupation of space by obstacles.

[0010] Fig. 1 is a discretized representation of the environment in top view (or "Bird's Eye View" according to the established anglicism) of an autonomous vehicle, in the form of an occupancy grid where the occupancy probabilities are represented in different shades of grey.

[0011] Each cell in the matrix corresponds to a portion of the space surrounding the autonomous vehicle and contains a probability of occupancy for that portion of space. A cell in an occupancy grid is generally defined in two dimensions (2D), the space being, for example, delimited by a 10 cm x 10 cm square. A cell is then characterized by a value between 0 and 1, representing the probability that the cell is occupied. For example, a probability of 0 (represented by a level of white) means that the cell is definitely empty, while a probability of 1 (represented by a level of black) means that the cell definitely contains an object. All intermediate values ​​(different shades of gray) quantify the uncertainty about the occupancy state of the cell.

[0012] The collection of all the cells, generally millions of cells, forms the occupancy grid.

[0013] The merging of sensor measurements into an occupancy grid is done in two steps. In the first step, an occupancy grid is constructed for each sensor using a suitable measurement noise model. In the second step, all the occupancy grids are merged into a single grid using a Bayesian merging process. When two cells merge, the occupancy probability of the resulting cell is more certain (closer to 0 or 1) if the two merged cells have occupancy probabilities that are consistent with the cell's occupancy status. Otherwise, the resulting occupancy probability is less certain (closer to 0.5). This process synthesizes information from different sensors into a compact representation. Occupancy grids are therefore an important model for representing the real-time environment, which is multimodal and takes into account the noise inherent in the different sensors.

[0014] The calculation of probabilities for an occupancy grid is generally performed using parallel architectures such as GPUs (Graphics Processing Units). It has recently been shown that occupancy grids can be computed and merged using only integer arithmetic on low-power microcontrollers, making them an important environment model for real-time perception in embedded systems.

[0015] However, occupancy grids do not provide the essential information for autonomous navigation, such as the nature and geometry of the various obstacles present in a scene.

[0016] In order to enable the use of occupancy grids in an autonomous navigation context, i.e. in autonomous driving assistance systems, it is necessary to develop algorithms enabling the interpretation of occupancy grids with an on-board computer which is subject to the constraints inherent in the autonomous vehicle, namely real-time execution constraints and constraints of hardware limited in computing resources.

[0017] The present invention proposes a solution meeting these constraints for the real-time recognition of vehicles present in the immediate environment of a vehicle equipped with distance sensors.

[0018] More generally, a solution is proposed for segmenting material objects present in the environment of an autonomous robot. Brief description of the invention

[0019] The invention relates to a method (and an associated device) for segmenting vehicles present in the environment of a vehicle, which is based on the analysis of occupancy grids.

[0020] More generally, the invention relates to a method of segmenting predefined material objects in a space surrounding an autonomous robot.

[0021] The method of the invention presents a new approach which allows a reduction in the dimensionality of occupancy grids via a transformation of the grid of occupancy in the frequency domain.

[0022] In a novel way, the present invention makes it possible to perform a segmentation of occupancy grids using dense neural networks operating in the frequency domain, thus offering quality segmentations with a limited computational cost.

[0023] The present invention also relates to a dense neural network using as input a low-dimensional spectral representation of an occupancy grid, and allowing at the output of the network to perform a segmentation of the vehicles or objects present in the occupancy grid in the same low-dimensional spectral space.

[0024] Advantageously, the present invention requires less computing power and allows for significant gains in terms of computing cost, making it possible to consider implementation in an autonomous vehicle.

[0025] The invention can be applied to any type of robot, for example land robots (autonomous cars), aerial robots (drones), underwater robots, etc.

[0026] To this end, a method for segmenting material objects in a space surrounding an autonomous robot is proposed, the method being implemented by computer in an inference phase.

[0027] In one embodiment, the process comprises steps consisting of: - receive in the form of an occupancy grid data relating to the environment of the autonomous robot, an occupancy grid providing a discretized spatial representation of an environment where each cell corresponding to an area of ​​the environment contains an estimated probability of the presence of an object in that area; - transform the spatial occupancy grid into a spectral representation, the coefficients of the spectrum of the occupancy grid corresponding to frequency components associated with the data of the occupancy grid; - reduce the dimension of the occupancy grid spectrum by applying a frequency mask preserving the coefficients of the spectrum having the highest contribution to the occupancy grid; - execute an artificial intelligence algorithm on the compressed occupancy grid spectrum, said artificial intelligence algorithm implementing a neural network made up of dense layers, trained during a learning phase; - segment the original occupancy grid by applying an inverse transform that projects the spectrum resulting from the artificial intelligence algorithm into the spatial domain; and - identify physical objects in the space surrounding the autonomous robot starting from segmentation.

[0028] In one embodiment, the environmental data reception step includes a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of occupancy probabilities of an occupancy grid.

[0029] In another embodiment, the frequency domain transformation step consists of applying a discrete cosine transform.

[0030] In one embodiment, the spectrum obtained in the transformation step contains the amplitude of each frequency indexed over two dimensions.

[0031] In one embodiment, the dimension reduction step consists of applying a frequency mask that allows specific frequencies to be retained.

[0032] In one embodiment, the frequency mask is a mask learned by learning on the training dataset of the dense neural network, and removing frequencies containing the least information on the occupancy grids.

[0033] In one embodiment, the neural network comprises dense layers of equal input and output sizes.

[0034] In one embodiment, the execution step of an artificial intelligence algorithm includes a step of generating a one-dimensional vector of frequency coefficients for the spectrum of the occupancy grid, providing this vector as input to the dense neural network and obtaining a resulting spectrum.

[0035] In one embodiment, the learning phase of the dense neural network includes at least: a step of segmenting an occupancy grid from ground truth bounding boxes; a step of transforming the segmented occupancy grid into a one-dimensional vector of frequency coefficients; a step of comparing with the one-dimensional vector of frequency coefficients of the segmentation spectrum of the original occupancy grid obtained by inference of the dense neural network; and a step of updating the weights of the neurons of the network.

[0036] Another object of the invention relates to a device for segmenting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the process of the invention.

[0037] The invention also relates to an autonomous vehicle incorporating a driving assistance system, said system comprising a device configured to recognize vehicles in the environment of the autonomous vehicle.

[0038] The invention also addresses a computer program comprising code instructions which, when the program is executed by a computer, cause the computer to implement the material object segmentation process of the invention. Brief description of the figures

[0039] Other features, details and advantages of the invention will become apparent from the description given with reference to the accompanying drawings provided by way of example, which represent:

[0040] Fig. 1 illustrates, in the form of an occupancy grid, a discretized representation of the environment of an autonomous vehicle in top view.

[0041] Figure 2 is a flowchart of the steps in the process of the invention. material object construction according to a mode of realization.

[0042] Figure 3 illustrates the transformation of a spatial domain occupancy grid into a frequency domain occupancy grid spectrum.

[0043] Figure 4 illustrates the use of different masks to compress a spectrum of an occupancy grid in a reduced number-dimensional spectrum according to the method of the invention, for different values ​​of said dimensions.

[0044] The [Fig.5] is a flow representation for a learning mode and for an inference mode, according to the method of the invention. Detailed description of the invention

[0045] Figure 2 is a flowchart of the steps in the object segmentation process. materials of the invention.

[0046] The steps of the process 200 of the invention are implemented by a computer, in inference mode, and allow the segmentation of material objects in the environment of an autonomous robot.

[0047] The method of the invention can be operated as a computer program comprising code instructions which, when the program is executed by a computer, lead the latter to implement the material object segmentation method of the invention.

[0048] In one embodiment, and for ease of description, such an autonomous robot is considered to be an autonomous vehicle, and the method 200 is implemented by a processor of an autonomous driving assistance system which is configured to segment in real time vehicles present in its environment.

[0049] The real-time segmentation of the process of the invention is understood as being real-time with respect to the frequency of data production at the level of the distance sensors equipping the autonomous vehicle from which occupancy grids are produced.

[0050] A first step 202 of the process consists of receiving data relating to the environment of the autonomous vehicle in the form of an occupancy grid. The data are thus representative values ​​of occupancy probabilities for each cell of the occupancy grid.

[0051] In one embodiment, the received environmental data is generated from data from at least one distance sensor on board the autonomous vehicle, and the method may include a preliminary step of transforming the sensor data into representative occupancy probability data for an occupancy grid.

[0052] In an embodiment where the autonomous vehicle carries several distance sensors, the method allows, before the step of transforming the sensor data into data representing the probabilities of occupancy of an occupancy grid, the fusion of the data received from the different sensors. The fusion process can be a Bayesian process and take into account sensor noise and uncertainties.

[0053] After receiving data from the occupancy grid representing the environment of the autonomous vehicle, the process in a subsequent preprocessing step 204 allows a transformation of the occupancy grid from the spatial domain to the frequency domain.

[0054] Advantageously, the proposed preprocessing of occupancy grids makes it possible to replace the costly convolutional layers of convolutional neural networks with inexpensive processing algorithms capable of summarizing the information contained in the occupancy grids into a smaller number of dimensions.

[0055] In one embodiment, a discrete cosine transform (DCT for "Discrete Cosine Transform" according to the established anglicism) is used to perform a projection of the occupancy grids from the spatial domain to the frequency domain, and to obtain an occupancy grid spectrum.

[0056] A person skilled in the art who is familiar with the Fast Fourier Transform (FFT) for image transformation knows that the Discrete Cosine Transform (DCT) produces the same types of spectra as the FFT but with real coefficients.

[0057] Advantageously and innovatively, the inventors propose to use the discrete cosine transform to obtain a frequency representation of an occupancy grid in order to facilitate the recognition of material objects in a scene.

[0058] For example, an occupancy grid composed of cells (xi,j)0 <i,j<255, peut être transformée en un spectre (yk,l )0<k,l<255 de même taille par l’application de la transformée en cosinus discrète rapide, selon l’équation suivante : ' 'Hs H® ' Œ ' ' gjg

[0059] The spectrum obtained contains the amplitude of each frequency indexed according to two di- dimensions.

[0060] Although this dual representation is the same size as the original occupancy grid, its main advantage is that the low frequencies contain most of the information from the original occupancy grid.

[0061] Figure 3 illustrates the transformation of a spatial domain occupancy grid shown on the left into a frequency domain occupancy grid spectrum shown on the right, by applying a discrete cosine transform. The values ​​of the spectrum are represented by their base-10 logarithm. The frequency increases from the upper left corner of the spectrum to the lower right corner.

[0062] The spectrum of an occupancy grid obtained by applying the discrete cosine transform is, unlike occupancy grids, a representation in the frequency domain and not in the spatial domain. This change of domain implies the loss of the locality property of the information, and the processing of a spectrum must be carried out by neural networks that can exploit it directly, which is not possible with convolutional neural networks.

[0063] The method of the invention implements neural networks composed of dense layers.

[0064] In one embodiment, the neural network comprises dense layers of equal input and output sizes.

[0065] Alternatively, the dense neural network architecture can be constructed to operate with dense layers of different input and output sizes.

[0066] However, dense layers have a number of trainable weights and therefore a memory storage cost that increases linearly with the size of their input.

[0067] Considering 'D' as a dense layer with x inputs and y outputs, the number of weights of layer D is: #D = (x + 1 )y.

[0068] Also, in a subsequent step 206, the process makes it possible to reduce the size of the grid occupancy spectra to ensure a low memory footprint of the dense neural networks used, while retaining the maximum amount of information on the original grid occupancy.

[0069] The spectrum of an occupancy grid obtained by discrete cosine transform is of the same dimension as the occupancy grid, i.e. for example 256 x 256 = 65536 dimensions.

[0070] In one embodiment, dimension reduction can be achieved by applying a mask to the original spectrum, setting the coefficients of all masked frequencies to zero.

[0071] Figure 4 illustrates the use of different masks to compress an occupancy grid spectrum according to the method of the invention, for different dimension values.

[0072] Three families of masks (square, triangular, learned) are proposed from top to bottom, which can be further divided into various masks depending on the desired resultant dimension. The number of coefficients retained in the compressed spectrum is indicated in each box.

[0073] In an embodiment shown on line 402, a family of square masks defined by the expression: mk,l = 0 <==> (k > c) V (1 > c), with c obtained by the formula: . .. can be applied.

[0074] In an embodiment shown on line 404, a family of triangular masks is defined by the expression: mk,l = 0 <==> k + 1 - 1 > c, with c obtained by the formula: can be applied.

[0075] In an embodiment shown on line 406, a family of learned masks defined from occupancy grids contained in a training dataset can be applied.

[0076] Initially, the yk,l spectra of each occupancy grid are calculated. The contribution of each coefficient to each occupancy grid is defined as:

[0077] The individual contributions of each frequency are then averaged over all the occupancy grids in the dataset. Finally, for a chosen output dimension 'd', the 'd' frequencies with the highest contribution are retained in the mask, while the coefficients of the other frequencies are set to zero. A normalization procedure, called contribution normalization, is applied to each occupancy grid spectrum before the contribution is calculated. This procedure ensures that each coefficient accurately represents the frequency's contribution to the occupancy grid.

[0078] It can be observed in [Fig. 4] that the number of dimensions of triangular masks is generally slightly less than the number of dimensions of square masks and learned masks. This phenomenon is due to the number of dimensions of triangular masks of the form d = c(c + 1) / 2, which restricts d to certain possible values.

[0079] Returning to [Fig.2], after the step of reducing the dimensionality of the grid occupancy spectrum by applying a frequency mask, the next step 208 of the process allows the execution of an artificial intelligence algorithm based on a dense neural network trained during a supervised learning phase.

[0080] The inputs of the dense neural network are spectra of the occupancy grids, truncated using a frequency filter.

[0081] In a preferred embodiment for the autonomous vehicle application, the filter is a learned mask.

[0082] The objective of recognizing other vehicles present in a scene requires that the output of a neural network encode such information. In the context of the invention, for a dense neural network using inputs in the frequency domain, the network output encodes the information in the frequency domain in the form of a segmentation of the vehicles.

[0083] The use of dense layers in a neural network quickly leads to a significant computational and memory cost, which is incompatible with the computational efficiency objectives inherent in autonomous navigation on embedded platforms. Therefore, the inventors have developed a new architecture with as few dense layers as possible, but still capable of exploiting truncated occupancy grid spectra to identify vehicles present in a scene.

[0084] Each occupancy grid spectrum is flattened into a one-dimensional coefficient vector. The order in which the different coefficients are ordered in the one-dimensional vector is irrelevant, since it is used as input to a dense layer that makes no assumptions about the order of the features received as input, i.e., no assumptions about locality properties in the input data.

[0085] The execution of the AI ​​algorithm produces a resulting spectrum which will be used to reconstruct, using the inverse discrete cosine transform, a grid which can be used, in a subsequent step 210, to segment the cells of the original occupancy grid corresponding to vehicles.

[0086] The process ends with a step 212 enabling the identification of vehicles present on the occupancy grid via the interpretation of the grid obtained by inverse discrete cosine transform into a segmentation of the vehicles.

[0087] Thus advantageously, the present invention combines the use of occupancy grid spectra, i.e. in the frequency domain, with deep learning methods, i.e. dense neural networks, to facilitate the recognition of material objects present in the environment of an autonomous robot.

[0088] In the context of autonomous driving, the inventors' work also focused on training a dense neural network capable of segmenting vehicles present on an occupancy grid from a low-dimensional frequency representation of the original occupancy grid.

[0089] Training a neural network is equivalent to developing a function to identify, extract, and transform relevant patterns in input data for the performance of an expected task.

[0090] The learning phase is specific to the use of neural networks as parametric functions to be optimized using an efficient gradient descent algorithm (known as the backpropagation algorithm) for training datasets.

[0091] The [Fig.5] is a flow representation for a learning mode (or training mode, as indicated in the legend) and for an inference mode, according to the method of the invention.

[0092] During training, information is initially extracted from an occupancy grid 502 (spatial domain) to obtain, using the discrete cosine transform, a spectral representation (frequency domain).

[0093] Then the spectrum is masked (504) using the mask (learned from the training data for the learned mask), and the spectrum is flattened into a one-dimensional vector.

[0094] To enable the outputs of the dense neural network to encode the positions of the vehicles, the choice was made to encode the vehicles in the form of a segmentation of the occupancy grid generated from ground truth bounding boxes.

[0095] Thus, at the same time as the preprocessing of the occupancy grids, ground truth bounding boxes (510) are used to produce a segmentation (512) of the occupancy grid.

[0096] This segmentation is then projected into the frequency domain in the same way as for the occupancy grids and the neural network is trained to reproduce these output occupancy grid segmentation spectra from the input occupancy grid spectra.

[0097] The segmentation spectra are transformed into one-dimensional vectors of frequency coefficients by the same process (508) as for the original occupancy grid.

[0098] The one-dimensional vector of the spectrum of the original occupancy grid is passed into the neural network (506) and the result is compared with the one-dimensional vector of the segmentation spectrum to calculate a gradient and update the weights of the neurons in the network.

[0099] During inference, the resulting spectrum of the neural network (step 208) is decoded into a segmentation of the occupancy grid (step 212).

[0100] The method of the invention can be implemented in a device for segmenting material objects in a space surrounding an autonomous robot, this device comprising appropriate means for implementing all the steps of the method of the invention.

[0101] A method and associated device for segmenting material objects in the space surrounding an autonomous robot has thus been described. The characteristics of the The process involves compression in the frequency domain of occupancy grids, their inference in a dense neural network, and a prediction, still in the spectral domain, of a segmentation of the occupancy grid.

[0102] The characteristics also relate to a method for designing an optimal mask by learning that retains the maximum information of an occupancy grid for a given mask size.

[0103] The benefits of the invention are very low computational costs compared to methods based on convolutional networks, and that these costs are parameterizable thanks to the choice that can be made of the dimension of the spectrum.

[0104] Thus, advantageously, from a remote processing perspective, the proposed method allows only a compressed version of the occupancy grid to be sent to an autonomous vehicle to execute the AI ​​algorithm.

Claims

Demands

1. A method (200) for segmenting material objects in a space surrounding an autonomous robot, the method being implemented by computer in an inference phase, and comprising steps of: receiving (202) in the form of an occupancy grid data relating to the environment of the autonomous robot, an occupancy grid providing a discretized spatial representation of an environment in which each cell corresponding to an area of ​​the environment contains an estimated probability of the presence of an object in that area; transforming (204) the spatial occupancy grid into a spectral representation, the coefficients of the spectrum of the occupancy grid corresponding to frequency components associated with the data of the occupancy grid;reduce (206) the dimension of the occupancy grid spectrum by applying a frequency mask preserving the coefficients of the spectrum having the highest contribution to the occupancy grid; run (208) an artificial intelligence algorithm on the compressed occupancy grid spectrum, said artificial intelligence algorithm implementing a neural network consisting of dense layers, trained during a learning phase; segment (210) the original occupancy grid by applying an inverse transform projecting the spectrum resulting from the artificial intelligence algorithm into the spatial domain; and identify (212) physical objects in the space surrounding the autonomous robot from the segmentation.

2. The method according to claim 1 wherein the environmental data reception step (202) comprises a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of occupancy probabilities of an occupancy grid.

3. The method according to claim 1 or 2 wherein the frequency domain transformation step (204) consists of applying a discrete cosine transform.

4. The method according to any one of claims 1 to 3 wherein the spectrum obtained in transformation step (204) contains the amplitude of each frequency indexed on two dimensions.

5. The method according to any one of claims 1 to 4 wherein the dimension reduction step (206) consists of applying a frequency mask enabling the retention of determined frequencies.

6. The method according to any one of claims 1 to 5 wherein the frequency mask is a mask learned by learning on the training dataset of the dense neural network, and removing frequencies containing the least information on the occupancy grids.

7. The method according to any one of claims 1 to 6 wherein the neural network comprises dense layers of equal input and output sizes.

8. The method according to any one of claims 1 to 7 wherein the step (208) of executing an artificial intelligence algorithm comprises a step of generating a one-dimensional vector of frequency coefficients for the spectrum of the occupancy grid, providing this vector as input to the dense neural network and obtaining a resulting spectrum.

9. The method according to any one of claims 1 to 8 wherein the learning phase of the dense neural network comprises at least: a step of segmenting an occupancy grid from ground truth bounding boxes; a step of transforming the segmented occupancy grid into a one-dimensional vector of frequency coefficients; a step of comparing with the one-dimensional vector of frequency coefficients the segmentation spectrum of the original occupancy grid obtained by inference of the dense neural network; and a step of updating the weights of the neurons of the network.

10. A computer program comprising code instructions for carrying out the steps of the process according to any one of claims 1 to 9, when said program is executed by a processor.

11. A device for segmenting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the method according to any one of claims 1 to Q

12. i.e. y. An autonomous vehicle incorporating a driver assistance system comprising a segmentation device according to claim 11, the device being configured to segment vehicles in the space surrounding the autonomous vehicle.

Images