Method and device for segmenting material objects in a space surrounding an autonomous robot

Abstract

Method and device for segmenting material objects in a space surrounding an autonomous robot The present invention relates to a method and an associated device for segmenting material objects in a space surrounding an autonomous robot. In particular, the invention addresses a method based on the analysis of frequency spectra of occupancy grids by dense neural networks to segment vehicles in the environment of an autonomous vehicle. Figure for abstract: Fig.2

Description

Title of the invention: Method and device for segmenting material objects in a space surrounding an autonomous robot Field of invention

[0001] The invention relates to the technical field of object recognition, and more particularly relates to a method and a device for segmenting material objects in a space surrounding an autonomous robot. State of the art

[0002] The recognition of material objects in a space surrounding an autonomous robot generally consists of the interpretation of data provided by different sensors located on the autonomous robot in order to produce a model of the immediate environment of the vehicle. For applications to vehicles, the modeling of the environment falls within the field of perception for the autonomous car.

[0003] Driving automation in the context of autonomous vehicles is a major research challenge of our time. Such a system must combine, on the one hand, very high precision because an error can lead to serious accidents and, on the other hand, the system must be able to run in real time on on-board computers with limited computing power. Also, the recognition of other vehicles present in the environment of an autonomous vehicle is of particular importance.

[0004] To enable precise location of obstacles, an autonomous vehicle is equipped with distance sensors (range sensors) such as 'LiDARs' (acronym for "Light Detection And Ranging"), radars, sonars or even stereoscopic cameras. These sensors have the particularity of their ability to position obstacles in space by measuring their distance and their angular position relative to that of the sensor.

[0005] Each of these types of sensors has its own advantages and limitations; stereoscopic cameras, for example, only work during the day, while LiDARs experience reduced acuity in the rain. To ensure maximum reliability in all circumstances, these sensors are generally used in combination on autonomous vehicles. A data fusion and interpretation process, called perception, allows the measurements from the different sensors to be assembled.

[0006] The perception task is performed in real time by a computer on board the autonomous vehicle. Constraints in terms of cost, power consumption and volume occupied by the on-board computer severely limit the available computing resources. Thus, all of the calculations performed during the perception task must be lightweight enough to allow their execution in real time on this highly constrained hardware. The resulting representation of this perception process is a digital representation of the scene in which the autonomous vehicle is located, and it is called an environment model.

[0007] There are generic environment models that allow the fusion of several heterogeneous sensors into a single representation of the environment. The two-dimensional occupancy grid is one of these environment models of interest commonly used in the context of autonomous vehicles and more generally mobile robotics.

[0008] The two-dimensional occupancy grid offers the advantages of the possibility of merging measurements from different sensors into a single grid, the possibility of representing the occupancy uncertainty resulting from noisy measurements or from the fusion of contradictory data and finally a low computational cost compatible with the significant hardware constraints of the autonomous vehicle.

[0009] An occupancy grid is a computer model of the environment commonly used in robotics and autonomous driving. It is a matrix representation of the environment seen from above, probabilistically representing the occupation of space by obstacles.

[0010] [Fig.l] is a discretized representation of the environment in top view (or “Bird's Eye View” according to the accepted Anglicism) of an autonomous vehicle, in the form of an occupancy grid where the occupancy probabilities are represented in different gray levels.

[0011] Each cell of the matrix corresponds to a portion of the space surrounding the autonomous vehicle and contains a probability of occupation of this portion of space. A cell of an occupancy grid is generally defined in two dimensions (2D), the space being for example delimited by a square of 10 cm x 10 cm. A cell is then characterized by a value between 0 and 1, representing the probability that the cell is occupied. For example, a probability of 0 (represented by a white level) means that the cell is definitely empty, while a probability of 1 (represented by a black level) means that the cell definitely contains an object. All intermediate values ​​(different gray levels) quantify the uncertainty on the occupancy state of the cell.

[0012] The collection of all cells, usually millions of cells, forms the occupancy grid.

[0013] Merging sensor measurements into an occupancy grid is done in two steps. In a first step, an occupancy grid is constructed for each sensor using a suitable measurement noise model. In a second step, all occupancy grids are merged into a single grid using a Bayesian fusion process. When merging two cells, the resulting cell occupancy probability is more certain (closer to 0 or 1) if the two merged cells have an occupancy probability that agrees with respect to the cell's occupancy state. Otherwise, the resulting occupancy probability is less certain (closer to 0.5). This process synthesizes information from different sensors into a compact representation. Occupancy grids are therefore an important model for representing the real-time environment, which is multimodal and takes into account the noise inherent in different sensors.

[0014] Computing occupancy grid probabilities is typically performed using parallel architectures such as GPUs (Graphics Processing Units). It has recently been demonstrated that occupancy grids can be computed and merged using only integer arithmetic on low-power microcontrollers, making them an important environment model for real-time perception in embedded systems.

[0015] However, occupancy grids do not provide information essential to autonomous navigation, such as the nature and geometry of the different obstacles present in a scene.

[0016] In order to enable the use of occupancy grids in an autonomous navigation context, i.e. in autonomous driving assistance systems, it is necessary to develop algorithms enabling the occupancy grids to be interpreted with an on-board computer which is subject to the constraints inherent to the autonomous vehicle, namely real-time execution constraints and constraints of hardware limited in computing resources.

[0017] The present invention proposes a solution meeting these constraints for the real-time recognition of vehicles present in the immediate environment of a vehicle equipped with distance sensors.

[0018] More generally, a solution is proposed for segmenting material objects present in the environment of an autonomous robot. Brief description of the invention

[0019] The invention relates to a method (and an associated device) for segmenting vehicles present in the environment of a vehicle, which is based on the analysis of occupancy grids.

[0020] More generally, the invention relates to a method for segmenting predefined material objects in a space surrounding an autonomous robot.

[0021] The method of the invention presents a new approach which allows a reduction in the dimensionality of the occupancy grids via a transformation of the grid. occupancy in the frequency domain.

[0022] In a novel manner, the present invention makes it possible to perform segmentation of occupancy grids using dense neural networks operating in the frequency domain, thus providing quality segmentations with a limited computational cost.

[0023] The present invention also relates to a dense neural network using as input a low-dimensional spectral representation of an occupancy grid, and allowing at the output of the network to perform a segmentation of the vehicles or objects present in the occupancy grid in the same low-dimensional spectral space.

[0024] Advantageously, the present invention requires less computing power and makes it possible to obtain significant gains in terms of computing cost, which makes it possible to envisage implementation in an autonomous vehicle.

[0025] The invention can be applied to any type of robots, for example terrestrial robots (autonomous cars), aerial robots (drones), underwater robots, etc.

[0026] To this end, a method is proposed for segmenting material objects in a space surrounding an autonomous robot, the method being implemented by computer in an inference phase.

[0027] In one embodiment, the method comprises steps of: - receive in the form of an occupancy grid data relating to the environment of the autonomous robot, an occupancy grid providing a discretized spatial representation of an environment of which each cell corresponding to an area of ​​the environment contains an estimated probability of the presence of an object in this area; - transforming the spatial occupancy grid into a spectral representation, the coefficients of the spectrum of the occupancy grid corresponding to frequency components associated with the data of the occupancy grid; - reduce the dimension of the spectrum of the occupancy grid by applying a frequency mask preserving the coefficients of the spectrum having a contribution to the highest occupancy grid; - executing an artificial intelligence algorithm on the compressed occupancy grid spectrum, said artificial intelligence algorithm implementing a neural network consisting of dense layers, trained during a learning phase; - segment the original occupancy grid by applying an inverse transform projecting the spectrum resulting from the artificial intelligence algorithm into the spatial domain; and - identify material objects in the space surrounding the autonomous robot from segmentation.

[0028] In one embodiment, the step of receiving environmental data comprises a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of probabilities of occupation of an occupation grid.

[0029] In another embodiment, the transformation step in the frequency domain consists of applying a discrete cosine transform.

[0030] In an alternative embodiment, the spectrum obtained in the transformation step contains the amplitude of each frequency indexed on two dimensions.

[0031] In an alternative embodiment, the dimension reduction step consists of applying a frequency mask making it possible to retain determined frequencies.

[0032] In an alternative embodiment, the frequency mask is a mask learned by training on the training data set of the dense neural network, and removing the frequencies containing the least information on the occupancy grids.

[0033] In an alternative embodiment, the neural network comprises dense layers of equal input and output sizes.

[0034] In an alternative embodiment, the step of executing an artificial intelligence algorithm comprises a step of generating a one-dimensional vector of frequency coefficients for the spectrum of the occupancy grid, providing this vector as input to the dense neural network and obtaining a resulting spectrum.

[0035] In an alternative embodiment, the training phase of the dense neural network comprises at least: a step of segmenting an occupancy grid from ground truth bounding boxes; a step of transforming the segmented occupancy grid into a one-dimensional vector of frequency coefficients; a step of comparing with the one-dimensional vector of frequency coefficients of the segmentation spectrum of the original occupancy grid obtained by inference of the dense neural network; and a step of updating the weights of the neurons of the network.

[0036] Another object of the invention relates to a device for segmenting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the method of the invention.

[0037] The invention also relates to an autonomous vehicle incorporating a driving assistance system, said system comprising a device configured to recognize vehicles in the environment of the autonomous vehicle.

[0038] The invention also addresses a computer program comprising code instructions which, when the program is executed by a computer, cause the latter to implement the method of segmenting material objects of the invention. Brief description of the figures

[0039] Other characteristics, details and advantages of the invention will emerge from reading the description given with reference to the appended drawings given by way of example and which represent:

[0040] [Fig.l] illustrates in the form of an occupancy grid, a discretized representation of the environment of an autonomous vehicle in top view.

[0041] [Fig.2] is a flowchart of the steps of the method of the invention of seg material object management according to an embodiment.

[0042] [Fig.3] illustrates the transformation of a spatial domain occupancy grid into a frequency domain occupancy grid spectrum.

[0043] [Fig.4] illustrates the use of different masks to compress a spectrum of an occupancy grid into a spectrum of reduced number of dimensions according to the method of the invention, for different values ​​of said dimensions.

[0044] [Fig.5] is a flow representation for a learning mode and for an inference mode, according to the method of the invention. Detailed description of the invention

[0045] [Fig.2] is a flowchart of the steps in the object segmentation method materials of the invention.

[0046] The steps of the method 200 of the invention are implemented by a computer, in inference mode, and allow the segmentation of material objects in the environment of an autonomous robot.

[0047] The method of the invention can be operated as a computer program comprising code instructions which, when the program is executed by a computer, cause the latter to implement the method of segmenting material objects of the invention.

[0048] In one embodiment, and to facilitate the description, such an autonomous robot is considered to be an autonomous vehicle, and the method 200 is implemented by a processor of an autonomous driving assistance system which is configured to segment in real time vehicles present in its environment.

[0049] The real-time segmentation of the method of the invention is understood to be real-time relative to the frequency of production of data at the level of the distance sensors equipping the autonomous vehicle from which occupancy grids are produced.

[0050] A first step 202 of the method consists of receiving data relating to the environment of the autonomous vehicle in the form of an occupancy grid. The data are thus values ​​representative of occupancy probabilities for each cell of the occupancy grid.

[0051] In one embodiment, the received environmental data is generated from data from at least one distance sensor on board the autonomous vehicle, and the method may comprise a preliminary step consisting of transforming the sensor data into data representative of occupancy probabilities for an occupancy grid.

[0052] In an embodiment where the autonomous vehicle has several distance sensors on board, the method makes it possible, before the step of transforming the sensor data into data representative of probabilities of occupation of an occupancy grid, to merge the data received from the different sensors. The fusion process can be a Bayesian type process and take into account the noise of the sensors and the uncertainties.

[0053] After receiving data from the occupancy grid representative of the environment of the autonomous vehicle, the method in a following preprocessing step 204, allows a transformation of the occupancy grid from the spatial domain to the frequency domain.

[0054] Advantageously, the proposed preprocessing of the occupancy grids makes it possible to replace the expensive convolution layers of the convolutional neural networks with inexpensive processing algorithms capable of summarizing the information contained in the occupancy grids in a smaller number of dimensions.

[0055] In one embodiment, a discrete cosine transform (DCT) is used to carry out a projection of the occupancy grids from the spatial domain to the frequency domain, and obtain an occupancy grid spectrum.

[0056] The person skilled in the art, familiar with the fast Fourier transform (FFT) for image transformation, knows that the discrete cosine transform (DCT) produces the same types of spectrum as the FFT but with real coefficients.

[0057] Advantageously in an innovative manner, the inventors propose to use the discrete cosine transform to obtain a frequency representation of an occupancy grid in order to facilitate the recognition of material objects in a scene.

[0058] For example, an occupancy grid composed of cells (xi,j )0 <i,j<255, peut être transformée en un spectre (yk,l )0<k,l<255 de même taille par l’application de la transformée en cosinus discrète rapide, selon l’équation suivante : ' 'Hs H® ' Œ ' ' gjg

[0059] The spectrum obtained contains the amplitude of each frequency indexed according to two di- measurements.

[0060] Although this dual representation is the same size as the original occupancy grid, its main advantage is that the low frequencies contain most of the information from the original occupancy grid.

[0061] [Fig.3] illustrates the transformation of a spatial domain occupancy grid shown on the left into a frequency domain occupancy grid spectrum shown on the right, by applying a discrete cosine transform. The spectrum values ​​are represented by their base 10 logarithm. The frequency increases from the upper left corner of the spectrum to the lower right corner.

[0062] The spectrum of an occupancy grid obtained by applying the discrete cosine transform is, unlike occupancy grids, a representation in the frequency domain and not in the spatial domain. This change of domain implies the loss of the property of locality of the information, and the processing of a spectrum must be done by neural networks which can exploit it directly, which convolutional neural networks do not allow.

[0063] The method of the invention implements neural networks composed of dense layers.

[0064] In one embodiment, the neural network comprises dense layers of equal input and output sizes.

[0065] Alternatively, the dense neural network architecture can be constructed to operate with dense layers of different input and output sizes.

[0066] However, dense layers have a number of trainable weights and therefore a memory storage cost that increases linearly with the size of their input.

[0067] Considering 'D' a dense layer with x inputs and y outputs, the number of weights of layer D is: #D = (x + 1 )y.

[0068] Also, in a following step 206, the method makes it possible to reduce the size of the occupancy grid spectra to ensure a low memory footprint of the dense neural networks used, while retaining the maximum amount of information on the original occupancy grid.

[0069] The spectrum of an occupancy grid obtained by discrete cosine transform is of the same dimension as the occupancy grid, for example 256 x 256 = 65536 dimensions.

[0070] In one embodiment, the dimension reduction may be performed by applying a mask to the original spectrum, setting the coefficients of all masked frequencies to zero.

[0071] [Fig.4] illustrates the use of different masks to compress an occupancy grid spectrum according to the method of the invention, for different dimension values.

[0072] Three families of masks are proposed from top to bottom (square, triangular, learned) which can be broken down into various masks depending on the desired resulting dimension. The number of coefficients kept on the compressed spectrum is indicated in each box.

[0073] In one embodiment shown on line 402, a family of square masks defined by the expression: mk,l = 0 <==> (k > c) V (1 > c), with c obtained by the formula: . .. can be applied.

[0074] In one embodiment shown on line 404, a family of triangular masks defined by the expression: mk,l = 0 <==> k + 1 - 1 > c , with c obtained by the formula: can be applied.

[0075] In one embodiment shown on line 406, a family of learned masks defined from occupancy grids contained in a training data set may be applied.

[0076] First, the spectra yk,l of each occupancy grid are calculated. The contribution of each coefficient to each occupancy grid is defined as:

[0077] The individual contributions of each frequency are then averaged over all occupancy grids in the dataset. Finally, for a chosen output dimension 'd', the 'd' frequencies with the highest contribution are kept in the mask while the coefficients of the other frequencies are set to zero. A normalization procedure, called contribution normalization, is applied to each occupancy grid spectrum before calculating the contribution. This procedure ensures that each coefficient actually represents the frequency's contribution to the occupancy grid.

[0078] It can be observed in [Fig.4] that the number of dimensions of triangular masks is generally slightly lower than the number of dimensions of square masks and learned masks. This phenomenon is due to the number of dimensions of triangular masks of the form d = c(c + 1 ) / 2 which restricts d to certain possible values.

[0079] Returning to [Fig.2], after the step of reducing the dimensionality of the occupancy grid spectrum by applying a frequency mask, the following step 208 of the method allows the execution of an artificial intelligence algorithm based on a dense neural network trained during a supervised learning phase.

[0080] The inputs of the dense neural network are spectra of the occupancy grids, truncated using a frequency filter.

[0081] In a preferred embodiment for the autonomous vehicle application, the filter is a learned mask.

[0082] The objective of recognizing other vehicles present in a scene means that the output of a neural network must encode such information. In the context of the invention, of a dense neural network using inputs in the frequency domain, the output of the network encodes the information in the frequency domain in the form of a segmentation of the vehicles.

[0083] The use of dense layers in a neural network quickly leads to a significant computational and memory cost that is not compatible with the objectives of lightness of calculations inherent in autonomous navigation on an on-board platform. Also, the inventors have developed a new architecture comprising the fewest possible dense layers, but nevertheless capable of exploiting the truncated spectra of occupancy grids to identify the vehicles present in a scene.

[0084] Each occupancy grid spectrum is flattened into a one-dimensional coefficient vector. The order in which the different coefficients are ordered in the one-dimensional vector is irrelevant, since it is used as input to a dense layer that makes no assumption about the order of the features received as input, i.e. no assumption of locality property in the input data.

[0085] Execution of the AI ​​algorithm produces a resulting spectrum that will be used to reconstruct, using the inverse discrete cosine transform, a grid that can be used, in a subsequent step 210, to segment the cells of the original occupancy grid corresponding to vehicles.

[0086] The method ends with a step 212 allowing the identification of the vehicles present on the occupancy grid via the interpretation of the grid obtained by inverse discrete cosine transform into a segmentation of the vehicles.

[0087] Thus advantageously, the present invention combines the use of occupancy grid spectra, i.e. in the frequency domain, with deep learning methods, i.e. dense neural networks, to facilitate the recognition of material objects present in the environment of an autonomous robot.

[0088] In the context of autonomous driving, the inventors' work also focused on training a dense neural network capable of segmenting the vehicles present on an occupancy grid from a low-dimensional frequency representation of the original occupancy grid.

[0089] Training a neural network is equivalent to developing a function to identify, extract, and transform relevant patterns in input data to perform an expected task.

[0090] The specificity of the learning phase is the use of neural networks as parametric functions to be optimized using an efficient gradient descent algorithm (known as a backpropagation algorithm) for training data sets.

[0091] [Fig.5] is a flow representation for a learning mode (or training, as indicated in the legend) and for an inference mode, according to the method of the invention.

[0092] During training, information is initially extracted from an occupancy grid 502 (spatial domain) to obtain, using the discrete cosine transform, a spectral representation (frequency domain).

[0093] Then the spectrum is masked (504) using the mask (learned on the training data for the learned mask), and the spectrum is flattened into a one-dimensional vector.

[0094] To enable the outputs of the dense neural network to encode the positions of the vehicles, the choice was made to encode the vehicles in the form of a segmentation of the occupancy grid generated from ground truth bounding boxes.

[0095] Thus, at the same time as the preprocessing of the occupancy grids, ground truth bounding boxes (510) are used to produce a segmentation (512) of the occupancy grid.

[0096] This segmentation is then projected into the frequency domain in the same way as for the occupancy grids and the neural network is trained to reproduce these output occupancy grid segmentation spectra from the input occupancy grid spectra.

[0097] The segmentation spectra are transformed into one-dimensional vectors of frequency coefficients by the same method (508) as for the original occupancy grid.

[0098] The one-dimensional vector of the spectrum of the original occupancy grid is passed into the neural network (506) and the result is compared with the one-dimensional vector of the spectrum of the segmentation to calculate a gradient and update the weights of the neurons in the network.

[0099] During inference, the resulting spectrum of the neural network (step 208) is decoded into a segmentation of the occupancy grid (step 212).

[0100] The method of the invention can be implemented in a device for segmenting material objects in a space surrounding an autonomous robot, this device comprising appropriate means for implementing all the steps of the method of the invention.

[0101] A method and an associated device have thus been described for the segmentation of material objects in a space surrounding an autonomous robot. The characteristics of the The method involves compression in the frequency domain of occupancy grids, their inference in a dense neural network, and a prediction, still in the spectral domain, of a segmentation of the occupancy grid.

[0102] The characteristics also relate to a method for designing an optimal mask by learning, retaining the maximum information from an occupancy grid for a given mask size.

[0103] The benefits of the invention are very low computational costs compared to methods based on convolutional networks, and that these costs are configurable thanks to the choice that can be made of the dimension of the spectrum.

[0104] Thus, advantageously, from a remote processing perspective, the proposed method allows only a compressed version of the occupancy grid to be sent to an autonomous vehicle to execute the AI ​​algorithm.

Claims

Claims

1. Method (200) for segmenting material objects in a space surrounding an autonomous robot, the method being implemented by computer in an inference phase, and comprising steps of: receiving (202) in the form of an occupancy grid data relating to the environment of the autonomous robot, an occupancy grid providing a discretized spatial representation of an environment, each cell of which corresponding to a zone of the environment contains an estimated probability of the presence of an object in this zone; transforming (204) the spatial occupancy grid into a spectral representation, the coefficients of the spectrum of the occupancy grid corresponding to frequency components associated with the data of the occupancy grid;reducing (206) the dimension of the spectrum of the occupancy grid by applying a frequency mask preserving the coefficients of the spectrum having a contribution to the highest occupancy grid; executing (208) an artificial intelligence algorithm on the compressed spectrum of the occupancy grid, said artificial intelligence algorithm implementing a neural network consisting of dense layers, trained during a learning phase; segmenting (210) the original occupancy grid by applying an inverse transform projecting the spectrum resulting from the artificial intelligence algorithm into the spatial domain; and identifying (212) material objects in the space surrounding the autonomous robot from the segmentation.;

2. The method according to claim 1 wherein the step (202) of receiving environmental data comprises a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of probabilities of occupation of an occupation grid.

3. The method of claim 1 or 2 wherein the step (204) of transforming into the frequency domain comprises applying a discrete cosine transform.

4. The method according to any one of claims 1 to 3 wherein the spectrum obtained in the transformation step (204) contains the amplitude of each frequency indexed on two dimensions.

5. The method according to any one of claims 1 to 4 wherein the dimension reduction step (206) consists of applying a frequency mask making it possible to retain determined frequencies.

6. The method according to any one of claims 1 to 5 wherein the frequency mask is a mask learned by training on the training data set of the dense neural network, and removing the frequencies containing the least information on the occupancy grids.

7. The method of any one of claims 1 to 6 wherein the neural network comprises dense layers of equal input and output sizes.

8. The method of any one of claims 1 to 7 wherein the step (208) of executing an artificial intelligence algorithm comprises a step of generating a one-dimensional vector of frequency coefficients for the spectrum of the occupancy grid, providing this vector as input to the dense neural network and obtaining a resulting spectrum.

9. The method according to any one of claims 1 to 8 wherein the training phase of the dense neural network comprises at least: a step of segmenting an occupancy grid from ground truth bounding boxes; a step of transforming the segmented occupancy grid into a one-dimensional vector of frequency coefficients; a step of comparing with the one-dimensional vector of frequency coefficients of the segmentation spectrum of the original occupancy grid obtained by inference of the dense neural network; and a step of updating the weights of the neurons of the network.

10. A computer program comprising code instructions for carrying out the steps of the method according to any one of claims 1 to 9, when said program is executed by a processor.

11. A device for segmenting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the method according to any one of claims 1 to Q

12. id y. An autonomous vehicle carrying a driving assistance system comprising a segmentation device according to claim 11, the device being configured to segment vehicles in a space surrounding the autonomous vehicle.

Images