Method and device for detecting material objects in the space surrounding an autonomous robot
The use of single-stage convolutional neural networks on occupancy grids for vehicle detection in autonomous vehicles addresses computational limitations, enabling precise and efficient real-time vehicle detection with oriented bounding boxes.
Patent Information
- Application Number
- FR2022008833
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-09-02
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-09-02
AI Technical Summary
Existing methods for detecting vehicles in the environment of autonomous vehicles are computationally intensive and require significant resources, limiting their real-time execution on onboard computers with limited power, and often fail to accurately differentiate between different types of objects.
A method and device using static occupancy grids as input data, processed by a single-stage convolutional neural network, to generate a visual representation of bounding boxes around detected vehicles, with orientation, requiring less computational power and executing in 10 milliseconds per grid.
The solution provides precise, real-time detection of vehicles with oriented bounding boxes, reducing hardware resource usage and enabling efficient execution on autonomous vehicles, while maintaining high accuracy and orientation prediction.
Smart Images

Figure 00000020_0000 
Figure 00000020_0001 
Figure 00000021_0000
Abstract
Description
Title of the invention: Method and device for detecting material objects in a space surrounding an autonomous robot Field of the invention
[0001] The invention relates to the technical field of object detection, and more particularly relates to a method and a device for detecting material objects in a space surrounding an autonomous robot. State of the art
[0002] The detection of material objects in a space surrounding an autonomous robot generally consists of the interpretation of data provided by different sensors located on the autonomous robot in order to produce a model of the immediate environment of the vehicle. For applications to vehicles, the modeling of the environment falls within the field of perception for the autonomous car.
[0003] Driving automation in the context of autonomous vehicles is a major research challenge of our time. Such a system must combine, on the one hand, very high precision because an error can lead to serious accidents and, on the other hand, the system must be able to run in real time on on-board computers with limited computing power. Also, the detection of other vehicles present in the environment of an autonomous vehicle is of particular importance.
[0004] To enable precise location of obstacles, an autonomous vehicle is equipped with distance sensors (range sensors) such as 'LiDARs' (acronym for "Light Detection And Ranging"), radars, sonars or even stereoscopic cameras. These sensors have the particularity of their ability to position obstacles in space by measuring their distance and their angular position relative to that of the sensor.
[0005] Each of these types of sensors has its own advantages and limitations; stereoscopic cameras, for example, only work during the day, while LiDARs experience reduced acuity in the rain. To ensure maximum reliability in all circumstances, these sensors are generally used in combination on autonomous vehicles. A data fusion process then makes it possible to assemble the measurements from the different sensors and produce a unified model of the environment around the vehicle.
[0006] Different approaches to modeling the environment of a vehicle are proposed in the following articles: 1. H. Moravec and A. Elfes. “High resolution maps from wide angle sonar”. In 1985 IEEE International Conference on Robotics and Automation Pro-ceedings, volume 2, pages 116-121, Mar 1985. 2. Sascha Wirges, Tom Fischer, Christoph Stiller, and Jésus Balado Frias. “Object détection and classification in occupancy grid maps using deep convolutional networks”. In 2018 21 st International Conférence on Intelligent Transportation Systems (ITSC), pages 3530-3535. ISSN : 2153-0017. 3. Martin Simon, Stefan Milz, Karl Amende, and Horst-Michael Gross. Complex-YOLO: “An euler-region-proposal for real-time 3d object détection on point clouds”. In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision - ECCV 2018 Workshops, Lecture Notes in Computer Science, pages 197-209. Springer International Publishing. 4. Stefan Hoermann, Philipp Henzler, Martin Bach, and Klaus Dietmayer. “Object détection on dynamic occupancy grid maps using deep learning and automatic label génération”. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 826-833. ISSN: 1931-0587.
[0007] Article (1) by Moravec et al. proposes an environmental model based on occupancy grids which allows the fusion of measurements from different sensors and the taking into account of their noise and uncertainties via the use of a probabilistic mechanism.
[0008] Article (2) by Wirges et al. presents a model for detecting vehicles, pedestrians and bicycles on a top-down grid created from data from LiDAR sensors and containing LiDAR-specific data such as the intensity and height of the points. The detection architectures used are called "two-stage" with two neural networks and run in 100ms per grid.
[0009] Paper (3) by Simon et al. presents a model for detecting vehicles, pedestrians and bicycles on a top-view grid produced from LiDAR data very similar to that of paper (2). The grids are also LiDAR-specific because they contain information that is only produced by a LiDAR sensor such as the intensity of the points. The output predicts classes for "vehicle" objects, "pedestrian" objects and "bicycle" objects. The architecture of the neural network is inspired by that of the known YOLOv2 neural network which is a so-called "single-stage" architecture in a single neural network.
[0010] Article (4) by Hoermann et al. presents a model for object detection on dynamic occupancy grids (DOGMa). Dynamic occupancy grids combine the occupancy probabilities of static grids with velocity vectors quantifying the movement of each cell in the grid. The architecture of the neural network is of the "single-stage" type. Bounding boxes are produced and correspond to all the dynamic objects present in the scene. However, they are not classified, and therefore it is not possible to differentiate a car from a bicycle. or a pedestrian. It is more of a method of classification ("clustering" according to the established Anglicism) of dynamic cells with a non-zero speed vector s se, using a deep learning model.
[0011] The present invention provides an alternative to known approaches for detecting vehicles present in the immediate environment of a vehicle equipped with distance sensors. Brief description of the invention
[0012] The invention relates to a method (and a device) for detecting other vehicles present in the environment of a vehicle, which is based on the analysis of occupancy grids, and which restores in a visual representation in top view of the environment bounding boxes oriented around the detected vehicles.
[0013] More generally, the invention relates to a method for detecting predefined material objects in a space surrounding an autonomous robot.
[0014] Advantageously, the method of the invention uses static occupancy grids as input data for the analysis, the occupancy grids containing occupancy probabilities which can be generated from any distance sensor using a suitable sensor model.
[0015] Advantageously, the architecture of the circuit for analyzing and processing the input data is of the single-stage neural network type.
[0016] Advantageously, the architecture of the data analysis and processing circuit of the invention requires less computing power and is faster than known solutions.
[0017] The single-stage architecture of the convolutional neural network as proposed in different embodiment variants is inexpensive in terms of hardware resources (i.e. computational resources and memory usage), which allows its execution in computing units embedded on board a vehicle or an autonomous robot.
[0018] In an operational embodiment, the method of the invention executes in 10 milliseconds (ms) per occupancy grid.
[0019] Advantageously, the detection method of the invention makes it possible to generate a visual representation in top view of the environment of a vehicle where any vehicle detected around the vehicle is indicated by a bounding box, the bounding boxes being oriented, an orientation angle being calculated for each vehicle detected and restored by the orientation of the corresponding bounding box.
[0020] To this end, in one embodiment, the method for detecting material objects in a space surrounding an autonomous robot of the invention comprises the following steps:
[0021] - receive data relating to the environment of said autonomous robot, said data being representative of probabilities of occupation of each cell of at least minus an occupancy grid, an occupancy grid providing a discretized spatial representation of the environment around said autonomous robot;
[0022] - execute an artificial intelligence algorithm on the received data to generating an output matrix each cell of which contains a confidence score and parameters defining an oriented bounding box, said artificial intelligence algorithm implementing a single-stage convolutional neural network, trained during a supervised learning phase to produce, from occupancy grid data, a confidence score indicating the probability that a region contains a predefined object as well as coordinates of a bounding box for said predefined object; and
[0023] - extract from the cells of the output matrix, the cells whose score of confidence is above a predefined threshold.
[0024] In one embodiment, the step of receiving environmental data comprises a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of probabilities of occupation of an occupation grid.
[0025] In another embodiment, the method further comprises, before the transformation step, a step consisting of merging the data received from several on-board distance sensors.
[0026] In an alternative embodiment, the step of determining the regions comprising a predefined material object further comprises a step of calculating an orientation angle for the oriented bounding box.
[0027] Advantageously, the method may comprise a step consisting of determining that the same predefined material object is detected several times, and a step consisting of retaining only a single occurrence of detection.
[0028] Another object of the invention relates to a device for detecting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the method of the invention.
[0029] In one embodiment, the means for executing an artificial intelligence algorithm comprise means for implementing a neural network composed of alternating convolutional layers for extracting features and sub-sampling layers, said neural network further comprising two output filters on the last convolutional layer configured to produce an orientation angle of the bounding boxes.
[0030] In one embodiment of the device of the invention, the neural network is based on a YOLOv2 or YOLOv3 architecture.
[0031] In a particular implementation, the input occupancy grid of the network of neurons is a matrix of size 256 x 256 and the neural network is configured to produce an output matrix of size 8 x 8 or an output matrix of size 16 x 16.
[0032] Advantageously, each cell of the output matrix contains an n-tuple composed of elements representative of a confidence score and characteristics of a bounding box, including elements making it possible to calculate the orientation of said bounding box.
[0033] In another embodiment of the device of the invention, the means for executing an artificial intelligence algorithm comprise means for implementing a neural network based on a PIXOR architecture.
[0034] In a particular implementation, the occupancy grid provided as input to the neural network is a matrix of size 256 x 256 and the output matrix is a matrix of size 64 x 64, each cell of which contains an n-tuple composed of elements representative of a confidence score and characteristics of an oriented bounding box.
[0035] Advantageously, the device of the invention comprises a human-machine interface for displaying a representation of the output matrix obtained according to the method of the invention.
[0036] The invention also relates to an autonomous vehicle incorporating a driving assistance system, said system comprising a device for detecting other vehicles which itself comprises means for implementing the steps of the vehicle detection method according to the invention.
[0037] Advantageously, an object of the invention relates to a method for creating a learning database for training an artificial neural network, the method comprising at least steps consisting of:
[0038] - receive in the form of a 3D point cloud, data representative of a environment around an autonomous robot;
[0039] - annotate the 3D point cloud to identify predefined material objects;
[0040] - generate a 2D discretized spatial representation of the annotated 3D point cloud, said 2D discretized spatial representation being a matrix whose cells contain occupancy probabilities; and
[0041] - storing said 2D discretized spatial representation as data learning.
[0042] Advantageously, the method of the invention comprises a preliminary step consisting of creating a learning database according to the method of creating a learning database for training a proposed artificial neural network.
[0043] The invention can be applied to any type of robot, for example terrestrial robots (autonomous cars), aerial robots (drones), underwater robots, etc.
[0044] The invention also addresses a computer program comprising code instructions which, when the program is executed by a computer, cause the latter to implement the method of detecting material objects of the invention. Brief description of the figures
[0045] Other characteristics, details and advantages of the invention will emerge from reading the description given with reference to the appended drawings given by way of example and which represent:
[0046] [Fig.l] illustrates a discretized representation in the form of an occupancy grid, of the environment of an autonomous vehicle in top view.
[0047] [Fig.2] is a flowchart of the steps of the method of the invention for detecting a material object according to one embodiment.
[0048] [Fig.3] illustrates a first embodiment of the material object detection device according to the invention.
[0049] [Fig.4] illustrates a second embodiment of the material object detection device according to the invention.
[0050] Figures 5a and 5b illustrate a third embodiment of the material object detection device according to the invention. Detailed description of the invention
[0051] First, a reminder is given about the occupation grids.
[0052] An occupancy grid is a computer model of the environment commonly used in robotics and autonomous driving, introduced in the aforementioned article (1).
[0053] An occupancy grid is a probability matrix where each cell of the matrix corresponds to a portion of the space surrounding an autonomous robot and contains a probability of occupation of this portion of space.
[0054] A cell of an occupancy grid is generally defined in two dimensions (2D), the space being for example delimited by a square of 10 cm x 10 cm. A cell is characterized by a value between 0 and 1, representing the probability that the cell is occupied. [Fig.l] is a discretized representation of the environment in top view of an autonomous vehicle (or "Bird's Eye View" according to the established Anglicism), in the form of an occupancy grid where the occupancy probabilities are represented in different gray levels. For example, a probability of 0 (represented by a white level) means that the cell is definitely empty, while a probability of 1 (represented by a black level) means that the cell definitely contains an object. All intermediate values (different gray levels) quantify the uncertainty on the occupancy state of the cell.
[0055] The collection of all cells, usually millions of cells, forms a occupancy grid.
[0056] Computing occupancy grid probabilities is typically performed using parallel architectures such as GPUs (Graphics Processing Units). It has recently been demonstrated that occupancy grids can be computed and merged using only integer arithmetic on low-power microcontrollers, making them an important environment model for low-power real-time perception.
[0057] According to the prior art mentioned, an occupancy grid representative of the environment of an autonomous robot is calculated from the data constituting the image(s) of the environment provided by one or more sensors equipping the robot (for example point clouds for a LiDAR), after, where appropriate, fusion of the sensor data.
[0058] Merging sensor measurements into an occupancy grid is done in two steps. In a first step, an occupancy grid is constructed for each sensor using a suitable measurement noise model. In a second step, all occupancy grids are merged into a single grid using a Bayesian fusion process. When merging two cells, the resulting cell occupancy probability is more certain (closer to 0 or 1) if the two merged cells respectively have an occupancy probability in agreement regarding the cell occupancy state. Otherwise, the resulting occupancy probability is less certain (closer to 0.5). This process synthesizes the information from the different sensors into a compact representation.Occupancy grids are therefore an important model for the representation of the environment in real time, which is multimodal and takes into account the noise inherent in the different sensors.
[0059] The present invention uses occupancy grids as input data to a device for detecting material objects in the environment of an autonomous robot.
[0060] The proposed architectures produce accurate detection results with a prediction time allowing real-time execution.
[0061] In the context of autonomous driving where real-time detection of vehicles present in the environment of an autonomous vehicle is essential, the proposed detectors (i.e. the device of the invention in different implementation variants) allow precise detection of vehicles, via the prediction of their position, their size and their orientation.
[0062] The real-time detection of the method of the invention is understood to be real-time relative to the frequency of production of the data at the level of the distance sensors equipping the autonomous vehicle.
[0063] [Fig.2] is a flowchart of the steps in inference of the method 200 of the computer-implemented invention, which enables the detection of material objects in the environment of an autonomous robot, according to one embodiment.
[0064] A first step 202 of the method consists of receiving data relating to the environment of the autonomous robot (such as an autonomous vehicle) in the form of an occupancy grid. The data are thus values representative of occupancy probabilities for each cell of the occupancy grid.
[0065] In one embodiment, the received environmental data are generated from data from at least one distance sensor on board said autonomous robot, and the method may comprise a preliminary step consisting of transforming the sensor data into data representative of occupancy probabilities for an occupancy grid.
[0066] In an embodiment where the autonomous robot (i.e. the autonomous vehicle) has several distance sensors on board, the method makes it possible, before the step of transforming the sensor data into data representative of probabilities of occupation of an occupancy grid, to merge the data received from the different sensors. The fusion process can be a Bayesian type process and take into account the noise of the sensors and the uncertainties.
[0067] After receiving the data from the occupation grid representative of the environment of the autonomous robot, the method in a subsequent step 204, makes it possible to process the received data (possibly after fusion) by executing an artificial intelligence algorithm.
[0068] The corresponding neural network implemented was trained during a supervised learning phase, in order to determine from data from an occupancy grid, regions which contain a predefined material object as well as the coordinates of the bounding boxes of this object in the regions where it is present.
[0069] In the context of autonomous driving, the predefined material object to be detected is a vehicle which has a position, dimensions, an orientation, a front and a rear.
[0070] In one embodiment, the processing of the occupancy grid data is done by implementing a single-stage trained convolutional neural network.
[0071] In the context of autonomous driving, neural network training is done to enable the neural network, from a given occupancy grid as input, to determine the regions that contain a vehicle, as well as the position, dimensions and orientation of each vehicle.
[0072] Advantageously, the architecture of the data processing circuit of the occupancy grid is of the single-stage neural network type, using only a single neural network, and being executed only once per occupancy grid. This architecture allows real-time uses.
[0073] The inference accuracy of a neural network depends directly on the learning from which it was trained. Effective supervised learning relies on the quantity, variety, and quality of the input data sets that are submitted to the neural network for it to learn. In the context of the invention, where the neural network must analyze the data of an occupancy grid (i.e. occupancy probabilities per cell), the inventors had to constitute a new training database containing occupancy grid type data sets, to train different neural networks.
[0074] In particular, training data were generated from point clouds from LiDAR sensors onboard autonomous vehicles, which were annotated to indicate the position, dimensions and orientation of vehicles present in the environment of an autonomous vehicle. The annotated point clouds were then transformed, using a Bayesian process, into 2D occupancy grids that can be used by neural networks.
[0075] The annotations, which consist of 3D bounding boxes of the vehicles, are flattened to be transformed into 2D bounding boxes expressed in the same coordinate system as the 2D occupancy grids. These annotations are then used as ground truths during the training phase.
[0076] Step 204 thus makes it possible to generate an output matrix, each cell of which corresponds to a portion of the space surrounding the autonomous vehicle and which contains a confidence score making it possible to quantify the probability of the presence of the predefined material object as well as the coordinates of an oriented bounding box which delimits the contours of the material object.
[0077] In a following step 206, the method makes it possible to extract from among the cells of the output matrix, the cells whose confidence score is higher than a predefined threshold. The choice of this threshold depends on the intended application. Lowering the threshold makes it possible to obtain more bounding boxes at the risk that some of them do not correspond to the predefined material object.
[0078] Advantageously, the predefined material objects being objects having an orientation, the predicted bounding box is oriented. The method of the invention makes it possible, during the execution of the artificial intelligence algorithm, to directly produce or deduce an orientation angle 0 for the predicted bounding box.
[0079] In an alternative embodiment, the method makes it possible to determine that the same predefined material object is detected several times (duplicates), and makes it possible to retain only a single detection occurrence.
[0080] In one embodiment, to delete a duplicate, the method implements a step called “Non-Maximum Suppression” (NMS) which is a post-processing algorithm, known to those skilled in the art, which is responsible for merging all the de- protections that belong to the same object.
[0081] Advantageously, the implementation of the device for executing the method of the invention can be based on different architectures for the single-stage convolutional neural network.
[0082] Figures 3 to 5 illustrate different architectures suitable for implementing the method of the invention.
[0083] [Fig. 3] illustrates a first embodiment of the material object detection device according to the invention, in which the neural network is based on a YOLOv2 type architecture. Those skilled in the art of artificial intelligence know the YOLOv2 (or YOLO9000) model as a single-stage real-time object detection model, which makes it possible to detect objects via the use of a convolutional neural network. The article by J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger”. In 2017, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6517-6525. ISSN: 1063-6919 provides a more precise description of this model.
[0084] Generally, in its current use of image detection, this neural network takes an image as input and produces an 8x8 matrix as output which represents a subdivision of the input image. Each cell of this matrix contains a n-tuple (s, x, y, log w, log h) which represents a bounding box, where x and y are the coordinates of the center of the bounding box in the cell, w and h are its dimensions and s is a confidence score in the interval [0, 1] which represents the confidence level associated with this detection. A thresholding of the 64 predicted bounding boxes according to their confidence score makes it possible to obtain the object detections on the image. In parallel, a class is predicted for each output cell of the detector to categorize the detected object. The architecture used is a succession of convolutional layers whose number of filters increases with the depth of the layer.These layers are interspersed with max pooling layers that decrease the dimensions of the tensors by a factor of 2 at each pass. The cost function used is represented by the equation "L = Lclassification + Lregression" with Lclassification representing the binary cross-entropy on the confidence score s, and Lregression representing the smoothed L1 cost on (x, y, log w, log h) for cells containing a ground truth bounding box.
[0085] In the context of the invention, the architecture of the neural network 304 of the detector 300 shown in [Fig.3], is based on that of a YOLOv2 detector for 8 x 8 matrix, but the number of filters of each convolutional layer is divided by two (compared to the original detector).
[0086] This organization makes it possible to obtain a reduced neural network containing 4,961 463 trainable parameters, to provide low inference time.
[0087] As for the YOLOv2 detector, the architecture of the neural network of the invention is an alternation of convolutional layers which extract the characteristics of the input occupancy grid, and of “max-pooling” layers which sub-sample the width and height of the grid, until reaching an output matrix of target size.
[0088] Thus in the example illustrated, the architecture of the neural network 304 takes as input an occupancy grid of size 256 x 256, and is made up of five sub-sampling layers by maximum value (MaxPool2D), and a plurality of convolutional layers (Conv2D) whose dimensions of their convolution kernel (KxK = 1x1 or 3x3) and the number of filters (F = 16 or 32 or 64 or 128 or 256 or 512) are specific to each layer. These layers are followed by a classic “ReLU” activation function in deep learning.
[0089] Two convolutional layers (306, 308) both use the same input which is the output of the convolutional layer directly above.
[0090] A first convolutional layer 306 (Conv2D - 1x1 - l) of kernel (KxK=lxl) contains only one filter (F = 1), which after passing through a sigmoid activation function represents the confidence score of each cell of the output matrix.
[0091] Another convolutional layer 308 (Conv2D - 1x1 - 6) of kernel (KxK=lxl) contains 6 filters (F=6) in order to represent the coordinates and orientation of the bounding box of each cell in the form of an n-tuple (x, y, log w, log h, cos 0, sin 9).
[0092] Thus, in operation, the neural network 304 takes as input the data from an occupancy grid 302, and provides as output an output matrix, each cell of which corresponds to a portion of the space surrounding the autonomous vehicle and which contains a confidence score making it possible to quantify the probability of the presence of the predefined material object as well as the coordinates of an oriented bounding box which delimits the contours of the predefined material object.
[0093] For the example illustrated in [Fig.3], the output matrix produced is of size 8x8, with each cell covering a portion of the surrounding space equal to a square of 3.2 meters x 3.2 meters.
[0094] As indicated, each cell of the output matrix contains 7 channels (i.e. coming from the filter for the confidence score and the six filters for the bounding box coordinates) represented by an n-tuple (s, x, y, log w, log h, cos 0, sin 0) where:
[0095] - se [0, 1] is a confidence score used to classify detections as positive or negative with a chosen threshold;
[0096] - x and y represent the offset of the center of a bounding box from the corner upper left of the prediction cell;
[0097] - log w and log h represent the logarithm of the dimensions of the predicted cell; and
[0098] - cos 0 and sin 0 represent the cosine and sine of the bounding box heading.
[0099] During the training phase the difference between the output matrices produced by the neural network and the ground truth output matrices is quantified using the cost function. These ground truth output matrices are created from the bounding box annotations present in the dataset by fixing the tuple to the values (1, x, y, log w, log h, cos 0, sin 0) for each output cell that contains a center of a predicted bounding box and for the other cells by fixing all values of the tuple to zero (0, 0, 0, 0, 0, 0, 0, 0).
[0100] Since the detection operation is limited to a single type of material object, which is predefined, it is not necessary to predict a class per output cell. Also, the output of the neural network is adapted and does not contain a class prediction circuit.
[0101] Furthermore, unlike generic objects observable on images, the material objects to be detected in the context of the invention have a well-defined orientation that is useful to predict. Two output filters (cos 0, sin 0) are added to the last convolutional layer 308 of the neural network. These filters predict for each predefined material object that is detected, a cosine (cos 0) and a sine (sin 0) from which an orientation angle 0 with values in the interval [-ji, ji] will be extracted by an atan2 function (arctangent function with two arguments), thus making it possible to orient the corresponding predicted bounding box.
[0102] The loss used (i.e. the cost function) is given by the equation:
[0103] Lclassif ication "t" ^regression
[0104] with 'Lclassification' representing the binary cross-entropy loss on the confidence scores s, and with 'Lregression' representing the smooth L1 loss on the regression parameters (x, y, log w, log h, cos 0, sin 0) for cells containing the center of a bounding box on the ground truth output matrix.
[0105] [Fig.4] illustrates a second embodiment 400 of the material object detection device according to the invention, in which the neural network 404 based on a YOLOv2 type architecture, receives as input the data from an occupancy grid 402, and produces as output 410 an output matrix of size 16 xl6.
[0106] The subdivision as illustrated for the first embodiment forces the detector to offer at most only one bounding box per cell. For occupancy grids covering an area of 25.6 meters on each side, the implementation with an 8x8 subdivision according to the first embodiment gives at most only one vehicle detected per area of 3.2 meters on each side. Also, this can lead to not detecting certain vehicles (or predefined material objects) which are too close to each other.
[0107] To prevent the lack of detection of close vehicles, for example in dense traffic situations, the detector is modified to produce an output matrix of size 16 x 16.
[0108] The detector is implemented as for the first embodiment as an alternation of convolutional feature extraction layers and “max-pooling” layers, but the last max-pooling layer is removed in order to obtain an output matrix of size 16 x 16. This modification then makes it possible to detect at most one vehicle per area of 1.6 meters by 1.6 meters, thus reducing the risk of non-detection of close vehicles. The detector thus obtained contains 4,961,463 trainable parameters.
[0109] Each cell present in the output matrix of size 16 x 16, contains the same 7 channels (s, x, y, log w, log h, cos 0, sin 0) as each cell of the output matrix of size 8 x 8 of the first embodiment.
[0110] The loss used is the same combination of binary cross-entropy and smoothed L1 loss functions as for the first embodiment.
[0111] Similar to the first embodiment, two convolutional layers (406, 408) process the features into confidence scores and regression parameters, using a sigmoid activation function for the first layer 406 and a linear activation function for the second layer 408.
[0112] Still in a similar manner to the first embodiment, two output filters (cos 0, sin 0) are added to the last convolutional layer 408 of the neural network. These filters predict, for a detected material object, a cosine (cos 0) and a sine (sin 0) from which an orientation angle 0 with values in the interval [-jr, ji] will be extracted by a function atan2 (arctangent function with two arguments), and thus making it possible to orient the corresponding bounding box.
[0113] In a variant complementary to the second embodiment, an optional post-processing step of non-maximum suppression (NMS) is carried out on the output matrix in order to remove duplicates.
[0114] Indeed, switching to an output detection matrix of size 16 x 16 allows the detection of more vehicles, but this can introduce duplicates in the detection, i.e. having several bounding boxes predicted for the same vehicle.
[0115] The NMS post-processing step allows you to merge all the detections that relate to the same object.
[0116] In a non-illustrated alternative embodiment, the detector architecture is based on a YOLOv3 architecture, and produces an 8x8 output matrix with oriented bounding boxes.
[0117] Each cell in the 8x8 matrix contains the same 7 channels (s, x, y, log w, log h, cos 0, sin 0). The architecture is also an alternation of blocks of convolutional feature extraction layers and feature subsampling layers to go to the 8 x 8 output matrix. The main difference with the first embodiment lies in the internal architecture for extracting features from the occupancy grids.
[0118] Feature extraction blocks are composed of convolution layers with skip connections to aid network training. A skip connection is a convolutional layer that will use the output of a previous layer in addition to that of the immediately preceding layer. A network with skip connections is represented as a computational graph with branches.
[0119] The subsampling layers used here are convolutional layers with a step size of 2, unlike the first embodiment which uses subsampling layers by maximum value.
[0120] The loss used is the same combination of binary cross-entropy and smooth L1 loss as in the first embodiment. This architecture contains 10,158,679 trainable parameters.
[0121] [Fig.5a] illustrates a third embodiment 500 of the material object detection device according to the invention, and [Fig.5b] details an implementation of the neural network used.
[0122] In this embodiment, the detector is based on a single-stage deep neural network architecture, called PIXOR, which is used to detect vehicles on LiDAR point clouds. This PIXOR network directly produces parameter estimates (e.g., size, location, and orientation) of a bounding box for an oriented 3D object.
[0123] For more details on the PIXOR architecture, those skilled in the art may refer to the article by Yang et al, "PIXOR: Real-time 3D Object Detection from Point Clouds", arXiv: 1902.06326.
[0124] A PIXOR representation of a point cloud is a Birds Eye View (BEV) image representation that uses occupancy values to indicate the presence or absence of a corresponding point in the point cloud and, for 3D point clouds, uses height values to fully represent the points in the point cloud (similar to the depth channel of an RGBD image).
[0125] The BEV image is encoded as an input tensor, comprising an occupancy channel which encodes an occupancy map (usually binary) and, for a 3D point cloud, a height channel which encodes the height of each occupied pixel.
[0126] Since a PIXOR network directly produces bounding boxes oriented, it is then not necessary for this embodiment variant to add the additional filters of the other embodiments to calculate a heading angle.
[0127] For each occupancy grid provided as input 502 of the neural network 504 as a matrix of size 256 x 256, the detector provides an output matrix 510 of size 64 x 64 where each cell contains 7 channels represented by an n-tuple (s, cos 0, sin 0, dx, dy, log w, log 1) where:
[0128] - se [0, 1] is a confidence score used to classify detections as positive or negative with a chosen threshold;
[0129] - dx and dy represent the distance in x and y from the upper left corner of the cell of prediction at the center of the predicted vehicle;
[0130] - log w and log 1 represent the logarithm of the dimensions of the prediction cell; And
[0131] - cos 0 and sin 0 are the cosine and sine of the heading angle of the bounding box.
[0132] The ground truth output matrices are created from the annotations present in the dataset by fixing for each output cell confined in a bounding box, the values of the tuples corresponding to (1, cos 0, sin 0, dx, dy, log w, log 1) and by fixing to zero (0, 0, 0, 0, 0, 0, 0) the values of the other tuples for all other cells.
[0133] An additional and optional non-maximum suppression step 512 can be applied to positive bounding boxes (i.e. those whose confidence score is higher than the chosen predefined threshold) in post-processing to eliminate multiple detections.
[0134] In this embodiment, and according to the PIXOR architecture, the neural network calculates three levels of granularity of features which are combined into a single level as for a U-Net neural network. The features are extracted using residual blocks containing convolutional layers of increasing depth.
[0135] The loss used is the same as that used by the other embodiments according to the YOLOv2 or YOLOv3 architecture, i.e. by a combination of the binary cross-entropy loss function for classification and the smooth L1 loss function for regression.
[0136] This architecture contains 2,132,091 trainable parameters.
[0137] Different embodiments have been described for which variants can be envisaged.
[0138] Thus, the detection of object classes other than vehicles could be implemented by adding output layers of the neural network predicting a confidence score as to the object class to which the detection belongs. These output layers would be followed by a "softmax" function (normalized exponential function) allowing to normalize the confidence scores in the interval [0, 1].
[0139] A variation could involve the use of temporally contiguous occupancy grids to improve the results of the vehicle detector. One way to do this would be to stack these successive occupancy grids in the input of the detection model. This modification would allow the detector to use this rich data without requiring an architectural change, but it would require retraining the neural network.
[0140] In one embodiment, the device of the invention comprises or is coupled to a human-machine interface for displaying a representation of the output matrix obtained according to the method of the invention.
Claims
Claims
1. Method (200) for detecting material objects in a space surrounding an autonomous robot, the method being implemented by computer and comprising at least steps of: receiving (202) data relating to the environment of said autonomous robot, said data being representative of probabilities of occupation of each cell of at least one occupation grid, an occupation grid providing a discretized spatial representation of the environment around said autonomous robot;executing (204) an artificial intelligence algorithm on the received data to generate an output matrix each cell of which contains a confidence score and parameters defining an oriented bounding box, said artificial intelligence algorithm implementing a single-stage convolutional neural network, trained during a supervised learning phase to produce, from occupancy grid data, a confidence score indicating the probability that a region contains a predefined object as well as coordinates of a bounding box for said predefined object; and extracting (206) from among the cells of the output matrix, the cells whose confidence score is greater than a predefined threshold.;
2. The method according to claim 1 wherein the step of receiving environmental data comprises a step of receiving data from at least one distance sensor on board said autonomous robot, and a step of transforming the sensor data into data representative of probabilities of occupation of an occupation grid.
3. The method of claim 2 further comprising before the transforming step, a step of merging the data received from several on-board distance sensors.
4. The method according to any one of claims 1 to 3 wherein the step of determining the regions comprising a predefined material object further comprises a step of calculating an orientation angle for the oriented bounding box.
5. The method according to any one of claims 1 to 4 further comprising a step of determining that the same predefined material object is detected multiple times, and a step of retaining only one detection occurrence.
6. A computer program comprising code instructions for executing the steps of the method according to any one of claims 1 to 5, when said program is executed by a processor.
7. A device for detecting material objects in a space surrounding an autonomous robot, comprising means for implementing the steps of the method according to any one of claims 1 to 5.
8. The device according to claim 7 wherein the means for executing an artificial intelligence algorithm comprise means for implementing a neural network composed of alternating convolutional layers for extracting features and sub-sampling layers, said neural network further comprising two output filters on the last convolutional layer configured to produce an orientation angle of the bounding boxes.
9. The device of claim 8 wherein the neural network is based on a YOLOv2 or YOLOv3 architecture.
10. The device of claim 7 or 8 wherein the input occupancy grid of the neural network is a matrix of size 256 x 256 and wherein the neural network is configured to produce an output matrix of size 8 x 8 or an output matrix of size 16 x 16.
11. The device according to claim 10 wherein each cell of the output matrix contains an n-tuple composed of elements representative of a confidence score and characteristics of a bounding box, including elements making it possible to calculate the orientation of said bounding box.
12. The device according to claim 7 wherein the means for executing an artificial intelligence algorithm comprises means for implementing a neural network based on a PIXOR architecture.
13. The device according to claim 12 wherein the occupancy grid provided as input to the neural network is a matrix of size 256 x 256 and the output matrix is a matrix of size 64 x 64, each cell of which contains an n-tuple composed of elements representative of a confidence score and characteristics of an oriented bounding box.
14. The device according to any one of claims 7 to 13 further comprising a human-machine interface for displaying a representation of the output matrix obtained according to the method of one any of claims 1 to 5.
15. An autonomous vehicle carrying a driving assistance system comprising a device for detecting vehicles according to any one of claims 7 to 14.
16. A method according to any one of claims 1 to 5 comprising a preliminary step of creating a training database for training said neural network, the preliminary step consisting of: receiving in the form of a 3D point cloud, data representative of an environment around an autonomous robot; annotating the 3D point cloud to identify predefined material objects; generating a 2D discretized spatial representation of the annotated 3D point cloud, said 2D discretized spatial representation being a matrix whose cells contain occupancy probabilities; and storing said 2D discretized spatial representation as training data.
