SYSTEM AND METHOD FOR CONTROLLING MACHINE LEARNING-BASED VEHICLES

Description

technical field

[0001] The invention relates generally to control systems and in particular to vehicle control systems and methods.

[0002] Automated or semi-automated vehicles generally incorporate control systems such as driver assistance systems to control vehicle driving and safety, such as an ACC (Adaptive Cruise Control) system used to regulate the distance between vehicles.

[0003] Such driver assistance systems typically use a perception system comprising a set of sensors (e.g., cameras, LIDARs or RADARs) arranged on the vehicle to detect environmental information used by the control device to control the vehicle.

[0004] The perception system comprises a set of perception modules associated with sensors to perform object detection and / or position prediction of objects in the vehicle environment using information provided by the sensors.

[0005] Each sensor provides information associated with each detected object. This information is then delivered from the perception modules to a fusion system.

[0006] The sensor fusion system processes object information delivered by perception modules to determine an enhanced and consolidated view of detected objects.

[0007] In existing solutions, machine learning systems are used by the perception system to predict the position of an object (such as SSD, YOLO, and SQUEEZEDET systems). This prediction is achieved by implementing an offline learning phase, using a history of data determined or measured in previous time windows. Because the learning is offline, the data collected in real time by the perception system and fusion modules is not used for training; the learning takes place during phases when the driver assistance system is not operational.

[0008] To perform this offline learning phase, a training image database and a set of tables containing real-world data are typically used. A machine learning algorithm is implemented to initialize the neural network weights from the image database. In existing solutions, this weight initialization phase is performed offline, meaning outside of the vehicle control system's operational phases.

[0009] The neural network with the weights thus fixed can then be used in a so-called generalization phase implemented "online" ("offline" in Anglo-Saxon language) to estimate characteristics of objects in the environment of the vehicle, for example detecting objects in the environment of the vehicle or predicting trajectories of objects detected during the online operation of the driver assistance system.

[0010] Thus, in existing solutions, the learning phase which allows the weights of the neural network to be fixed is carried out offline, the estimation of object characteristics is then carried out online (i.e. during the operation of the vehicle control system) from these fixed weights.

[0011] However, this type of learning cannot account for new images collected in real time while the vehicle is operating and is limited to training data stored in the static database. Since the detected objects are by definition unknown a priori, it is not possible to update the model parameters (neural network weights) in real time. Therefore, new predictions are made without updating the model parameters (neural network weights) and can thus be unreliable.

[0012] Various learning solutions have been proposed in the context of driver assistance.

[0013] For example, in US 10 254 759 B1, a method and system using offline enhanced learning techniques are proposed. Such learning techniques are used to train a virtual interactive agent. They rely on extracting observational information for learning within a simulation system, which is not suitable for a driver assistance system in a vehicle. In particular, such an approach does not provide an online, embedded solution for continuously improving predictions based on data provided by the fusion system. Furthermore, this approach is not suitable for object trajectory prediction or object detection in a vehicle.

[0014] In US 2018 / 0124423 A1, a trajectory prediction method and system is described for determining prediction samples for agents in a scene based on a past trajectory. The prediction samples are associated with a score based on a probability score that incorporates agent interactions and a semantic scene context. The prediction samples are iteratively refined using a regression function that accumulates scene context and agent interactions across iterations. However, such an approach is also unsuitable for trajectory prediction and object detection in a vehicle.

[0015] In US 2019 / 0184561 A1, a neural network-based solution was proposed. This solution uses an encoder and a decoder. However, it relies on input that is highly specific to LiDAR data and offline learning. Furthermore, such a solution is suitable for decision support or planning techniques and is not appropriate for trajectory prediction or object detection in a vehicle.

[0016] Thus, existing solutions do not allow for improved estimation of the characteristics of objects detected in the vehicle's environment based on machine learning.

[0017] Other solutions are known, such as the article by Dirdal Johann: "End-to-end learning and sensor fusion with deep convolutional networks for steering an off-road unmanned ground vehicle", June 11, 2018, or that of Luyang Jing et al.: "An adaptive Multi-Sensor data fusion method based on deep convolutional neural networkds for fault diagnosis of planetary gearbox, sensors vol. 17, n. 2, February 21, 2017 page 414, but these solutions are also not satisfactory.

[0018] There is therefore a need for a machine learning-based vehicle control device and method capable of providing an improved estimation of the characteristics of objects detected in the vehicle's environment. General definition of the invention

[0019] The invention improves the situation by proposing a control device implemented in a vehicle. The vehicle comprises a perception system using a set of sensors, each sensor providing data. The perception system includes a device for estimating a quantity comprising at least one characteristic relating to one or more objects detected in the vehicle's environment. The estimation device includes a learning module using a neural network to estimate the quantity, the neural network being associated with a set of weights. Advantageously, the learning module may include: a forward propagation module configured to propagate data from one or more sensors applied as input to the neural network, so as to provide a predicted output including an estimate of the magnitude, a fusion system configured to determine a fusion output by implementing at least one sensor fusion algorithm from at least some of the predicted values, a backpropagation module configured to update the weights associated with the online neural network, by determining a loss function representing the error between an improved predicted value of the fusion output and the predicted output and performing backpropagation by gradient descent.

[0020] In one embodiment, the quantity can be a state vector comprising information relating to the position and / or movement of an object detected by the perception system.

[0021] Advantageously, the state vector can also include information relating to one or more detected objects.

[0022] The state vector may also include trajectory parameters of a target object.

[0023] In one embodiment, the improved predicted value can be determined by applying a Kalman filter.

[0024] In one embodiment, the device may include a replay buffer configured to store the outputs predicted by the estimation device and / or the fusion outputs delivered by the fusion system.

[0025] In some embodiments, the device may include a recurrent neural network encoder configured to encode and compress data before storage in the replay buffer, and a decoder configured to decode and decompress data extracted from the replay buffer.

[0026] In particular, the encoder can be a recurrent neural network encoder and the decoder can be a corresponding recurrent neural network decoder.

[0027] In some embodiments, the replay buffer can be prioritized.

[0028] The disposition can implement a test condition of input data applied as input to a neural network, with input data being removed from the replay buffer if the loss function between the predicted value for that input sample and the merge output can be less than a predefined threshold.

[0029] A control method implemented in a vehicle is further proposed, the vehicle comprising a perception system using an array of sensors, each sensor providing data. The control method comprises an estimation of a quantity including at least one characteristic relating to one or more objects detected in the vehicle's environment. The estimation implements a learning-by-line step using a neural network to estimate the quantity, the neural network being associated with a set of weights. Advantageously, the learning-by-line step may include the steps of: propagate data from one or more sensors applied as input to the neural network, which provides a predicted output including an estimate of the magnitude, determine a fusion output by implementing at least one sensor fusion algorithm from at least some of the predicted values, update the weights associated with the online neural network, by determining a loss function representing the error between an improved predicted value of the fusion output and the output predicted by performing gradient descent backpropagation. Brief description of the drawings

[0030] Other features, details and advantages of the invention will become apparent from the description provided with reference to the accompanying drawings given by way of example, which represent, respectively: [ Fig.1] is a diagram representing a driver assistance system using machine learning to estimate characteristics of detected objects, according to certain embodiments of the invention; [ Fig.2 ] is a diagram representing an estimation device, according to certain embodiments of the invention; [ Fig.3 ] is a simplified diagram representing the driver assistance system 10, according to an example implementation; [ Fig.4 ] is a flowchart representing the online learning process of the neural network, according to certain embodiments; Fig.5 ] is a flowchart representing the learning process according to an example implementation, in an application of the invention to trajectory prediction; [ Fig.6] shows an example of a control system implementation in which the perception system uses a single smart camera-type sensor for an object trajectory prediction application; and [ Fig.7 ] shows another example of a control system embodiment in which encoding / decoding of data predicted by the neural network is used. Detailed Description

[0031] There figure 1 represents a control system 10 embedded in a mobile device 1, such as a vehicle. The remainder of the description will be given with reference to a vehicle-type mobile device as a non-limiting example.

[0032] The control system 10 (also referred to below as 'driver assistance system') is configured to assist the driver in performing complex driving operations or maneuvers, detecting and avoiding dangerous situations, and / or limiting the impact of such situations on the vehicle 1.

[0033] The control system 10 includes a perception system 2 and a fusion system 3, mounted on the vehicle.

[0034] The control system 10 may further include a planning and decision support unit and one or more controllers (not shown).

[0035] The perception system 2 includes one or more sensors 20 arranged in the vehicle 1 to measure quantities relating to the vehicle and / or the vehicle's environment. The control system 10 uses the information provided by the perception system 2 of the vehicle 1 to control the operation of the vehicle 1.

[0036] The driver assistance system 10 includes an estimation device 100 configured to estimate a quantity relating to one or more object features representing features of one or more objects detected in the environment of vehicle 1 using information provided by the perception system 2 of vehicle 1 and implementing an online ML machine learning algorithm using a neural network 50.

[0037] Initially, learning is implemented to learn the weights of the neural network, from a training database 12 storing past values ​​(field reality) observed for the quantity corresponding to data captured by the sensors.

[0038] Advantageously, online learning is further implemented during vehicle operation to update the neural network weights using the output delivered by the fusion system 3, determined from the output predicted by the perception system 2 and determining the error between an improved predicted value derived from the output of the fusion system 3 and the predicted output delivered by the perception system 2.

[0039] The weights of the neural network 50 constitute the parameters of the neural or perception model represented by the neural network.

[0040] The training database 12 can include images of objects (cars for example) and roads, and associated with each image the expected value of the quantity relating to the object characteristics corresponding to the reality of the ground.

[0041] The estimation device 100 is configured to estimate (or predict), in a so-called generalization phase, the magnitude of object features for an image captured by a sensor 200, using the neural network with the latest model parameters (weights) updated online. Advantageously, the predicted magnitude is itself used to update the weights of the neural network 50 based on the error between the magnitude predicted by the perception system 2 and the value of the magnitude obtained after fusion by the fusion system 3.

[0042] Such learning, carried out online during the operation of the driving assistance system 10, makes it possible to update the parameters of the model, represented by the weights of the neural network 50, dynamically or almost dynamically instead of using fixed weights previously determined offline according to the state of the art approach.

[0043] In embodiments, the quantity estimated by the estimation device 100 may include position information relating to an object detected in the environment of a vehicle, such as another vehicle, in an object detection application, or target object trajectory data, in a target object trajectory prediction application.

[0044] The control system 10 can be configured to implement one or more control applications 14, such as an ACC remote control application capable of regulating the distance between vehicles, configured to implement a control process relating to the control of driving or vehicle safety from information delivered by the fusion system 3.

[0045] The 200 sensors of the perception system 2 can include various types of sensors, such as, for example and without limitation, one or more Lidar (Laser Detection And Ranging) sensors, one or more radars, one or more cameras which can be cameras operating in the visible spectrum and / or cameras operating in the infrared spectrum, one or more ultrasonic sensors, one or more steering angle sensors, one or more wheel speed sensors, one or more brake pressure sensors, one or more yaw rate and lateral acceleration sensors, etc.

[0046] Objects in the environment of vehicle 1 that can be detected by the estimation device 100 include moving objects such as, for example, vehicles moving in the environment of the vehicle.

[0047] In embodiments where the perception system 2 uses sensors to detect objects in the environment of the vehicle 1 (e.g., LIDAR and / or RADAR), the object characteristic value estimated by the estimation device can be, for example, a state vector comprising a set of object parameters for each object detected by the radar, such as, for example: The type of object detected; A position associated with the detected object; and A measure of uncertainty represented by a covariance matrix.

[0048] The fusion system 3 is configured to apply one or more processing algorithms (fusion algorithms) to the quantities predicted by the perception system 2 from information from various sensors 200 and to provide a fusion output corresponding to a consolidated predicted quantity for each detected object, determined from the quantities predicted for the object from the information from the different sensors. For example, for position information of a detected object, predicted by the estimation device 100 from information from sensors 200, the fusion system 3 provides more accurate position information corresponding to an improved view of the detected object.

[0049] The perception system 2 can be associated with perception parameters that can be defined offline by calibrating the performance of the perception system 2 based on the 200 onboard sensors.

[0050] Advantageously, the control system 10 can be configured to: use past and / or future output data from fusion unit 3 (fusion data) relative to a current time; Process such past and / or future fusion data to determine a more accurate estimate of the output of fusion unit 3 at a current time (which provides an improved output from the fusion system); Use such an improved output from fusion system 3 in place of the ground reality data, stored in the training database 12, to perform supervised "online" learning of perception models and improve the estimation of object feature magnitude (used for example to detect objects in the vehicle environment and / or predict trajectories of target objects).

[0051] Online learning can thus be based on a delayed exit from the estimation device 100.

[0052] The embodiments of the invention thus advantageously exploit the output of the fusion system 3 to update the weights of the neural networks online.

[0053] In particular, the estimation device 100 may include a learning unit ML 5 neural network-based 50 implementing: an initial learning (or training) phase to train the neural network 50 from the image database 12, a generalization phase to estimate (or predict) the magnitude of detected object features (e.g. detected object positions or object trajectory prediction) from the current weights, an online learning to update the weights of the neural network 50 from the output of the fusion system (determined from the magnitude predicted in phase B), the updated weights being used for new estimations in the generalization phase.

[0054] The machine learning (ML) algorithm allows, for example, taking input images from sensor(s) and returning an estimated quantity (output predicted by perception system 2) that includes the number of objects detected (cars, for example) and their positions during the generalization phase. The estimation of this estimated quantity (output predicted by perception system 2) is improved by the fusion system 3, which provides a fusion output corresponding to the consolidated predicted quantity.

[0055] A neural network is a computational model that mimics the functioning of biological neural networks. A neural network comprises neurons interconnected by synapses, generally implemented as digital memories (resistive components, for example). A neural network can consist of multiple successive layers, including an input layer carrying the input signal, an output layer carrying the result of the prediction made by the neural network, and one or more intermediate layers. Each layer of a neural network takes its inputs from the outputs of the preceding layer.

[0056] The signals propagated into and out of the layers of a neural network 50 can be numerical values ​​(information coded in the value of the signals), or electrical impulses in the case of impulse coding.

[0057] Each connection (also called a "synapse") between the neurons of the neural network 50 has a weight Θ (parameter of the neural model).

[0058] The training (learning) phase of neural network 50 consists of determining the weights of the neural network for use in the generalization phase.

[0059] An ML (Machine Learning) algorithm is applied in the learning phase to optimize these weights.

[0060] By training the model represented by the neural network online with a lot of data including the outputs of the fusion system 3, the neural network 50 can learn more accurately the importance that one weight had relative to another.

[0061] In the initial training phase (which can be done offline), the neural network 50 first initializes the weights randomly and adjusts them by checking whether the error between the network's output (predicted output) from an input sample taken from the training set and the network's target output (expected output), calculated using a loss function, decreases using a gradient descent algorithm. Numerous iterations of this phase can be implemented, in which the weights are updated at each iteration, until the error reaches a certain value.

[0062] In the online learning phase, the neural network 50 adjusts the weights based on the error between: the output delivered by the neural network 50 (predicted output) obtained in response to images provided by the sensors 200, and a value derived from the fusion output consolidated from such outputs predicted by the estimation device (improved predicted output).

[0063] The error between the perception system's prediction and the fusion output is represented by a loss function L, using a gradient descent algorithm. Numerous iterations of this phase can be implemented, with the weights updated at each iteration, until the error reaches a certain value.

[0064] The learning unit 5 may include a forward propagation module 51 configured to apply, at each iteration of the online learning phase, the inputs (samples) to the neural network 50 which will produce, in response to such an input, an output, called the predicted output.

[0065] Learning unit 5 may further include a backpropagation module 52 to backpropagate the error in order to determine the weights of the neural network by applying a gradient descent backpropagation algorithm.

[0066] The ML 5 learning unit is advantageously configured to perform a backprogression of the error between the improved predicted output derived from the fusion output and the predicted output delivered by the perception system 2 and update the weights of the neural network "inline".

[0067] Learning unit 5 thus allows the neural network 50 to be trained for an "online" prediction (in real time or not) in a dynamic or quasi-dynamic way and thus to obtain a more reliable prediction.

[0068] In embodiments where the estimation device 100 is configured to determine the characteristics of objects detected by the perception system 2 (e.g., by radar), the estimation device 100 can provide, for example, a predicted output representing an object state vector comprising a set of predicted position information (perception output). The perception system 2 can transmit to the fusion system 3 the object state vectors corresponding to the different detected objects (perception object state vectors), as determined by the estimation device 100. The fusion system 3 can apply fusion algorithms to determine a consolidated object state vector (fusion output) for each detected object that is more accurate than the perception output, based on the state vectors determined by the perception system 2 for the detected objects.Advantageously, the consolidated object state vectors (also referred to hereafter as "improved object state vectors"), determined by the fusion system 3 for the different objects, can be used by the backpropagation module 52 of the online learning unit 5 to update the weights based on the error between: . the improved predicted output derived from the output of fusion system 3 (improved object state vectors), and the output of perception system 2 (perception object state vectors).

[0069] The driver assistance system 10 may include an error calculation unit 4 to calculate the error between the improved predicted output derived from the fusion system 3 (improved object state vectors) and the output of the perception system 2 (perception object state vectors).

[0070] The calculated error is represented by a loss function. This loss function is then used to update the parameters of the perception models. The parameters of a perception model, also called a "neural model," correspond to the weights Θ of the neural network 50 used by the estimation device 100.

[0071] The backpropagation algorithm can advantageously be a Stochastic Gradient Descent algorithm based on the gradient of the loss function (the gradient of the loss function will be denoted hereafter (∇L( y ( i )< , ŷ ( i )< )).

[0072] The backpropagation module 52 can be configured to compute the partial derivatives of the loss function (error metric determined by the error calculation unit 4) with respect to the machine learning model parameters (neural network weights) by implementing the gradient descent backpropagation algorithm.

[0073] The weights of the neural networks can thus be updated (adjusted) with each update provided as output from the fusion system 3 and therefore with each update of the error metric calculated by the error calculation unit 4.

[0074] Such an interface between the fusion system 3 and the perception system 2 advantageously allows for the implementation of "online" backpropagation.

[0075] Weights can be updated locally or remotely using, for example, V2X communication when vehicle 1 is equipped with V2X communication means (autonomous vehicle for example).

[0076] The updated weights represent a small modification of the weights used for object detection or object trajectory prediction, which in turn generated the error metric used for online learning. These weights can then be used for further object detection or trajectory prediction by the sensors, which provides new information about the detected objects. This new information is then iteratively used to update the online weights again, creating a feedback loop.

[0077] Such iterative online updates of perception or prediction model weights allow for incremental and continuous improvement of perception or prediction models.

[0078] Estimates of object state vectors can thus be used to determine an error measure suitable for online learning via error backpropagation.

[0079] The embodiments of the invention thus allow a more precise prediction of detected object characteristics (object detection and / or object trajectory prediction for example) which can be used in parallel, even if the prediction is delayed.

[0080] There figure 2 is a diagram representing an estimation device 100, according to certain embodiments.

[0081] In such an embodiment, the estimation device 100 may include an encoder 1001 configured to encode and compress the object information returned by the fusion system 3 and / or the perception system 2 for use by the learning unit 5. In one embodiment, the encoder 1001 may be an encoder for a Recurrent Neural Network (RNN), for example, of the LSTM type (Long Short-Term Memory). Such an embodiment is particularly well-suited to cases where the object information requires significant memory, such as object trajectory information used for object trajectory prediction. The following description will be given primarily with reference to an RNN-type encoder 1001 as a non-limiting example.

[0082] The estimation device 100 may further include an experience replay buffer 1002 configured to store compressed object data (object trajectory data for example).

[0083] In one embodiment, the estimation device 100 may include a transformation unit 1003 configured to transform non-"independent and identically distributed" data into "independent and identically distributed" ("iid") data using filtering or delayed sampling of data from the replay buffer 1002.

[0084] Indeed, in certain embodiments, when the estimation process implemented by the estimation device 100 is, for example, based on a trajectory prediction algorithm, the data used by the estimation device are preferably independent and identically distributed data ("iid").

[0085] Indeed, highly correlated samples can skew the assumption that the data are independent and identically distributed (iid), which needs to be verified for gradient estimation by the gradient descent algorithm.

[0086] The replay buffer 1002 can be used to collect data sequentially as it arrives, overwriting previously stored data in buffer 1002, thereby strengthening learning.

[0087] To perform weight updates during online training, a batch of data can be randomly sampled from the replay buffer 1002 and used to update the neural model weights. Some samples may have more influence than others on updating the weight parameters. For example, a gradient of the loss function (∇L( y ( i )< , ŷ ( i)< ) larger can lead to larger updates of the weights θ. In one embodiment, storage in buffer 1002 can be further prioritized and / or a replay of the prioritized buffer can be implemented.

[0088] In such an embodiment, the estimation device 100 thus enables online and incremental machine learning to train neural networks using object data (trajectory data for example) compressed and encoded, then stored in buffer memory 1002.

[0089] A decoder 1004 can be used to decode data extracted from the replay buffer 1002. The decoder 1004 is configured to perform an operation inverse to that implemented by the encoder 1001. Thus, in the embodiment where an RNN-type encoder 1001 is used, an RNN-type decoder 1004 is also used.

[0090] The embodiments of the invention advantageously provide a feedback loop between the output of the fusion system 3 and the perception system 2.

[0091] The embodiments of the invention thus make it possible to consolidate the information associated with each object detected by a plurality of sensors 200 in such a way that the accuracy of the information is improved at the output of the fusion system 3 compared to the information provided by each perception unit 20 associated with an individual sensor 200. The error between the output of the perception system 2 and the output of the fusion system 3 is calculated and used to guide online learning and an update of the weights of the perception model (weights of the neural network 50). The error is then backpropagated to the neural network model 50, and partial derivatives of the error function (also called the "cost function") for each parameter (i.e., weight) of the neural network model are calculated.

[0092] There figure 3is a simplified diagram representing the operation of the driver assistance system 10 according to an example implementation.

[0093] In the example of the figure 3 A two-sensor pipeline is considered as a non-limiting example. It is further assumed that a convolutional neural network (CNN) model is used for object detection by a camera-type sensor and a LiDAR-type sensor. However, it should be noted that the invention can more generally be applied to any neural network model capable of performing online learning in a pipeline where a perception system is followed by a fusion system.

[0094] Considering more generally a pipeline of M sensors, assuming that each 200-i sensor among the M sensors detects P objects, the quantity estimated by the estimation device 100 for each sensor and each k-th object detected by a 200-i sensor can be represented by a state vector comprising: The position (x ki , y ki ) of the object Obj k in a Cartesian coordinate system having an x-axis and a y-axis chosen: A covariance matrix Cov ki associated with the object Ob jk which captures a measure of uncertainty of the predictions made by the 200-i sensor.

[0095] In the example of the figure 3 For example, two sensors 200-1 and 200-2 are considered, the first sensor 200-1 being the camera and the second sensor 200-2 being the LIDAR, each sensor detecting the same two objects Obj 1 and Obj 2.

[0096] The magnitude predicted from the data captured by the first 200-1 camera-type sensor ("C") can then include: the following state vector for the object Obj 1: {x 1C , y 1C , Cov 1C} comprising the position data x 1C , y 1C of the first object Obj1 and the covariance matrix COV 1C; the following state vector for the object Obj 2: {x 2L , y 2L , Cov 2L} comprising the position data x 2L , y 2L of the second object Obj 2 and the covariance matrix Cov 2L .

[0097] The magnitude predicted from the data captured by the second 200-2 Lidar-type sensor ("L") may include: the following state vector for the object Obj1: {x 1S , y 1S , Cov 1S} comprising the position data x 1S , y 1S of the first object Obj1 and the covariance matrix Cov 1S associated with the first object and the sensor 200-1; the following state vector for the object Obj2: {x 2L , y 2L , Cov 2L} comprising the position data x 2L , y 2L of the second object Obj2 and the covariance matrix Cov 2L associated with the second object and the sensor 200-2.

[0098] The information relating to the detected objects provided by the perception system can then be consolidated (by merging them) by the fusion system 3 which determines from the consolidated sensor information, a consolidated predicted quantity (fusion output) comprising, for each detected object Objk, the state vector (x kS , y kS , CovkS), including the consolidated position data (x kS , y kS ) for the first object Obj1 and the consolidated covariance matrix Cov kS associated with the first object.

[0099] The coordinates (x kS , y kS ) are determined from the information (xik, yik) provided for each object k and each sensor 200-i. The covariance matrix Cov kS is determined from the information Cov ki provided for each object k and each sensor i.

[0100] In the example considered, with two sensors comprising a camera-type sensor and a LiDAR sensor, and both sensors detecting two objects, the information relating to the detected objects consolidated by fusion unit 2 includes: the following state vector for the object Obj1: {x 1S , y 1S , Cov 1S} comprising the consolidated position data for the first object Obj1 from the information x 1C , y 1C , x 1L , y 1L and the consolidated covariance matrix associated with the first object from Cov 1C and Cov 1L; the following state vector for the object Obj2: x 2S , y 2S , Cov 2S comprising the consolidated position data for the second object Obj2 from the information x 2C , y 2C , x 2L , y 2L and the consolidated covariance matrix associated with the second object from Cov 2C and Cov 2L.

[0101] The positioning information x kS, y kS provided by the fusion unit 2 for each k-th object has an associated uncertainty less than or equal to that associated with the positioning information provided individually by the 200-i sensors. Thus, there is a measurable error between the output of the perception system 2 and the output of the fusion unit 3.

[0102] The Stochastic Gradient Descent backpropagation algorithm uses this error between the output of perception system 2 and the output of fusion unit 3, represented by the loss function, to update the weights of neural network 50.

[0103] The feedback loop between the output of the fusion system 3 and the input of the perception system 2 thus allows the error metric to be used to update the weights of the model represented by the neural network 50, used by the estimation device 100, in real time. The error metric is therefore used as input to the learning module 5 for online learning, while the output of the online learning is used to update the perception model represented by the neural network 50. The accuracy of the estimation device (detection or prediction) is consequently continuously improved compared to prior art driver assistance systems that perform learning and weight updates "offline".

[0104] There figure 4 is a flowchart representing the online learning process of the neural network, according to certain embodiments.

[0105] The machine learning (ML) process uses one or more neural networks parameterized by a set of parameters. θ (weight of the neural network) and: The predicted values ŷ k by the neural network in response to input data, also called "input samples", noted x = image k . The outputs or predicted values ŷ k are defined by: ŷ k = NeuralNet ( image k , θ ), A cost function, also called a loss function L ( yk , ŷ k ) defining an error between: an improved predicted value yk derived from the output and fusion of fusion system 3, the fusion output being calculated from predicted outputs ŷ k delivered by perception system 2, and a predicted value ŷ k by the neural network in response to input data representing images captured by one or more sensors 200.

[0106] The fusion system 3 (real-time or not, delayed or not) does indeed provide an estimate and fusion more precise object data ŷ k obtained after application of one or more fusion algorithms implemented by the fusion system 3.

[0107] In some embodiments, the improved predicted value yk (still noted) x̂ K | N ) derived from the fusion output and fusion can be obtained by performing a process carried out by the transformation unit 1003, for example by applying a Kalman filter. In one embodiment, the improved predicted value yk perhaps the fusion output and fusion herself.

[0108] The learning process also uses: An approximation of the loss function L ( yk , ŷ k ) ; An update of the weights Θ by gradient descent of network parameters such as: θ ← θ - α∇ θ L ( yk , ŷ k ) where ∇ θ L ( yk , ŷ k ) represents the gradient of the loss function.

[0109] More specifically, at step 400, an image x corresponding to one or more detected objects is captured by a sensor 200 of the perception system 2 and is applied to the neural network 50.

[0110] At step 402, the answer ŷ k The input x of neural network 50, representing the output predicted by neural network 50, is determined using the current value of the weights θ according to: y ^ k = NeuralNetwork x θ

[0111] The predicted exit ŷ kIn response to this input, x corresponds to a quantity estimated by the estimation device 100 relating to characteristics of objects detected in the vehicle's environment. For example, in an object detection application where the quantity estimated by the estimation device 100 is an object state vector comprising the position data of the detected object and the associated covariance matrix, the predicted output ŷ k for the image x captured by the sensor 200 represents the state vector predicted by the neural network from the detected image x.

[0112] At step 403, the pair of values ​​including the input x and the predicted output ŷ k The obtained data can be stored in memory.

[0113] Steps 402 and 403 are repeated for x images corresponding to captures taken by different 200 sensors.

[0114] At step 404, when a condition for sending to the merge system 3 is detected (for example, the expiration of a given or predefined time), the merge output and fusion , corresponding to the different predicted values ŷ k is calculated by the perception system 2, which provides an improved estimate of the magnitude relative to the characteristics of detected objects (e.g., position data or trajectory data of a target object). The fusion output and fusion is determined by applying at least one fusion algorithm to the different predicted values ŷ k corresponding to the different 200 sensors.

[0115] In one embodiment, samples corresponding to observations accumulated over a predefined time period (e.g., 5 seconds) can be stored in an experiment replay buffer 1002, which may or may not be prioritized. In another embodiment, the samples may be compressed and encoded by an encoder 1001 (e.g., an RNN encoder) before being stored in the replay buffer 1002.

[0116] At step 406, the error between an improved predicted output derived from the fusion outputs yk of the fusion system and the output ŷ k of perception system 2 is calculated.

[0117] The improved predicted output yk perhaps an exit (noted) x̂ K | N) derived from the output of the fusion system by applying a processing (e.g., Kalman filtering implemented by the transformation unit 1003). In one embodiment, the fusion output can be directly used as an improved predicted output. This error is represented by a loss function L ( yk , yk The error function can be determined from the data stored in buffer 1002 after possible decoding by a decoder 1004 and the improved predicted output yk .

[0118] In step 408, the neural network weights are updated by applying a stochastic gradient descent backpropagation algorithm to determine the gradient of the loss function ∇ θ L( yk , ŷ k ))

[0119] The weights can be updated by replacing each weight θ by value θ - α∇L( yk , ŷ k )) : θ ← θ − α ∇ θ L y k y ^ k

[0120] Steps 404 and 408 can be repeated until a convergence condition is detected.

[0121] The driver assistance system 10 thus enables the implementation of online, incremental learning, using a neural network parameterized by a set of weights θ Continuously updated online.

[0122] In one embodiment, the predicted output yk by neural network 50 can be the response of neural network 50 to an input value corresponding to the previous output of the fusion system 3. In such an embodiment, the improved predicted output ŷ k is an output calculated from the output of the fusion system (3) after processing, for example by Kalman-type filtering. In such an embodiment, the error function is determined between the improved predicted output derived from the output of the fusion system and the output of the fusion system.

[0123] In one embodiment, the predicted output yk The neural network 50 can be the response of the neural network 50 to an input value corresponding to the real-time captures taken by a sensor 200. In such an embodiment, the improved predicted output ŷ k The output can be calculated from the output of the fusion system (3) after processing, for example by Kalman filtering, or the fusion output itself. In such an embodiment, the error function is determined between the improved predicted output derived from the output of the fusion system and the output of the perception system.

[0124] In one embodiment, the predicted output yk by neural network 50 can be the response of neural network 50 to an input value corresponding to the previous output of the fusion system 3. In such an embodiment, the improved predicted output ŷ kis an output calculated from the output of the fusion system (3) after processing, for example by Kalman-type filtering. In such an embodiment, the error function is determined between the improved predicted output derived from the output of the fusion system and the output of the fusion system.

[0125] A person skilled in the art will readily understand that the invention is not limited to a quantity estimated by the estimation device 100 of the state vector type comprising object positions x, y and a covariance matrix.

[0126] For example, in an application of the invention to object detection, the neural network 50 can be, for example, a YOLO type neural network (convolutional neural network that loads the image only once before performing the detection).

[0127] In such an implementation, to detect objects, a rectangular frame ('bounding box') can be predicted around the objects of interest by the neural network 50. Each rectangular frame is associated with a vector comprising a set of object characteristics for each object, constituting the quantity estimated by the estimation device 100 and including, for example: a probability of presence of object pc, coordinates defining the position of the rectangular frame (bx , by , bh , bw ) in a Cartesian coordinate system, and a probability that the object belongs to one or more classes (c 1 , c 2 ,..., c M ), such as for example a car class, a truck class, a pedestrian class, a motorcycle class, etc.

[0128] In an example of the application of the invention to object detection, the determination of the improved predicted output x̂ K | N derived from the predicted merge output and fusionIt can use a Kalman filtering technique. Such filtering processing can be implemented by the transformation unit 1003.

[0129] The fusion system 3 can thus use Kalman filtering to provide an improved estimate x̂ k | N object data yk (consolidated object detection data or prediction data).

[0130] For k = 0 to N, the following equations for a state vector xk at time k are considered: x k + 1 = A k x k + u k + α k (Prediction model with αk representing Gaussian noise) y k = C k x k + β k (Observation model with βk representing Gaussian noise)

[0131] The state vector is a random variable denoted x̂ k | k , at time k based on the last measurement processing at time k' where k' = k or k-1. This random variable is characterized by an estimated mean vector x̂ k |k -1 and a covariance matrix of the associated prediction error, denoted Γ k | k -1.

[0132] The Kalman filtering step comprises two main steps.

[0133] In the first stage, known as prediction, a prediction is made consisting of determining: The predicted average: x k + 1 = A k x k + u k The predicted covariance (representing the level of increase in uncertainty): Γ k k + 1 = A k Γ k k A k T + Γ αk

[0134] In a second step, called the "correction step", the values ​​predicted in the Kalman filtering prediction step are corrected by determining: The "innovation" (difference between the measured value and the predicted value) derived from the measurement yk for which the neural network 50 is used as a measurement system: y ˜ k = y k − C k x ^ k k − 1 The "innovation" of covariance: S k = C k Γ k k − 1 C k T + Γ βk Kalman's Gain: K k = Γ k k − 1 C k T S k − 1 The adjusted average: x k k = x k k − 1 + K k y ˜ k The corrected covariance represents the level of uncertainty reduction: Γ k k = I − K k C k Γ k k − 1

[0135] In order to use such a Kalman filter, the data produced by the Kalman filter (merge data) can advantageously be stored for a duration in the replay buffer 1002.

[0136] The stored data can also be processed by Kalman smoothing to improve the accuracy of the Kalman estimates. Such processing is suitable for online learning, as the incremental online learning according to the invention can be delayed.

[0137] Kalman smoothing involves implementing the following treatments for K = 0 to N: J k = Γ k k A k T Γ k + 1 k − 1 x ^ k N = x ^ k k + K k x ^ k + 1 N − x ^ k + 1 k Γ k N = Γ k k + J k Γ k + 1 N − Γ k + 1 k J k T

[0138] The smoothing step applied to the sensor fusion outputs stored in buffer 1002 provides an estimated x̂ k | N more precise values ykpredicted by the neural network 50.

[0139] In a first example of application of the invention to object detection, according to certain embodiments, a YOLO-type neural network with 3 classes is considered, for example, for which the quantity estimated by the estimation device is given by: y k = p c b x b y b h b w c 1 c 2 c 3 ⊤

[0140] Furthermore, it is considered: The coordinates of a rectangular frame (`bounding box') associated with the loss of location, designated by ( xi , this , wi , hi ) ; A confidence score there representing the confidence level of the model according to which the frame contains the object; Conditional class probabilities represented by Pr(Class i | Object).

[0141] The loss function L ( yk , ŷ k ) perhaps, for example, defined from the parameters xi , go , hi , hi , down and Pr(Class i | Object).

[0142] In such a first example, the learning process implements steps 402 to 408 as described below:

[0143] At step 402, neural network 50 predicts the output: y ^ k = NeuralNetwork x θ At step 404, the predicted value yk is set to the corresponding merge value x̂ k | N determined by the fusion system 2. At step 406, the loss function L ( yk = x̂ k | N , ŷ k ) is calculated for each detected object (for example, for each rectangular frame in the YOLO neural network example) using, for example, a non-maximal deletion algorithm. In step 408, the neural network weight update step is implemented for each detected object (for each rectangular frame in the YOLO neural network example) using a gradient descent algorithm; each weight θ being updated to the value θ - α∇ θ L ( x̂ k | N , ŷ k ).

[0144] The weights θ Updates made at step 404 can be adjusted so that the new prediction of neural network 50 is as close as possible to the estimated x̂ k | N improved yk .

[0145] In a second application example, the estimation process can be applied to trajectory prediction.

[0146] In what follows, the notation y ( i )< will be used to represent the predicted trajectory vector: y i = x y 1 … … . x y T y

[0147] Furthermore, the rating ŷ ( i )< will be used to represent the fusion trajectory vector: y ^ i = μ x μ y σ x σ y ρ 1 … … . μ x μ y σ x σ y ρ T y

[0148] In this second example, it is assumed that the perception system 2 does not use 1002 type replay buffer memory to store the data used to determine the loss function.

[0149] Furthermore, to ensure that the merge data is "iid" data, a random time counter can be used, its value being set after each weight update.

[0150] When the set value for the time counter has expired, a new update of the weights can be performed iteratively.

[0151] The loss function L or loss function can be any type of loss function including a quadratic error function, a negative log probability function (`Negative Log Likelihood'), etc.

[0152] In the second example, it is assumed that the loss function Lnll is used, applied to a bivariate Gaussian distribution. However, those skilled in the art will readily understand that any other loss function can be used. The Lnll function is defined by: L = log σ x σ y 1 − ρ 2 + 0 1 − ρ 2 x − μ x 2 σ x 2 + y − μ y 2 σ y 2 − 2 ρ x − μ x y − μ y σ x σ y

[0153] The online learning process, in such a second example, implements the steps of the figure 4 as follows: At step 400, a trajectory vector x ( i )< , corresponding to the capture of a sensor 200 of the perception system 2, is applied as input to the neural network 50. At step 402, the predicted trajectory ŷ ( i )< is determined over T seconds from the trajectory vector x ( i )< applied to the input of the neural network and weights θ neural network currents: y ^ i = NeuralNet x i θ At stage 403, the couple ( ŷ ( i )< , x ( i)< ) including the predicted trajectory ŷ ( i )< = y ^ perception i and the input trajectory vector x ( i )< are saved in memory 1002. The process is paused until T seconds have elapsed (timer). At step 404, the fusion trajectory vector and fusion is determined. In step 406, the loss function is calculated, representing the error between the output of the fusion system and the output of perception system 2. In step 408, the weight values θ is put to θ − α ∇ θ L y fusion y ^ perception i . The saved pair can then be deleted and a new value can be set for the timer.

[0154] The previous steps can be repeated until a convergence condition is satisfied.

[0155] There figure 5is a flowchart representing the learning process according to a third example in an application of the invention to trajectory prediction (the quantity estimated by the quantity estimation process relative to a detected object includes object trajectory parameters).

[0156] In such an example of implementation, the online learning process uses a prioritized experience replay buffer of 1002.

[0157] In this embodiment, for each trajectory prediction, an associated prediction loss is calculated online using the output of the delayed or undelayed fusion system.

[0158] The reality on the ground corresponding to the predicted value can be approximated by performing updates to the output of the fusion system (delayed or not).

[0159] The loss function can be calculated between an improved predicted output derived from the fusion output and fusion(delayed or not) and the trajectory predicted by the neural network y ^ pred i for each sensor considered. Depending on a threshold value, it can further be determined whether an input x ( i )< is useful or not for online learning. If it is determined to be useful for learning, a compact representation of the trajectory associated with this input, for example determined by means of an RNN encoder 1001, can be stored in the replay buffer 1002 (experience replay buffer).

[0160] This embodiment allows for the optimization and prioritization of the experience corresponding to the inputs used to populate the training table 12. Furthermore, the data stored in the replay buffer 1002 can be randomly sampled to ensure that the data is i.i.d. (by the transformation unit 1003). This embodiment allows for the optimization and reuse of samples.

[0161] The use of the RNN encoder allows the 1002 replay buffer to be optimized by compressing trajectory information.

[0162] In the example of the figure 5 The loss function L nll is also used as a non-limiting example.

[0163] At step 500, the history of the trajectory vector x ( i )< is extracted and encoded by the RNN 1001 encoder, which provides a compressed RNN vector enc (x ( i )< ).

[0164] At step 501, the compressed RNN vector enc (x ( i )< ) (encoded sample) is stored in replay buffer 1002.

[0165] At stage 502, the predicted trajectory ŷ ( i )< is determined from the trajectory vector x ( i )< applied to the input of the neural network 50 and the weights θ neural network currents, with y ^ i = y ^ pred i : y ^ i = NeuralNet x i θ

[0166] At step 504, the fusion trajectory vector y ( i )< previously determined by the fusion system is extracted (implementation with delay).

[0167] In step 506, the loss function is calculated from the fusion output y ( i )< and predicted values y ^ pred i corresponding to the perception output, and current weights θ of the network: L y i y ^ pred i , in a delayed implementation mode.

[0168] At step 507, if the loss function L y i y ^ pred i is small compared to a threshold, the sample value x ( i )< is removed from buffer 1002 (not needed).

[0169] At step 508, for each compressed RNN sample enc (x ( j )< ) of buffer 1002, the predicted trajectory ŷ ( j )< is determined from the compressed RNN trajectory vector enc (x ( j )< ) and weights θ neural network currents: y ^ j = NeuralNet RNNenc x j , θ

[0170] In step 509, the loss function is recalculated from the predicted value. ŷ ( j )< provided at the output of neural network 50, of the improved predicted output value (fusion output y ( j )< ) corresponding and current weights θ of the network: L( y ( j )< , ŷ ( j )< ).

[0171] At step 510, the value of the weights θ is put to θ − α ∇ θ L y j y ^ pred j .

[0172] The preceding steps can be iterated until a convergence condition is detected.

[0173] There figure 6 shows an example of implementation of control system 10 in which perception system 2 uses a single 200 smart camera type sensor for an application of the invention to object trajectory prediction.

[0174] In this example, the camera-type sensor (200) observes trajectory points of a target object detected in the vehicle's environment (6001). The data captured by the sensor 200 is used to predict a trajectory of the target object with current weights (6002) using the neural network-based machine learning unit 50.

[0175] Neural network 50 provides a predicted output (6003) representing the trajectory predicted by neural network 50 from the sensor 200 data applied as input to neural network 50.

[0176] The predicted output is transmitted to the fusion system (3) which calculates an improved predicted output (6004) corresponding to the quantity estimated by the estimation device 100. In this example the quantity represents the trajectory of the predicted target object and includes trajectory parameters.

[0177] The estimation device provides the predicted trajectory to the driver assistance system 10 for use by a control application 14.

[0178] Furthermore, the fusion system 3 transmits the improved predicted output to the error calculation unit 4. The error calculation unit can store (6008) the predicted outputs (perception outputs) in a buffer 1002 where the outputs corresponding to observations (6005) over a predefined period of time (for example 5s) are accumulated.

[0179] The transformation unit 1003 can apply additional processing to further improve the accuracy of the enhanced predicted outputs, for example, by applying a Kalman filter (6006) as described above, which provides a refined predicted output (6007). The error calculation unit 4 then determines the loss function (6009) representing the error between the output of the perception system 2 and the refined predicted output using the data stored in buffer 1002 and the refined predicted output. The weights are then updated by applying a gradient descent backpropagation algorithm using the loss function between the refined predicted output (delivered at the output of the Kalman filter 6006) and the output of the perception system, and a new ML prediction (6010) can be implemented by the online learning module 50 using the neural network 50 with the weights thus updated.

[0180] In the example of the figure 6 The output of the fusion system 3 is used as a real-world example for learning.

[0181] In the implementation of the figure 6 , the loss function corresponds to the error between the refined predicted output 6007 determined by the transformation module 1003 and the perception output 2 delivered by the perception system.

[0182] There figure 7 Figure 10 shows another embodiment of control system 10 in which RNN encoding / decoding of the data predicted by neural network 50 is used. In this example, the quantity represents the trajectory of a predicted target object and includes trajectory parameters. Furthermore, the output of the fusion system is used as the real-world data (input applied to neural network 50 for online learning).

[0183] In the implementation of the figure 7The output of fusion system 3 is used directly as input to the neural network to determine the loss function. The loss function then corresponds to the error between the output of fusion system 3 and the refined predicted output delivered by transformation unit 3.

[0184] In the implementation of the figure 7 , the fusion output (improved predicted output) delivered by the fusion system 3 is applied as input to the neural network 50 (7000) to predict a trajectory of a target object with current weights (7002) using the machine learning unit 5 based on the neural network 50.

[0185] Neural network 50 provides a predicted output (7003) representing the trajectory predicted by neural network 50 from the sensor 200 data applied as input to neural network 50.

[0186] The predicted output is passed to an RNN 1001 encoder which encodes and compresses the output predicted by the neural network 50 (7004).

[0187] Furthermore, the fusion system 3 transmits the improved predicted output to the error calculation unit 4. The error calculation unit can store (7008) the predicted outputs in a buffer 1002 where the perception outputs corresponding to observations (7005) over a predefined period of time (for example 5s) are accumulated.

[0188] The transformation unit 1003 can apply additional processing to further improve the accuracy of the enhanced predicted outputs, for example by applying a Kalman filter (7006) as described above, which provides a refined predicted output (7007). The error calculation unit 4 then determines the loss function (7010) representing the error between the output of the perception system 2 and the refined predicted output using the data stored in buffer 1002, after decoding by an RNN decoder (7009), and the refined predicted output (7007).The weights are then updated by applying a gradient descent backpropagation algorithm using the loss function between the refined predicted output (delivered at the output of the 6006 Kalman filter) and the output of the perception system and a new ML prediction (7011) can be implemented by the online learning unit 5 using the neural network 50 with the weights thus updated.

[0189] A variant of the implementation method of the figure 7 can be implemented without using an RNN encoder / decoder (blocks 7004 and 7009). In such a variant, output 7003 is directly stored in the buffer (block 7008) and the loss function is determined directly using the data from buffer 1002, without RNN decoding (block 7009).

[0190] The embodiments of the invention thus allow an improved estimation of a quantity relating to an object detected in the environment of the vehicle by implementing online learning.

[0191] The learning process according to the embodiments of the invention allows for the incorporation of new images collected in real time during vehicle operation and is not limited to the use of training data stored in the offline database. New estimates can be made during the operation of the driver assistance system, using neural network weights updated online.

[0192] A person skilled in the art will further understand that the system or subsystems according to embodiments of the invention can be implemented in various ways by hardware, software, or a combination of hardware and software, including in the form of program code that can be distributed as a program product in various forms. In particular, the program code can be distributed using computer-readable media, which may include computer-readable storage media and communication media. The methods described herein can, in particular, be implemented in the form of computer program instructions executable by one or more processors in a computer system. These computer program instructions can also be stored on computer-readable media.

[0193] Furthermore, the invention is not limited to the embodiments described above by way of non-limiting example. It encompasses all alternative embodiments that could be considered by a person skilled in the art.

[0194] In particular, a person skilled in the art will understand that the invention is not limited to particular types of sensors of the perception system 20, to a particular number of sensors.

[0195] The invention is not limited to any particular type of vehicle 1 and applies to any type of vehicle (examples of vehicles include, but are not limited to, cars, trucks, buses, etc.). Although not limited to such applications, embodiments of the invention have a particular advantage for implementation in autonomous vehicles connected by communication networks enabling them to exchange V2X messages.

[0196] The invention is also not limited to one type of object detected in the environment of the vehicle and applies to any object detectable by means of sensors 200 of the perception system 2 (pedestrian, truck, motorcycle, etc.).

[0197] Furthermore, a person skilled in the art will easily understand that the notion of "vehicle environment" used in relation to object detection is defined in relation to the range of the sensors implemented in the vehicle.

[0198] The invention is not limited to the quantities estimated by the estimation device 100, described above by way of non-limiting example. It applies to any quantity relating to an object detected in the vehicle's environment, which may include quantities relating to the object's position and / or motion (speed, trajectory, etc.) and / or object characteristics (object type, etc.). The quantity may have different formats. When the estimated quantity is a state vector comprising a set of parameters, the number of parameters may depend on the application of the invention and the specific features of the driver assistance system.

[0199] The invention is also not limited to the YO-LO neural network example cited as an example in the description and applies to any type of neural network 50 used for estimation of quantities relating to objects detected or detectable in the vehicle environment, based on machine learning.

[0200] A person skilled in the art will readily understand that the invention is not limited to the loss function examples cited in the description above by way of example.

Claims

1. Control device (10) implemented in a vehicle (1), the vehicle comprising a perception system (2) using a set of sensors (200), each sensor providing data, the perception system comprising an estimation device (100) for estimating a variable comprising at least one feature in relation to one or more objects detected in the environment of the vehicle, the estimation device comprising an online learning module (5) using a neural network (50) to estimate said variable, the neural network being associated with a set of weights, characterized in that the learning module (5) comprises: - a forward propagation module (51) configured to propagate data from one or more sensors (200) applied at input of the neural network, so as to provide one or more predicted outputs (X1C, Y1C, Cov1C, X2C, Y2C, Cov2C, X1L, Y1L, Cov1L, x2L, Y2L, Cov2L) comprising an estimation of said variable; - a fusion system (3) configured to determine a fusion output (X1S, Y1S, Cov1S, X2S, Y2S, Cov2S) by implementing at least one sensor fusion algorithm based on at least some of said one or more predicted outputs, - a backpropagation module (32) configured to update the weights associated with the neural network (50) online by determining one or more loss functions representing the error (DeltaObj1, DeltaObj2) between an improved predicted value of said fusion output and said predicted output by performing a gradient descent backpropagation.

2. Device according to Claim 1, characterized in that said variable is a state vector comprising information in relation to the position and / or the movement of an object detected by the perception system.

3. Device according to Claim 2, characterized in that said state vector furthermore comprises information in relation to one or more detected objects.

4. Device according to Claim 3, characterized in that said state vector furthermore comprises trajectory parameters of a target object.

5. Device according to one of the preceding claims, characterized in that said improved predicted value is determined by applying a Kalman filter.

6. Device according to one of the preceding claims, characterized in that it comprises a replay buffer (1002) configured to store the outputs predicted by the estimation device (100) and / or the fusion outputs delivered by the fusion system (2).

7. Device according to one of the preceding claims, characterized in that it comprises a recurrent neural network encoder (1002) configured to encode and compress the data prior to storage in the replay buffer, and a decoder for decoding and decompressing the data extracted from the replay buffer (1002).

8. Device according to Claim 7, characterized in that the encoder is a recurrent neural network encoder and the decoder is a recurrent neural network decoder.

9. Device according to one of preceding Claims 6 to 8, characterized in that the replay buffer is prioritized.

10. Device according to one of preceding Claims 6 to 8, characterized in that it implements a condition for testing input data applied at input of a neural network (50), input data being deleted from the replay buffer (1002) if the loss function between the value predicted for this input sample and the fusion output is lower than a predefined threshold.