Machine learning based systems and methods for controlling vehicles

Abstract

The present invention relates to a control device (10) for use in a vehicle (1), the vehicle comprising a perception system (2) using a set of sensors (200), each sensor providing data, the perception system comprising a device (100) for estimating variables comprising at least one characteristic of one or more objects detected in the vehicle's surrounding area, the estimation device comprising an online learning module (5) using a neural network (50) to estimate the variables, the neural network being associated with a set of weights. The learning module (5) comprises: a forward propagation module (51) configured to propagate data from the one or more sensors (200) as input to the neural network to provide a predicted output comprising an estimate of the variables; a fusion system (3) configured to determine a fused output by executing at least one sensor fusion algorithm using at least some of the predicted values; and a backpropagation module (32) configured to update the weights associated with the online neural network (50) by performing gradient descent backpropagation to determine a loss function representing the error between an improved prediction of the fused output and the predicted output.

Description

[Technical Field]

[0001] The present invention relates generally to control systems, and more particularly to vehicle control systems and methods.

[0002] Automated or semi-automated vehicles typically have embedded control systems, such as driver assistance systems for the control and safety of vehicle operation, e.g., ACC ("adaptive cruise control") distance regulation systems used to regulate the distance between vehicles.

[0003] Such driver assistance systems traditionally use a perception system comprising a set of sensors (e.g., cameras, lidar or radar) placed on the vehicle to detect environmental information that is used by a control device to control the vehicle.

[0004] The perception system comprises a set of perception modules associated with sensors to detect objects and / or predict the position of objects in the vehicle's environment using information provided by the sensors.

[0005] Each sensor provides information related to each detected object, which is then delivered at the output of the perception module to the fusion system.

[0006] The sensor fusion system processes the object information delivered by the perception module to determine an improved and integrated view of the detected objects.

[0007] In existing solutions, learning systems are used by perception systems (e.g., SSD, YOLO, SqueezeDet systems, etc.) to predict the location of objects. Such predictions are made by implementing an offline learning phase using a history of data determined or measured in previous time windows. In "offline" learning, data collected in real time by the perception system and the fusion module are not used for learning, and learning is performed in a phase when the driver assistance device is not operational.

[0008] To perform this offline learning phase, a database of learning images and a set of tables containing ground truth information are conventionally used. A machine learning algorithm is implemented to initialize the weights of the neural network from the image database. In existing solutions, this phase of initializing the weights is implemented "offline", i.e., outside the phase of use of the vehicle control system.

[0009] A neural network with weights fixed in this way can then be used in what is called a generalization phase, which is implemented online, to estimate the features of objects in the vehicle's environment, for example to detect objects in the vehicle's environment or to predict the trajectories of detected objects during online operation of a driver assistance system.

[0010] Therefore, in existing solutions, a learning phase that allows setting the weights of the neural network is performed offline, and the estimation of object features is then performed online (i.e., during operation of the vehicle control system) based on these fixed weights.

[0011] However, such training does not allow for the consideration of new images collected in real time during vehicle operation and is limited to training data stored in a static database. With detected objects that are, by definition, not known a priori, it is not possible to update the model parameters (neural network weights) in real time. Therefore, new predictions made are implemented without updating the model parameters (neural network weights) and may therefore be unreliable.

[0012] Various learning solutions have been proposed for driver assistance.

[0013] For example, U.S. Patent No. 10,254,759 (B1) proposes a method and system using offline reinforcement learning techniques. Such learning techniques are used to train virtual conversational agents. They are based on the extraction of observation information for learning in a simulation system, which is not suitable for driver assistance systems in vehicles. In particular, such approaches do not provide an online, embedded solution that allows for continuous improvement of predictions based on data provided by a fusion system. Furthermore, this approach is not suitable for object trajectory prediction or object detection in vehicles.

[0014] U.S. Patent Application Publication No. 2018 / 0124423(A1) describes a trajectory prediction method and system for determining predicted samples of agents in a scene based on past trajectories. The predicted samples are associated with a score based on a probability score that incorporates interactions between agents and semantic scene context. The predicted samples are iteratively refined using a regression function that accumulates scene context and agent interactions across iterations. However, such an approach is also not suitable for trajectory prediction and object detection in vehicles.

[0015] US Patent Application Publication No. 2019 / 0184561(A1) proposed a solution based on neural networks. This solution uses an encoder and a decoder. However, this solution uses inputs that are very specific to lidar data and offline learning. Furthermore, such a solution is not suitable for trajectory prediction or object detection in vehicles, which are related to decision-making or planning support techniques.

[0016] Therefore, existing solutions do not allow for improved estimation of the features of objects detected in the vehicle's environment based on machine learning.

[0017] Therefore, there is a need for machine learning based vehicle control devices and methods capable of providing improved feature estimates for objects detected in the vehicle's environment. Summary of the Invention

[0018] The present invention aims to improve the situation by proposing a control device implemented in a vehicle, the vehicle comprising a perception system using a set of sensors, each sensor providing data, the perception system comprising an estimation device for estimating variables comprising at least one feature for one or more objects detected in the environment of the vehicle, the estimation device comprising an online learning module for estimating the variables using a neural network, the neural network being associated with a set of weights. Advantageously, the learning module may comprise: a forward propagation module configured to propagate data from one or more sensors applied at the input of the neural network to provide a predicted output including an estimate of a variable; - a fusion system configured to determine a fusion output by implementing at least one sensor fusion algorithm based on at least some of the predicted values; a backpropagation module configured to update weights associated with the neural network online by determining a loss function representing an error between an improved prediction of the fused output and the predicted output and by performing gradient descent backpropagation.

[0019] In one embodiment, the variables may be state vectors that contain information regarding the position and / or movement of objects detected by the perception system.

[0020] Advantageously, the state vector may further include information regarding one or more detected objects.

[0021] The state vector may further include orbital parameters of the target object.

[0022] In one embodiment, an improved prediction may be determined by applying a Kalman filter.

[0023] In one embodiment, the device may include a playback buffer configured to store the output predicted by the estimation device and / or the fused output delivered by the fusion system.

[0024] In some embodiments, the device may include a recurrent neural network encoder configured to encode and compress data prior to storage in the playback buffer, and a decoder configured to decode and decompress data extracted from the playback buffer.

[0025] In particular, the encoder may be a recurrent neural network encoder, and the decoder may be a corresponding recurrent neural network decoder.

[0026] In some embodiments, the playback buffer may be prioritized.

[0027] The device may implement a condition for checking input data applied at the input of the neural network, and the input data is removed from the playback buffer if the loss function between the predicted value for this input sample and the fused output can be below a defined threshold.

[0028] A control method is also proposed to be implemented in a vehicle, the vehicle comprising a perception system using a set of sensors, each sensor providing data, the control method comprising estimating variables comprising at least one feature for one or more objects detected in the environment of the vehicle, the estimation implementing an online learning step for estimating the variables using a neural network, the neural network being associated with a set of weights. Advantageously, the online learning step may comprise the following steps: - propagating data from one or more sensors applied at the input of a neural network, thereby providing a predicted output comprising estimates of the variables; - determining a fusion output by implementing at least one sensor fusion algorithm based on at least some of the predicted values; - updating weights associated with the neural network online by performing gradient descent backpropagation to determine a loss function that represents the error between the improved prediction of the fused output and the predicted output.

[0029] Other characteristics, details and advantages of the invention will become apparent on reading the description given with reference to the accompanying drawings, given as an example. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 illustrates a driver assistance system that uses machine learning to estimate features of detected objects, according to some embodiments of the present invention. [Figure 2] FIG. 1 illustrates an estimation device according to some embodiments of the present invention. [Figure 3]1 is a simplified diagram illustrating a driver assistance system 10 according to one exemplary embodiment. [Figure 4] 1 is a flow diagram illustrating a neural network online training method according to some embodiments. [Figure 5] 1 is a flow diagram illustrating a training method according to one exemplary embodiment in one application of the present invention to trajectory prediction. [Figure 6] FIG. 1 illustrates one example implementation of a control system in which the perception system uses a single smart camera sensor for object trajectory prediction applications. [Figure 7] FIG. 10 illustrates another exemplary embodiment of a control system using encoding / decoding of data predicted by a neural network. DETAILED DESCRIPTION OF THE INVENTION

[0031] 1 shows a control system 10 embedded in a mobile device 1, for example a vehicle. The remaining description will be given with reference to a mobile device that is a vehicle, as a non-limiting example.

[0032] The control system 10 (hereinafter also referred to as the "driving assistance system") is configured to assist the driver in performing complex driving actions or maneuvers, to detect and avoid dangerous situations, and / or to limit the effects of such situations on the vehicle 1.

[0033] The control system 10 comprises a perception system 2 and a fusion system 3 embedded in the vehicle.

[0034] The control system 10 may further comprise a planning and decision support unit and one or more controllers (not shown).

[0035] The perception system 2 comprises one or more sensors 20 disposed on the vehicle 1 to measure variables related to the vehicle and / or the vehicle's environment. The control system 10 controls the operation of the vehicle 1 using information provided by the perception system 2 of the vehicle 1.

[0036] The driver assistance system 10 comprises an estimation device 100 configured to estimate variables relating to one or more object features representing characteristics of one or more objects detected in the environment of the vehicle 1 by using information provided by the perception system 2 of the vehicle 1 and by implementing an online machine learning ML algorithm using a neural network 50.

[0037] First, training is implemented to learn the weights of the neural network from a training database 12 that stores past (ground truth) observations of variables corresponding to data captured by the sensors.

[0038] Advantageously, online learning is further implemented during operation of the vehicle to update the weights of the neural network using the output delivered by the fusion system 3 determined based on the output predicted by the perception system 2, and determining the error between the improved prediction derived from the output from the fusion system 3 and the predicted output delivered by the perception system 2.

[0039] The weights of neural network 50 form the parameters of the neural or cognitive model represented by the neural network.

[0040] The training database 12 may include images of objects (eg, cars) and roads, and, associated with each image, expected values ​​of variables for object features that correspond to the ground truth.

[0041] The estimation device 100 is configured to estimate (or predict) object feature variables of images captured by the sensor 200 by using a neural network with up-to-date model parameters (weights) updated online, in a so-called generalization phase. Advantageously, the predicted variables themselves are used to update the weights of the neural network 50 based on the error between the variables predicted by the perception system 2 and the values ​​of the variables obtained after fusion by the fusion system 3.

[0042] Such learning, which is carried out online during operation of the driver assistance system 10, makes it possible to dynamically or quasi-dynamically update the parameters of the model, represented by the weights of the neural network 50, rather than using fixed weights determined in advance "offline" according to techniques from the prior art.

[0043] In some embodiments, the variables estimated by the estimation device 100 may include position information about objects detected in the environment of the vehicle, e.g., another vehicle, in an object detection application, or target object trajectory data in a target object trajectory prediction application.

[0044] The control system 10 may be configured to implement one or more control applications 14 configured to implement control methods relating to controlling the operation or safety of the vehicle based on information delivered by the fusion system 3, for example a cruise control application ACC that can regulate the distance between vehicles.

[0045] The sensors 200 of the perception system 2 may include various types of sensors, such as, but not limited to, one or more LIDAR (Laser Detection And Ranging) sensors, one or more radars, one or more cameras that may operate in visible and / or infrared, one or more ultrasonic sensors, one or more steering angle sensors, one or more wheel speed sensors, one or more brake pressure sensors, one or more yaw rate and lateral acceleration sensors, etc.

[0046] Objects in the environment of the vehicle 1 that can be detected by the estimation device 100 include moving objects, such as vehicles moving through the vehicle's environment.

[0047] In embodiments in which the perception system 2 uses sensors (e.g., lidar and / or radar) to detect objects in the environment of the vehicle 1, the object feature variables estimated by the estimation device may be, for example, a state vector containing a set of object parameters for each object detected by the radar, such as: - the type of object detected, - the location relative to the detected object, and - Uncertainty measures expressed by covariance matrices.

[0048] The fusion system 3 is configured to apply one or more processing algorithms (fusion algorithms) to the variables predicted by the perception system 2 based on information from the various sensors 200 and to provide a fusion output corresponding to a combined predicted variable for each detected object determined based on the variables predicted for the object based on information from the various sensors. For example, for location information of the detected objects predicted by the estimation device 100 based on the sensor information 200, the fusion system 3 provides more accurate location information corresponding to an improved view of the detected objects.

[0049] The perception system 2 may be associated with cognitive parameters that may be defined offline by calibrating the performance of the perception system 2 based on the embedded sensors 200.

[0050] Advantageously, the control system 10 may be configured to: - using past and / or future output data (fused data) from the fusion unit 3 with respect to the current time; - processing such past and / or future fused data to determine a more accurate estimate of the output from the fusion unit 3 at the current time (thereby providing an improved output from the fusion system); - Using such improved output from the fusion system 3 as a substitute for ground truth data stored in a training database 12, perform supervised "online" learning of a perception model to improve the estimation of object feature variables (e.g. used to detect objects in the vehicle's environment and / or to predict the trajectory of a target object).

[0051] Therefore, online learning may be based on delayed output from the estimation device 100.

[0052] Therefore, embodiments of the present invention advantageously use the output from the fusion system 3 to update the neural network weights online.

[0053] In particular, the estimation device 100 may comprise a neural network 50 based ML learning unit 5 that implements: - an initial learning (or training) phase to train a neural network 50 from an image database 12; a generalization phase for estimating (or predicting) detected object feature variables (e.g., detected object position or object trajectory prediction) based on the current weights; - Online learning to update the weights of the neural network 50 based on the output from the fusion system (determined in phase B based on the predictors), and the weights thus updated are used for new estimations in the generalization phase.

[0054] The ML (Machine Learning) learning algorithm makes it possible, for example, to obtain input images from one or more sensors and to return inferred variables (output predicted by the Perception System 2) comprising the number of detected objects (for example cars) and the positions of the detected objects in a generalization phase. The estimation of the inferred variables (output predicted by the Perception System 2) is improved by the Fusion System 3, which provides a fused output corresponding to the combined predicted variables.

[0055] A neural network is a computational model that mimics the operation of biological neural networks. A neural network includes neurons interconnected by synapses, which are typically implemented in the form of digital memory (e.g., resistive components). A neural network 50 may include multiple successive layers, including an input layer that carries input signals and an output layer that carries the results of predictions made by the neural network and one or more hidden layers. Each layer of a neural network derives its input from the output of the previous layer.

[0056] The signals propagated at the inputs and at the outputs of the layers of neural network 50 may be digital values ​​(information coded in the value of the signal) or electrical pulses in the case of pulse coding.

[0057] Each connection (also called a "synapse") between neurons in neural network 50 has a weight θ (a parameter of the neural model).

[0058] The training (learning) phase of neural network 50 consists in determining the neural network weights for use in the generalization phase.

[0059] ML (machine learning) algorithms are applied in a learning phase to optimize these weights.

[0060] By training the model represented by the neural network online using a large amount of data, including the output from the fusion system 3, the neural network 50 can more accurately learn the importance that one weight had relative to another.

[0061] In an initial learning phase (which may occur offline), neural network 50 first randomly initializes the weights and adjusts them using a gradient descent algorithm by checking whether the error, calculated using a loss function, between the output obtained from neural network 50 with input samples drawn from the training base (predicted output) and the target output (expected output) from the neural network decreases. Multiple iterations of this phase may be implemented, where the weights are updated at each iteration until the error reaches a certain value.

[0062] During the online learning phase, the neural network 50 adjusts the weights based on the error between: the output delivered by the neural network 50 obtained in response to the images provided by the sensor 200 (predicted output); - A value derived from the integrated fusion output based on such output predicted by the estimation device (improved predicted output).

[0063] The error between the predictions of the cognitive system and the fused output is represented by a loss function L using a gradient descent algorithm. Multiple iterations of this phase can be implemented, where the weights are updated at each iteration until the error reaches a certain value.

[0064] The learning unit 5 may comprise a forward propagation module 51 configured to apply inputs (samples) to the neural network 50 at each iteration of the online learning phase, which will produce an output, called a predicted output, in response to such inputs.

[0065] The learning unit 5 may further comprise a backpropagation module 52 for backpropagating the error to determine the weights of the neural network by applying a gradient descent backpropagation algorithm.

[0066] The ML learning unit 5 is advantageously configured to back-propagate the error between the improved predicted output derived from the fusion output and the predicted output delivered by the cognitive system 2, as well as to update the neural network weights "online".

[0067] Thus, the learning unit 5 allows to dynamically or quasi-dynamically train the neural network 50 for "online" (real-time or non-real-time) predictions and thereby obtain more reliable predictions.

[0068] In an embodiment in which the estimation device 100 is configured to determine characteristics of objects detected by the perception system 2 (e.g., by radar), the estimation device 100 can provide a predicted output (perception output) representing, for example, an object state vector including a set of predicted position information. The perception system 2 can transmit the object state vectors (perception object state vectors) corresponding to the various detected objects as determined by the estimation device 100 to the fusion system 3. The fusion system 3 can apply a fusion algorithm to determine a consolidated object state vector (fusion output) for each detected object that is more accurate than the perception output, based on the state vectors determined by the perception system 2 for the detected objects. Advantageously, the consolidated object state vectors (hereinafter also referred to as "improved object state vectors") determined by the fusion system 3 for the various objects can be used by the backpropagation module 52 of the online learning unit 5 to update weights based on the error between: - an improved prediction output (improved object state vector) derived from the output from the fusion system 3; - Output from Perception System 2 (Perception Object State Vector).

[0069] The driving assistance system 10 may include an error calculation unit 4 for calculating the error between the improved predicted output (improved object state vector) derived from the fusion system 3 and the output from the perception system 2 (perceived object state vector).

[0070] The calculated error is then represented by a loss function, which is then used to update the parameters of the cognitive model, also referred to as the "neural model." The parameters of the cognitive model correspond to the weights θ of the neural network 50 used by the estimation device 100.

[0071] The backpropagation algorithm advantageously uses the gradient of the loss function (the gradient of the loss function is TIFF0007796744000001.tif7170).

[0072] The backpropagation module 52 may be configured to calculate the partial derivatives of the loss function (the error metric determined by the error calculation unit 4) with respect to the parameters of the machine learning model (the neural network weights) by implementing a gradient descent backpropagation algorithm.

[0073] In this way, the weights of the neural network can be updated (adjusted) at each update provided at the output of the fusion system 3 and hence at each update of the error metric calculated by the error calculation unit 4 .

[0074] Such an interface between the fusion system 3 and the perception system 2 advantageously makes it possible to implement "online" backpropagation.

[0075] The weights may be updated locally or remotely, for example using V2X communication, when vehicle 1 is equipped with V2X communication means (for example, is an autonomous vehicle).

[0076] The weights updated in this way correspond to slight modifications of the weights used for object detection or object trajectory prediction that were used to generate the error metric used for online learning, which can then be used for new object detection or trajectory prediction performed by the sensor, which in turn provides new information about the detected object that will be used iteratively to update the weights online again in a feedback loop.

[0077] Such iterative online updating of the weights of a perception or prediction model allows for incremental and continuous improvement of the perception or prediction model.

[0078] Therefore, an estimate of the object state vector can be used to determine an error measure suitable for online learning via error backpropagation.

[0079] Thus, embodiments of the present invention enable more accurate prediction of detected object features (e.g., object detection and / or object trajectory prediction) that can be used in parallel, even if the prediction is delayed.

[0080] FIG. 2 is a diagram illustrating an estimation device 100 according to some embodiments.

[0081] In such an embodiment, the estimation device 100 may comprise an encoder 1001 configured to encode and compress object information returned by the fusion system 3 and / or the perception system 2 for use by the learning unit 5. In one embodiment, the encoder 1001 may be an encoder for a recurrent neural network (RNN), for example an LSTM (acronym for "long short-term memory") RNN. Such an embodiment is particularly suitable when the object information requires large memory, such as, for example, object trajectory information used for object trajectory prediction. The remaining description will be given mainly with reference to the RNN encoder 1001, as a non-limiting example.

[0082] The estimation device 100 may further comprise an experience playback buffer 1002 configured to store compressed object data (e.g., object trajectory data).

[0083] In one embodiment, the estimation device 100 may comprise a transformation unit 1003 configured to transform data that is not “independent and identically distributed” into “independent and identically distributed” (“iid”) data using filtering or delayed sampling of data from the playback buffer 1002.

[0084] Indeed, in some embodiments, when the estimation method implemented by the estimation device 100 is based, for example, on an orbit prediction algorithm, the data used by the estimation device is preferably independent and uniformly distributed (“iid”) data.

[0085] In practice, strongly correlated samples can distort the assumption that the data are independent and uniformly distributed (iid), which must be met for gradient estimation performed by gradient descent algorithms.

[0086] The playback buffer 1002 can be used to collect data sequentially as they arrive by clearing data previously stored in the buffer 1002, thereby allowing for enhanced learning.

[0087] To update the weights during online training, batches of data may be randomly sampled from the replay buffer 1002 and used to update the weights of the neural model. Some samples may have more influence than others on updating the weight parameters. For example, the loss function A larger gradient of TIFF0007796744000002.tif7170 may lead to a larger update of the weight θ. In one embodiment, storage in buffer 1002 may be given more priority and / or priority buffer reclamation may be implemented.

[0088] In such an embodiment, the estimation device 100 thus enables performing online and incremental machine learning to train a neural network using the object data (e.g., trajectory data) that has been compressed and encoded and then stored in the buffer 1002.

[0089] The decoder 1004 may be used to decode data extracted from the playback buffer 1002. The decoder 1004 is configured to perform the inverse operations of those implemented by the encoder 1001. Thus, in embodiments in which the RNN encoder 1001 is used, the RNN decoder 1004 is also used.

[0090] Embodiments of the present invention advantageously provide a feedback loop between the output from the fusion system 3 and the output from the perception system 2 .

[0091] Thus, embodiments of the present invention enable integrating information associated with each object detected by multiple sensors 200 such that the accuracy of the information is improved in the output from the fusion system 3 compared to the information provided by each perception unit 20 associated with each individual sensor 200. The error between the output from the perception system 2 and the output from the fusion system 3 is calculated and used to guide "online" learning and updating of the cognitive model weights (weights of the neural network 50). The error is then back-propagated to the neural network model 50, and partial derivatives of the error function (also referred to as a "cost function") for each parameter (i.e., weight) of the neural network model are calculated.

[0092] FIG. 3 is a simplified diagram illustrating the operation of the driver assistance system 10, according to one exemplary embodiment.

[0093] 3 considers, by way of non-limiting example, a pipeline of two sensors 200. It is further assumed that a convolutional neural network (CNN) based model is used for the object detection performed by the camera sensor 200 and the lidar sensor 200. However, it should be noted that the present invention may be applied more generally to any neural network model capable of performing online learning in a pipeline in which a perception system 2 is followed by a fusion system 3.

[0094] More generally, considering a pipeline of M sensors, and assuming each sensor 200-i of the M sensors detects P objects, the variables estimated by each sensor's estimation device 100 and each kth object detected by sensor 200-i can be represented by a state vector comprising: - Object Obj in a Cartesian coordinate system with the selected horizontal axis x and vertical axis y k Position (x ki ,y ki ), - an object Ob capturing a measure of the uncertainty of the prediction made by the sensor 200-i; jk The covariance matrix Cov associated with ki .

[0095] In the example of FIG. 3, for example, two sensors 200-1 and 200-2 are considered, where the first sensor 200-1 is a camera and the second sensor 200-2 is a lidar, and each sensor detects two identical objects Obj1 and Obj2, respectively.

[0096] The predicted variables based on the data captured by the first camera (“C”) sensor 200-1 may then include: - the state vector of the object Obj1: the position data x of the first object Obj1 1C , y 1C and the covariance matrix Cov 1C Contains {x 1C ,y 1C ,Cov 1C}, - the state vector of the object Obj2: the position data x of the second object Obj2 2L , y 2L and the covariance matrix Cov 2L Contains {x 2L ,y 2L ,Cov 2L}.

[0097] Variables predicted based on data captured by the second lidar (“L”) sensor 200-2 may include: - the state vector of the object Obj1: the position data x of the first object Obj1 1S , y 1S and the covariance matrix Cov relating the first object and the sensor 200-1 1S and {x 1S ,y 1S ,Cov 1S}, - the state vector of the object Obj2: the position data x of the second object Obj2 2L , y 2L and the covariance matrix Cov relating the second object and the sensor 200-2 2L and {x 2L ,y 2L ,Cov 2L}.

[0098] The information about the detected objects as provided by the perception system is then combined into the integrated position data (x kS ,y kS ) and the integrated covariance matrix Cov associated with the first object kS and the state vector (x kS ,y kS , CovkS) can be integrated by a fusion system 3 (by fusing the information) which determines, based on the integrated sensor information, an integrated predictor variable (fusion output) comprising for each detected object Objk,

[0099] Coordinates (x kS ,y kS ) is determined based on the information (xik, yik) provided for each object k and each sensor 200-i. kS is the information Cov provided for each object k and each sensor i. ki is determined based on the

[0100] In an example considering two sensors including a camera sensor and a lidar sensor, two sensors detecting two objects, the information about the detected objects as integrated by the fusion unit 2 includes: - The state vector of object Obj1: information x 1C , y 1C , x 1L , y 1L The integrated position data of the first object Obj1 based on Cov 1C and Cov 1L and an integrated covariance matrix associated with the first object based on {x 1S ,y 1S ,Cov 1S}, - The state vector of object Obj2: information x 2C , y 2C , x 2L , y 2L The integrated position data of the second object Obj2 based on Cov 2C and Cov 2L and an integrated covariance matrix {x 2S ,y 2S ,Cov 2S}.

[0101] The positioning information x provided by the fusion unit 2 of each k-th object kS , y kS has an associated uncertainty less than or equal to the uncertainty associated with the positioning information provided by sensors 200-i individually. Thus, there is a measurable error between the output from perception system 2 and the output from fusion unit 3.

[0102] The stochastic gradient descent backpropagation algorithm uses this error between the output from the cognitive system 2 and the output from the fusion unit 3, represented by the loss function, to update the weights of the neural network 50.

[0103] Thus, a feedback loop between the output from the fusion system 3 and the input of the perception system 2 allows the error metric to be used to online update the weights of the model represented by the neural network 50 used by the estimation device 100. The error metric is thus used as input for the learning module 5 for online learning, while the output from the online learning is used to update the perception model represented by the neural network 50. Thus, the accuracy of the estimation device (detection or prediction) is continuously improved compared to prior art driver assistance systems that perform weight learning and updating "offline".

[0104] FIG. 4 is a flow diagram illustrating a neural network online training method according to some embodiments.

[0105] The ML learning-based training method uses a set of parameters θ (neural network weights) and one or more neural networks 50 parameterized by: - x=image k The values ​​predicted by the neural network in response to the input data, also referred to as "input samples", denoted by TIFF0007796744000003.tif5170. Output or predicted value TIFF0007796744000004.tif5170 is defined by: TIFF0007796744000005.tif5170, - a loss function that defines the error between TIFF0007796744000006.tif5170, also known as Cost Function: Output y from fusion system 3 fusion The improved predicted value y derived from k , the predicted output delivered by Cognitive System 2 Fusion output calculated based on TIFF0007796744000007.tif5170, and Values ​​predicted by the neural network in response to input data representing images captured by one or more sensors 200 TIFF0007796744000008.tif5170.

[0106] The (real-time or non-real-time, delayed or non-delayed) fusion system 3 actually processes the object data obtained after applying one or more fusion algorithms implemented by the fusion system 3. A more accurate estimate of TIFF0007796744000009.tif5170 fusion to provide.

[0107] In some embodiments, the fused output y fusion The improved predicted value y derived from k ( TIFF0007796744000010.tif7170) can be obtained by performing the processing operations performed by the transformation unit 1003, for example by applying a Kalman filter. In one embodiment, the improved prediction y k is the fusion output y fusion It may be itself.

[0108] The learning method further uses: - Loss function Approximate value of TIFF0007796744000011.tif5170, - Updating the weights θ via gradient descent of the network parameters, such as: TIFF0007796744000012.tif18170, however, TIFF0007796744000013.tif5170 represents the gradient of the loss function.

[0109] More precisely, in step 400 , an image x corresponding to one or more detected objects is captured by the sensor 200 of the perception system 2 and applied to the neural network 50 .

[0110] At step 402, a response from neural network 50 to input x is calculated, which represents the output predicted by neural network 50. TIFF0007796744000014.tif5170 is determined using the current value of the weight θ according to: TIFF0007796744000015.tif5170

[0111] The predicted output in response to this input x TIFF0007796744000016.tif5170 corresponds to variables estimated by the estimation device 100 regarding features of objects detected in the vehicle's environment. For example, in an object detection application, where the variables estimated by the estimation device 100 are object state vectors containing position data of the detected objects and associated covariance matrices, the predicted output for an image x captured by the sensor 200 is TIFF0007796744000017.tif5170 represents the state vector predicted by the neural network based on the detected image x.

[0112] In step 403, the input x and the obtained predicted output A value pair containing TIFF0007796744000018.tif5170 may be stored in memory.

[0113] Steps 402 and 403 are repeated for images x corresponding to captures obtained by different sensors 200 .

[0114] In step 404, when a condition for transmission to the fusion system 3 is detected (e.g., expiration of a given or predefined time), various predicted values ​​are Fusion output y corresponding to TIFF0007796744000019.tif5170 fusionis calculated by the perception system 2, thereby providing an improved estimate of a variable related to the features of the detected object (e.g., position data or trajectory data of the target object). fusion are the various predicted values ​​corresponding to the various sensors 200. TIFF0007796744000020.tif5170 is determined by applying at least one fusion algorithm to the TIFF.

[0115] In one embodiment, samples corresponding to observations accumulated over a defined period (e.g., 5 seconds), which may or may not be prioritized, may be stored in an experience playback buffer 1002. In one embodiment, the samples may be pre-compressed and encoded by an encoder 1001 (e.g., an RNN encoder) before being stored in the playback buffer 1002.

[0116] In step 406, an improved predicted output y is derived from the fused output from the fusion system. k and output from cognitive system 2 The error between TIFF0007796744000021.tif5170 is calculated.

[0117] Improved predicted output y k is the output ( TIFF0007796744000022.tif7170). In one embodiment, the fusion output can be directly used as the improved prediction output. This error is expressed by the loss function The error function is represented by TIFF0007796744000023.tif5170. The error function is applied to the data stored in buffer 1002 after possible decoding by decoder 1004 as well as to the refined prediction output y k It can be determined based on:

[0118] In step 408, the weights of the neural network are calculated based on the gradient of the loss function. TIFF0007796744000024.tif5170 is updated by applying the stochastic gradient descent backpropagation algorithm.

[0119] The weights are set to the values ​​θ It can be updated by replacing it with TIFF0007796744000025.tif5170 TIFF0007796744000026.tif7170

[0120] Steps 404 and 408 may be repeated until a convergence condition is detected.

[0121] The driver assistance system 10 thus makes it possible to implement online, incremental learning using a neural network parameterized by a set of weights θ that are updated continuously and online.

[0122] In one embodiment, the output y predicted by neural network 50 k may be the response from the neural network 50 to input values ​​corresponding to the previous output from the fusion system 3. In such an embodiment, the improved predicted output TIFF0007796744000027.tif5170 is an output calculated based on the output from the fusion system (3) after processing, for example via Kalman filtering. In such an embodiment, an error function is determined between the output from the fusion system and an improved predicted output derived from the output from the fusion system.

[0123] In one embodiment, the output y predicted by neural network 50 k may be the response from neural network 50 to input values ​​corresponding to real-time captures obtained by sensor 200. In such an embodiment, the improved predicted output The TIFF0007796744000028.tif5170 may be an output calculated based on the output from the fusion system (3) after processing, e.g., via Kalman filtering, or may be the fusion output itself. In such an embodiment, an error function is determined between the output from the perception system and an improved predicted output derived from the output from the fusion system.

[0124] In one embodiment, the output y predicted by neural network 50 k may be the response from the neural network 50 to input values ​​corresponding to the previous output from the fusion system 3. In such an embodiment, the improved predicted output TIFF0007796744000029.tif5170 is an output calculated based on the output from the fusion system (3) after processing, e.g., via Kalman filtering. In such an embodiment, an error function is determined between the output from the fusion system and an improved predicted output derived from the output from the fusion system.

[0125] Those skilled in the art will readily appreciate that the present invention is not limited to variables estimated by the estimation device 100 being of the state vector type, including object positions x, y and covariance matrices.

[0126] For example, in one application of the present invention to object detection, neural network 50 may be, for example, a YOLO neural network (a convolutional neural network that loads an image only once before performing detection).

[0127] In such an exemplary embodiment, to detect objects, bounding boxes may be predicted around target objects by the neural network 50. Each bounding box has an associated vector that constitutes the variables estimated by the estimation device 100 and that includes a set of object features for each object, including, for example: - Presence p c and the object probability of - The bounding box (b x ,b y ,b h ,b w ) and the coordinates that define its position. - one or more classes (c1, c2, …, c M ), for example, the probability of an object belonging to a car class, a truck class, a pedestrian class, a bike class, etc.

[0128] In one exemplary application of the present invention to object detection, the predicted fusion output y fusion Improved prediction output derived from The determination of TIFF0007796744000030.tif7170 may use a Kalman filtering technique. Such a filtering operation may be implemented by the transformation unit 1003.

[0129] Therefore, the fusion system 3 uses Kalman filtering to calculate y k Improved estimation of object data TIFF0007796744000031.tif7170 (integrated detected object data or predicted data) can be provided.

[0130] For k=0 to N, the state vector x at time k k The following equation for is considered: x k+1 =A k x k +u k +α k (α represents Gaussian noise) k (prediction model with y k =C k x k +β k (β represents Gaussian noise) k (observation model with

[0131] The state vector is the state vector at time k based on the last measurement processing operation at time k'. The random variable denoted by TIFF0007796744000032.tif7170, where k'=k or k-1, is the estimated mean vector TIFF0007796744000033.tif7170 and Γ k|k-1 and the covariance matrix of the associated prediction errors, denoted as

[0132] The Kalman filtering step includes two main steps.

[0133] In a first step, called the prediction step, a prediction is made, which consists in determining: - Predicted mean: x k+1 =A k x k +u k - predicted covariance (representing the level of uncertainty increase): Γ k|k+1 =A k Gamma k|k A k T +Γ αk

[0134] In a second step, called the "correction step", the values ​​predicted in the prediction step of the Kalman filtering are corrected by determining: - the neural network 50 is used as a measurement system, the measurement y k "Innovation" (difference between measured and predicted values) derived from: TIFF0007796744000034.tif7170- Covariance "Innovation":S k =C k Gamma k|k-1 C k T +Γ βk - Kalman gain: TIFF0007796744000035.tif7170- Corrected average: TIFF0007796744000036.tif7170- Corrected covariances representing the level of uncertainty reduction: Gamma k|k =(IK k C k )Γ k|k-1

[0135] To be able to use such Kalman filtering, the data produced by the Kalman filter (fused data) may advantageously be stored for a sustained period in the playback buffer 1002 .

[0136] The stored data may be further processed by Kalman smoothing to improve the accuracy of the Kalman estimation. Such processing operations are suitable for online learning, and incremental online learning according to the present invention may be delayed.

[0137] Kalman smoothing involves implementing the following processing operations for K=0 to N: TIFF0007796744000037.tif29170

[0138] The smoothing step applied to the sensor fusion output stored in buffer 1002 reduces the value y predicted by neural network 50. k A more accurate estimate of Provide TIFF0007796744000038.tif7170.

[0139] In a first exemplary application of the present invention to object detection, according to some embodiments, for example, a YOLO neural network and three classes are considered, for which the variables estimated by the estimation device are given by:

[0140] y k =[p c b x b y b h b w c1c2c3] T

[0141] The following will also be considered: - (x i ,y i ,w i ,h i ) the bounding box coordinates associated with the location loss, - A confidence score c that indicates the model's confidence that the box contains the object i , - Pr(class i |object).

[0142] Loss function TIFF0007796744000039.tif5170, for example, is i , y i , w i , h i , c i and Pr(class i |Objects) can be defined based on

[0143] In such a first example, the learning method implements steps 402 to 408 as described below:

[0144] In step 402, neural network 50 predicts the output: TIFF0007796744000040.tif5170

[0145] In step 404, the predicted value y k is the corresponding fusion value determined by fusion system 2 It is set to TIFF0007796744000041.tif7170.

[0146] - In step 406, the loss function TIFF0007796744000042.tif7170 is computed for each detected object (e.g., for each bounding box in the YOLO neural network example), for example using a non-maximum suppression algorithm.

[0147] In step 408, a step of updating the neural network weights is implemented for each detected object (for each bounding box in the example of the YOLO neural network) by using a gradient descent algorithm, where each weight θ is updated with a value It will be updated to TIFF0007796744000043.tif8170.

[0148] The weights θ updated in step 404 are used to determine whether the new prediction of the neural network 50 is y k Improved estimate of It can be adjusted to be as close as possible to TIFF0007796744000044.tif7170.

[0149] In a second exemplary application, the estimation method may be applied to trajectory prediction.

[0150] In the following, the notation y (i) will be used to represent the predicted orbit vector: TIFF0007796744000045.tif13170

[0151] Furthermore, notation TIFF0007796744000046.tif6170 will be used to represent the fusion orbital vector: TIFF0007796744000047.tif26170

[0152] In this second example, it is believed that the cognitive system 2 does not use a playback buffer 1002 type memory 1002 to store the data used to determine the loss function.

[0153] Additionally, to ensure that the fused data is "iid" data, a random time counter may be used, the value of which is set after each update of the weights.

[0154] When the value set for the time counter expires, a new update of the weights may be repeatedly performed.

[0155] The loss function L or loss function may be any type of loss function, including a squared error function, a negative log-likelihood function, etc.

[0156] In the second example under consideration, the loss function L nll It is assumed that L is used as applied to a bivariate Gaussian distribution. However, it will be readily apparent to those skilled in the art that any other loss function may be used. nll is defined by: TIFF0007796744000048.tif11170

[0157] In such a second example, the online learning method implements the steps of FIG. 4 as follows:

[0158] In step 400, a trajectory vector x (i) is applied at the input of the neural network 50.

[0159] - In step 402, the predicted trajectory TIFF0007796744000049.tif6170 is the trajectory vector x applied at the input of the neural network. (i) and the neural network's current weights θ, which are determined over T seconds: TIFF0007796744000050.tif7170

[0160] - In step 403, the predicted trajectory TIFF0007796744000051.tif8170 and input trajectory vector x (i) ), including pairs TIFF0007796744000052.tif7170 is stored in memory 1002.

[0161] - The method is suspended until T seconds have elapsed (timer).

[0162] - In step 404, the fusion orbital vector y fusion is decided.

[0163] In step 406, a loss function is calculated, representing the error between the output from the fusion system and the output from the perception system 2.

[0164] In step 408, the value of the weight θ is It is set to TIFF0007796744000053.tif8170.

[0165] The stored pair can then be deleted and a new value can be set for the time counter.

[0166] The above steps may be repeated until a convergence condition is met.

[0167] FIG. 5 is a flow chart illustrating a third example learning method in one application of the present invention to trajectory prediction (variables estimated by the method for estimating variables related to detected objects include object trajectory parameters).

[0168] In such an exemplary embodiment, the online learning method uses a priority experience playback buffer 1002 .

[0169] In this embodiment, for each orbit prediction, the associated prediction loss is calculated online using the output from the delayed or non-delayed fusion system.

[0170] The ground truth corresponding to the predicted value can be approximated by performing updates to the output from the fusion system (delayed or non-delayed).

[0171] The loss function is the (delayed or non-delayed) fused output y fusionthe improved prediction output derived from the trajectory predicted by the neural network for each sensor under consideration. TIFF0007796744000054.tif8170. Depending on the threshold, the input x (i) It can further be determined whether the input x is useful for online learning. (i) is determined as useful for learning, a concise representation of the trajectory associated with this input, determined using, for example, an RNN encoder 1001, may be stored in a playback buffer 1002 (experience playback buffer).

[0172] Such an embodiment makes it possible to optimize and prioritize the experience corresponding to the input used to feed the learning table 12. Furthermore, the data stored in the playback buffer 1002 can be randomly sampled (by the transformation unit 1003) to ensure that the data is "iid". This embodiment makes it possible to optimize the samples used and to reuse samples.

[0173] The use of an RNN encoder makes it possible to optimize the playback buffer 1002 by compressing the trajectory information.

[0174] In the example in Figure 5, the loss function L nll is also used as a non-limiting example.

[0175] In step 500, the orbital vector x (i) The history of is extracted and encoded by the RNN encoder 1001, thereby generating a compressed vector RNN enc (x (i) ) is provided.

[0176] In step 501, the compressed vector RNN enc (x (i) ) (encoded samples) are stored in a playback buffer 1002.

[0177] In step 502, the predicted trajectory TIFF0007796744000055.tif6170 is the trajectory vector x provided as input to the neural network 50. (i) and is determined based on the current weights θ of the neural network. TIFF0007796744000056.tif8170: TIFF0007796744000057.tif7170

[0178] In step 504, the fusion orbit vector y is determined in advance by the fusion system. (i) is extracted (embodiment with delay).

[0179] In step 506, the loss function is calculated based on the fused output y (i) and predicted values ​​corresponding to cognitive output TIFF0007796744000058.tif8170 and the current weights θ of the network: TIFF0007796744000059.tif8170.

[0180] In step 507, the loss function If TIFF0007796744000060.tif8170 is smaller than the threshold, the sample value x (i) is removed from the buffer 1002 (it is not useful).

[0181] At step 508, each compressed sample RNN in buffer 1002 is enc (x (j) ) predicted trajectory TIFF0007796744000061.tif6170 is the compressed trajectory vector RNN enc (x (j) ) and the current weights θ of the neural network, which are determined as: TIFF0007796744000062.tif7170

[0182] In step 509, a loss function is applied to the predicted values ​​provided at the output of the neural network 50. TIFF0007796744000063.tif6170, the corresponding improved predicted output value (fusion output y (j) ) and the network's current weights θ, which are again calculated as: TIFF0007796744000064.tif7170

[0183] In step 510, the value of the weight θ is It is set to TIFF0007796744000065.tif8170.

[0184] The above steps may be repeated until a convergence condition is detected.

[0185] FIG. 6 shows one exemplary implementation of the control system 10 in which the perception system 2 uses a single smart camera sensor 200 for one application of the present invention to object trajectory prediction.

[0186] In this example, a camera sensor (200) observes trajectory points of a target object detected in the vehicle's environment (6001). The data captured by the sensor 200 is used to predict trajectories of the target object with current weights using a machine learning unit 5 based on a neural network 50 (6002).

[0187] The neural network 50 provides a predicted output 6003 that represents a trajectory predicted by the neural network 50 based on data from the sensors 200 applied at the inputs of the neural network 50 .

[0188] The predicted output is sent to a fusion system (3), which calculates (6004) an improved predicted output corresponding to the variables estimated by the estimation device 100. In this example, the variables represent the predicted trajectory of the target object and include trajectory parameters.

[0189] The estimation device provides a predicted trajectory to the driver assistance system 10 for use by the control application 14 .

[0190] Furthermore, the fusion system 3 sends the improved predicted output to the error calculation unit 4. The error calculation unit can store (6008) the predicted output (cognitive output) in a buffer 1002 where outputs corresponding to the observations (6005) are accumulated over a defined period (e.g., 5 seconds).

[0191] The transformation unit 1003 can apply additional processing operations to further increase the accuracy of the improved predicted output, for example by applying a Kalman filter (6006) as described above, thereby providing a refined predicted output (6007). The error calculation unit 4 then uses the data stored in the buffer 1002 and the refined predicted output to determine a loss function (6009) representing the error between the output from the perception system 2 and the refined predicted output. The weights are then updated by applying a gradient descent backpropagation algorithm using the loss function between the refined predicted output (delivered at the output of the Kalman filter 6006) and the output from the perception system, and a new ML prediction (6010) can be implemented by the online learning module 50 using the neural network 50 with the weights thus updated.

[0192] In the example of Figure 6, the output from fusion system 3 is used as ground truth for training.

[0193] In the embodiment of FIG. 6, the loss function corresponds to the error between the refined predicted output 6007 determined by the transformation module 1003 and the cognitive output 2 delivered by the cognitive system.

[0194] 7 shows another exemplary embodiment of the control system 10 that uses RNN encoding / decoding of data predicted by the neural network 50. In this example, the variables represent the predicted trajectories of the target object and include trajectory parameters. Additionally, the output from the fusion system is used as ground truth (input applied to the neural network 50 for online learning).

[0195] In the embodiment of Figure 7, the output from the fusion system 3 is used directly as an input that is applied to a neural network to determine a loss function, which then corresponds to the error between the output from the fusion system 3 and the refined predicted output delivered by the transformation unit 3.

[0196] In the embodiment of FIG. 7, the fusion output (improved prediction output) delivered by the fusion system 3 is applied (7000) at the input of the neural network 50 to predict (7002) the trajectory of the target object with the current weights based on the neural network 50 using the machine learning unit 5.

[0197] The neural network 50 provides a predicted output 7003 that represents a trajectory predicted by the neural network 50 based on data from the sensors 200 applied at the inputs of the neural network 50 .

[0198] The predicted output is sent to the RNN encoder 1001, which encodes and compresses the output predicted by the neural network 50 (7004).

[0199] Furthermore, the fusion system 3 sends the improved predicted output to the error calculation unit 4. The error calculation unit can store (7008) the predicted output in a buffer 1002 where the cognitive output corresponding to the observation is accumulated (7005) over a defined period (e.g., 5 seconds).

[0200] The transformation unit 1003 can apply additional processing operations to further increase the accuracy of the refined predicted output, for example by applying a Kalman filter (7006) as described above, thereby providing a refined predicted output (7007). The error calculation unit 4 then uses the data stored in the buffer 1002 to determine a loss function (7010) representing the error between the output from the perception system 2 and the refined predicted output, and after decoding by the RNN decoder (7009), determines the refined predicted output 7007. The weights are then updated by applying a gradient descent backpropagation algorithm using the loss function between the refined predicted output (delivered at the output of the Kalman filter 6006) and the output from the perception system, and a new ML prediction (7011) can be implemented by the online learning unit 5 using the neural network 50 with the weights thus updated.

[0201] 7 may be implemented without an RNN encoder / decoder (blocks 7004 and 7009). In such a variation, the output 7003 is stored directly in a buffer (block 7008) and the loss function is determined (block 7009) using data from the buffer 1002 directly, without RNN decoding.

[0202] Thus, embodiments of the present invention enable improved estimation of variables related to objects detected in a vehicle's environment by implementing online learning.

[0203] Training according to embodiments of the present invention allows for consideration of new images collected in real time while the vehicle is in operation and is not limited to using training data stored in an offline database: new estimations can be made while the driver assistance system is in operation, using neural network weights that are updated online.

[0204] Those skilled in the art will further appreciate that systems or subsystems according to embodiments of the present invention can be implemented in various ways using hardware, software, or a combination of hardware and software, particularly in the form of program code that can be distributed in the form of a program product. Specifically, the program code can be distributed using a computer-readable medium, which can include a computer-readable storage medium and a communication medium. The methods described herein can be implemented in the form of computer program instructions that can be executed by one or more processors in a computing device. These computer program instructions can also be stored in a computer-readable medium.

[0205] Furthermore, the present invention is not limited to the embodiments described above as non-limiting examples, but encompasses all variant embodiments that can be imagined by a person skilled in the art.

[0206] In particular, those skilled in the art will appreciate that the present invention is not limited to any particular type of sensor or to any particular number of sensors in the perception system 2 .

[0207] The present invention is not limited to any particular type of vehicle 1, but rather applies to any type of vehicle (examples of vehicles include, without limitation, cars, trucks, buses, etc.) Although they are not limited to such applications, embodiments of the present invention are particularly advantageous for implementation in autonomous vehicles, enabling autonomous vehicles connected by a communications network to exchange V2X messages.

[0208] The present invention is also not limited to any type of object detected in the vehicle's environment, but applies to any object that can be detected using the sensors 200 of the perception system 2 (pedestrians, trucks, motorbikes, etc.).

[0209] Furthermore, those skilled in the art will readily appreciate that the concept of "vehicle environment" as used in connection with object detection is defined relative to the range of sensors implemented in the vehicle.

[0210] The present invention is not limited to the variables estimated by the estimation device 100 described above as non-limiting examples. The present invention applies to any variables related to objects detected in the vehicle's environment, possibly including variables related to object position and / or object motion (speed, trajectory, etc.) and / or object characteristics (object type, etc.). The variables may have various formats. When the estimated variables are state vectors containing a set of parameters, the number of parameters may depend on the application of the present invention and on the specific features of the driver assistance system.

[0211] The present invention is also not limited to the example of a YOLO neural network given as an example herein, but applies to any type of neural network 50 that is used to estimate variables related to objects detected or detectable in the vehicle's environment based on machine learning.

[0212] Those skilled in the art will readily appreciate that the present invention is not limited to the exemplary loss functions provided above as examples.

Claims

1. A control device (10) implemented in a vehicle (1), the vehicle comprising a perception system (2) using a set of sensors (200), each sensor providing data, the perception system comprising an estimation device (100) for estimating variables comprising at least one feature for one or more objects detected in the vehicle's environment, the estimation device comprising an online learning module (5) using a neural network (50) to estimate the variables, the neural network being associated with a set of weights, the learning module (5) comprising: a forward propagation module (51) configured to propagate data from one or more sensors (200) applied to the input of said neural network so as to provide a predicted output comprising an estimate of said variables; a fusion system (3) configured to determine a fusion output by executing at least one sensor fusion algorithm based on at least some of the predicted values; a backpropagation module (32) configured to online update the weights associated with the neural network (50) by performing gradient descent backpropagation to determine a loss function representing the error between the improved prediction of the fused output and the predicted output; A control device comprising:

2. 2. The device of claim 1, wherein the variables are state vectors containing information about the position and / or movement of objects detected by the perception system.

3. The device of claim 2 , wherein the state vector further includes information about one or more detected objects.

4. 4. The device of claim 3, wherein the state vector further includes orbital parameters of the target object.

5. 5. A device according to claim 1, wherein the improved prediction is determined by applying a Kalman filter.

6. 6. The device according to claim 1, further comprising a playback buffer (1002) configured to store the output predicted by the estimation device (100) and / or the fused output delivered by the fusion system (3).

7. 7. The device of claim 1, further comprising a recurrent neural network encoder configured to encode and compress the data prior to storage in a playback buffer, and a decoder for decoding and decompressing the data extracted from the playback buffer (1002).

8. 8. The device of claim 7, wherein the encoder is a recurrent neural network encoder and the decoder is a recurrent neural network decoder.

9. 9. The device according to claim 6, wherein the device implements a condition for checking input data applied to the input of the neural network (50), and removes input data from the playback buffer (1002) if a loss function between a value predicted for this input data and the fused output is below a predetermined threshold.

10. A control method executed in a vehicle (1), the vehicle comprising a perception system (2) using a set of sensors (200), each sensor providing data, the control method comprising estimating variables comprising at least one feature for one or more objects detected in the environment of the vehicle, the estimation implementing an online learning step (5) using a neural network (50) to estimate the variables, the neural network being associated with a set of weights, the online learning step (5) comprising: - propagating data from one or more sensors (200) applied to the inputs of said neural network so as to provide a predicted output comprising estimates of said variables; - determining a fusion output by executing at least one sensor fusion algorithm based on at least some of the predicted values; - online updating the weights associated with the neural network (50) by performing gradient descent backpropagation to determine a loss function that represents the error between the improved prediction of the fused output and the predicted output; A control method comprising:

Images