Device and method for driving assisting or autonomous driving functions

Abstract

To provide a computer-implemented method (20) of predicting dynamics of objects in the surrounding of a vehicle (100).SOLUTION: This method starts with a step (S21) of receiving a first data sets characterizing dynamics of the objects respectively. Then, each of the first data sets is propagated through an encoder (11) outputting a latent representation for each of the first data sets (S22). Then, a graph based on the latent representations is generated (S23). Then, the graph is propagated through a Graph Neural Network (12) outputting an updated graph. Based on the updated graph, a decoder (13) outputs a predicted dynamic for a selected object for a subsequent time step.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention compromises an artificial intelligence based model architecture for driver assistance and autonomous driving functions, with the aim of enabling deep learning models in predictive driving tasks for safe and comfortable normal driving. [Background technology]

[0002] A driving scene is a highly dynamic environment with a constantly changing number of traffic participants. Any movement of a participant requires accurate recognition and prediction of the intentions of other participants around. For example, if the ego vehicle recognizes that a vehicle from an adjacent lane is cutting in, it can decide to brake for safety. In this setting, where predictions at one time step affect the behavior at the next time step, the ability to accurately model the interaction of participants is crucial. Moreover, predictions need to be made early enough before the behavior appears in order to be used for the functionality of driver assistance and autonomous driving systems.

[0003] A simple generative model of the driving task can be constructed as a dynamic Bayesian network, specifically a hidden Markov model, where the driving task is modeled as a hidden state that emits lateral position and velocity at each time step.

[0004] Such approaches are limited in their capabilities because they do not explicitly model interactions and cannot capture all actual dependencies in complex real-world driving scenarios. Moreover, they cannot learn from signal history, but rather operate on what is perceived at the current time step. Summary of the Invention [Problem to be solved by the invention]

[0005] Essentially, it is proposed to replace the dynamic Bayesian network with a deep neural network model that is able to explicitly model dependencies and interactions between traffic participants.

[0006] Models based on deep neural networks can learn complex dependencies in real-world driving scenarios from data. Deep neural network models outperform traditional models, especially as the complexity of interactions increases. This is due to the mastery of deep neural network models at learning interactions, which improves with increasing exposure to real-world driving data during training.

[0007] Another advantage of using deep neural networks is the better exploitation of signals that extend into the past, so more comprehensive information is available and more accurate predictions can be made based on it.

[0008] In summary, the advantages of the present invention are obtained by a deep learning based model architecture for behavior prediction in multi-agent settings by modeling their interactions, which allows taking into account historical driving data and learning complex joint behaviors across multiple traffic participants.

[0009] Furthermore, the proposed invention provides a framework for developing data-efficient and robust AI-based multi-agent predictive models to enable safe and comfortable normal driving.

[0010] The proposed architecture is capable of capturing the dynamics of the driving scene over a given time window. This application can be used for L0-L3 driving assistance functions and L4-L5 autonomous driving functions.

[0011] The modular design of the architecture enables it to be adapted to multiple application scenarios and simplifies optimization of the architecture since the contribution of each component can be easily identified. [Means for solving the problem]

[0012] In a first aspect, a computer-implemented method is proposed for predicting the mobility of objects in the surroundings of a vehicle. The mobility of an object can be the behavior or, preferably, the movement or trajectory of the object. The object can be another motorized vehicle (e.g., a motorcycle or a truck) or a traffic participant such as a pedestrian or a cyclist. Surroundings can be understood as the surroundings of the vehicle, in particular the surroundings in which objects may move and cross or come dangerously close to the vehicle's own trajectory.

[0013] The method starts with receiving first data sets each characterizing a mobility of an object. The first data sets are time series of features, one of the features of each series characterizing a current mobility and the other features of each series characterizing a previous mobility of the respective object. The current mobility is the mobility at a current time and the previous mobility is the mobility at a previous time. It should be noted that the number of objects, more precisely the number of first data sets, may vary over time, since one or more objects may appear and disappear around the vehicle. For example, the number of first data sets may remain constant from one time to the next, or may increase or decrease.

[0014] This is followed by propagating each of the first data sets through an encoder configured to output a latent representation for each of the first data sets.

[0015] Then, a step of generating a graph based on the latent representation is performed, where the nodes are in particular latent representations of each object, and the nodes are in particular connected to each other by edges. The step of generating a graph can be performed by defining the nodes and connecting each node by an edge. In other words, the latent representation can be stored in a graph-based data format.

[0016] This is followed by a step of propagating the graph through a graph neural network that outputs an updated graph, the graph neural network having been trained to update the graph.

[0017] This is followed by a step of propagating selected nodes of the updated graph through a decoder configured to output a predicted motility for at least a subsequent time step following the current time step. The decoder preferably predicts motility for a number of subsequent time steps. The method step of propagating may be performed for a number or each of the updated graph nodes in order to predict the motility of the object corresponding to the node.

[0018] GNNs (Graph Neural Networks) have the advantage that the number of objects can vary over time, so the method can handle different objects whose quantities change over time.

[0019] It is proposed that each feature of the first data set is assigned to a time point among subsequent time points t=0, 1, 2, ... k, where t=k may be the current time point in the current maneuver and t=k+1 may be the time point immediately preceding the current time point in the previous maneuver. The features include the object's speed and position at the corresponding time point, and preferably the object's category. The features may further include environmental features characterizing the environment of the corresponding object. The environmental features may be lane courses, traffic signs, etc. It is noted that the number of features may vary (e.g. increase or decrease) over time.

[0020] The encoder, the graph neural network and the decoder are preferably trained together end-to-end, in which a training dataset having a plurality of first datasets and assigned motility is provided, and parameters of the architecture are learned based on a loss function characterizing the difference between the output motility and the assigned motility.

[0021] Furthermore, it is proposed to control a technical system depending on the predicted mobility. The technical system can be a robot or a vehicle. The predicted mobility is used to provide actuator control signals for controlling actuators of the technical system. The first aspect of the invention is preferably utilized for controlling a vehicle by predicting the driving behavior of traffic participants around the ego-vehicle, in particular for automated cruise control or automated lane change functions. Furthermore, the proposed invention can be used as a prediction component of a fully autonomous driving software architecture stack.

[0022] In a second aspect of the invention, a computer program and a machine-readable storage device according to the first aspect are proposed.

[0023] Embodiments of the invention will now be described in further detail with reference to the following figures. [Brief description of the drawings]

[0024] [Figure 1] 1 is a schematic architecture. [Diagram 2] 1 is a flowchart of a method of utilizing the architecture. [Diagram 3] It is a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In a driving scene, traffic participants need to understand the intentions of nearby drivers and predict their future actions. For example, by predicting the next movements (e.g., overtaking, lane keeping) of all surrounding traffic participants, drivers can make safe decisions about their own movements. The behavior of one driver in one time step affects the behavior of nearby drivers in the next time step. Therefore, the ability to accurately model the interaction of participants is crucial for autonomous driving. Therefore, in the following embodiment, we propose a data-efficient and robust AI-based multi-agent prediction model to enable safe and comfortable normal driving.

[0026] The proposed artificial intelligence based model architecture 10 according to Fig. 1 is a multi-agent deep neural network model consisting of three main modules: an encoder 11, an interaction module 12 and a decoder 13. The model of the proposed architecture can be trained by real driving data collected at a selected sensor setup. It is noted that given the provided sensor signals of a selected sensor setup, data pre-processing and preferably post-processing are performed to obtain the required information of objects around the sensor setup.

[0027] The data required to train the model may include a set of traffic participant features, e.g., their estimated speed, acceleration, coordinate positions, etc., and a set of static environment features, e.g., lane division coordinate positions, lane presence probability, road curvature, etc.

[0028] Given a set of available features as input, the proposed architecture 10 encodes the input feature vector into an efficient latent representation, learns interactions using a graph representation of the encoded latent representation as input to a graph neural network, and converts the learned interactions into a future prediction sequence using a decoder 13 based on a recurrent neural network.

[0029] A preferred embodiment of the model architecture 10 comprises an encoder 11, an interaction module 12 and a decoder 13, where the encoder 11 is based on a Temporal Convolutional Network (TCN). It can be a general 1-d convolutional network, a modification of a classical 2-d Convolutional Neural Network (CNN) for time series prediction. The encoder 11 maps the input history into a latent space, where the mapping is done independently for each agent. Thus, for a given number of agents, the same number of latent representations are obtained. Each latent representation is preferably assigned to a corresponding agent, for example by an index.

[0030] The interaction module 12 models the interactions between any number of traffic participants and is preferably a graph neural network (GNN). It is known that the basic function of a GNN is to propagate information along a given graph and learn meaningful representations. That is, given the encoded input vectors of traffic agents, the GNN learns a new embedding of each agent by treating them as nodes in a graph, and information exchange between agents occurs by sending messages along the edges of the given graph. As a GNN architecture, the graph attention network is the preferred choice, as it can automatically learn which neighboring agents to pay attention to.

[0031] The decoder module 13 may consist of a feed-forward network, preferably a recurrent neural network (RNN), which converts the output of the interaction module into a future prediction sequence. The decoder maps the latent states to a goal prediction sequence. RNNs are a type of feed-forward neural network that can model such sequence data. RNNs assume that they can learn the actual dependencies between time steps to predict the future.

[0032] The proposed three-component model architecture 10 can be trained with driving data collected in an end-to-end manner and with various sensor configurations. More precisely, one or more driving scenarios with multiple objects / agents are recorded over a given time span. For a given time point in the time span, features x of each agent / object are determined. These features are used as training data. A short sequence of subsequent features (e.g., up to 5 seconds of subsequent time points for each object) is used as training input for the architecture 10, and features selected immediately after features of the training input characterizing the behavior (e.g., direction, speed) are used as labels for the output of the decoder 13. The content of the future prediction of the decoder module 13 can characterize the same content as the input or can characterize different content. For example, the prediction can be a category (e.g., whether the surrounding vehicle is keeping in the current lane or changing lanes) or the prediction can be the estimated coordinates of the surrounding vehicles. The prediction for each object is preferably the category of "lane keeping" or "lane changing" for the respective object. With supervised machine learning, the parameters of the three modules 11, 12, 13 of the architecture 10 are optimized to mimic the labels of the corresponding training input ground truth data.

[0033] FIG. 2 shows a flowchart 20 of a method for predicting the maneuverability of objects around a vehicle 100 using the trained architecture 10.

[0034] The method starts with a step S21 of receiving a first data set characterizing the motility of the object, the first data set comprising a time series of features (

number

number

number

number

number

number

number

number

[0035] Next, a propagation step S22 is performed. In this step, each of the first data sets is propagated to generate a latent representation (

number

[0036] Then, a generating step S23 is performed. In this step, the graph is converted into a latent representation (

number

[0037] Then, a propagating step S24 is performed, in which the updated graph is propagated through the graph neural network 12 which outputs the updated graph. Preferably, step S24 is performed L times until convergence.

[0038] Then, a propagating step (S25) is performed, in which the selected node of the updated graph is propagated through a decoder 13 configured to output a predicted mobility at a subsequent time step for the object assigned to the selected node.

[0039] An embodiment of an actuator with a control system 40 is shown in Figure 3. The actuator and its environment together are referred to as the actuator system. Preferably, equally spaced sensors 30 sense the state of the actuator system. The sensors 30 may comprise several sensors. The sensors 30 are preferably optical sensors that take images of the environment. An output signal S of the sensor 30 (or of each sensor if the sensor 30 comprises multiple sensors) encoding the sensed state is sent to the control system 40.

[0040] Thereby, the control system 40 receives a stream of sensor signals S. It then calculates a set of actuator control commands A in response to the stream of sensor signals S, which are then sent to the actuator units, which convert the control commands A into mechanical movements or changes in physical quantities. For example, the actuator units may convert the control commands A into electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical movements or changes. Specific, but non-limiting, examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc.

[0041] The control system 40 receives a stream of sensor signals S of the sensors 30 at an optional receiving unit 50. The receiving unit 50 converts the sensor signals S into input features x. The input features x are then passed to a module 60 that includes the trained architecture 10 of FIG.

[0042] The module 60 determines the prediction output by the trained architecture 10 and determines an output signal y depending on the prediction. The output signal y can be a control signal for controlling an actuator. Alternatively, the output signal y is sent to a transformation unit 80, which transforms the output signal y into a control command A. The actuator control command A is then sent to the actuator unit to control the actuator unit accordingly. Alternatively, the output signal y can be the control command A directly.

[0043] The actuator unit receives actuator control command A and is controlled accordingly to perform an action corresponding to actuator control command A. The actuator unit may comprise control logic to convert actuator control command A into a further control command, which is then used to control the actuator.

[0044] In further embodiments, the control system 40 may include the sensor 30. In yet other embodiments, the control system 40 may alternatively or additionally include an actuator.

[0045] In one embodiment, the module 60 may be designed to identify lane changes of traffic participants. Similarly, upon identifying a pedestrian and / or vehicle movement, a predicted future behavior of said pedestrian and / or vehicle may be estimated, and based on said estimated future behavior, a trajectory may be selected to avoid a collision with said pedestrian and / or vehicle, and the actuator control command A may be calculated to steer the vehicle to execute said trajectory.

[0046] In further embodiments, it may be envisioned that the control system 40 controls the display 10a instead of the actuators, and the display 10a may display control commands, etc. In other embodiments, the display 10a may be an output interface to a rendering device, such as a display, light source, speaker, vibration motor, etc., that may be used to generate a sensory-perceptible output signal that may be generated based on the output of the module 60.

[0047] Further, the control system 40 may include a processor 45 (or multiple processors) and at least one machine-readable storage medium 46 having stored thereon instructions that, when executed, cause the control system 40 to perform a method according to one aspect of the present invention.

[0048] In the preferred embodiment of FIG. 3, the control system 40 is used to control actuators in an at least partially autonomous robot, such as an at least partially autonomous vehicle 100 .

[0049] The sensors 30 may include one or more video sensors, one or more radar sensors, one or more ultrasonic sensors, one or more LiDAR sensors, and / or one or more location sensors (e.g., GPS), some or all of which are preferably, but not necessarily, integrated into the vehicle 100.

[0050] Alternatively or additionally, the sensor 30 may comprise an information system for determining the state of the actuator system, one example of such an information system is a weather information system that determines current or future weather conditions in the environment.

[0051] The actuator unit, preferably integrated in the vehicle 100, may be provided by the brakes, the propulsion system, the engine, the drive train or the steering of the vehicle 100. An actuator control command A may be determined such that an actuator (or actuators) unit is controlled such that the vehicle 100 avoids a collision with said detected object. The detected object may also be classified according to what the classifier 60 considers to be most likely, e.g. a pedestrian or a tree, and an actuator control command A may be determined depending on the classification.

[0052] In further embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown), which is capable of, for example, flying, swimming, diving or moving by stepping. The mobile robot may in particular be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all the above embodiments, the actuator command control A may be determined such that the propulsion unit and / or steering and / or braking of the mobile robot are controlled such that the mobile robot can avoid a collision with the identified object.

Claims

1. A computer-implemented method (20) for predicting the maneuverability of objects around a vehicle (100), comprising: Receiving (S21) a first data set characterising the motility of each of said objects (i), said first data set comprising a time series of features ( [0010] ), and one of the features of each time series is the current motility ( [0025] ), and the other features are characterized by previous motility ( [0030] (S21) characterizing the A latent representation ( [0045] (S22) propagating each of said first data sets through an encoder (11) configured to output The latent representation ( [0050] (S23) generating a graph based on the latent representation, where the nodes are the latent representations and the nodes are connected by edges; Propagating the graph through a graph neural network (12) (S24) which outputs the updated graph; Propagating (S25) at least selected nodes of the updated graph through a decoder (13) configured to output a predicted motility; The method (20) comprising:

2. 2. The method of claim 1, wherein the encoder (11) is a neural network, in particular a temporal convolutional network or a transformer, the graph neural network is a graph attention network, and the decoder (13) is a neural network, in particular a recurrent neural network or a LSTM or a transformer.

3. 3. The method of claim 1 or 2, wherein each feature of the first data set is assigned to a time point of subsequent time points (t=0, 1, 2, ... k), the features of the first data set characterizing at least the velocity and position of the object at the corresponding time point, in particular the category of the object.

4. The method of claim 3 , wherein the features further comprise environmental features that characterize an environment of the object.

5. The method according to any one of claims 1 to 4, wherein the encoder (11), the graph neural network (12) and the decoder (13) are trained jointly end-to-end.

6. The method according to claim 1 , wherein the predicted mobility is lane changes of other traffic participants.

7. The method according to any one of claims 1 to 6, wherein a technical system (100) is controlled depending on the predicted mobility.

8. A computer program arranged to cause a computer to carry out the method according to any one of claims 1 to 7 together with all the steps thereof when said computer program is executed by said computer.

9. A machine-readable storage medium storing the computer program of claim 8.

10. A system configured to carry out the method of any one of claims 1 to 7.

Images