Apparatus and method for driving assistance or autonomous driving function

Abstract

A computer-implemented method (20) of predicting the dynamics of an object in the surroundings of a vehicle (100). The method begins with the following steps: receiving (S21) a first data set characterizing the dynamics of the object in each case. Each of the first data sets is then propagated (S22) by an encoder (11) outputting a potential representation for each of the first data sets. A graph based on the potential representation is then generated (S23). The graph is then propagated through a graph neural network (12) outputting the updated graph. Based on the updated graph, the decoder (13) outputs the predicted dynamics of the selected object for the subsequent time step.

Description

Technical Field

[0001] This invention compromises AI-based model architectures for driver assistance and autonomous driving functions. The goal is to implement deep learning models in predictive driving tasks for safe and comfortable natural driving. Background Art

[0002] Driving scenarios are highly dynamic environments with a permanently changing number of traffic participants. Any movement of participants requires accurate identification and prediction of the intentions of other participants around them. For example, if the ego vehicle recognizes that a vehicle from an adjacent lane is cutting in, the ego vehicle can decide to brake for safety. In this setting, the ability to accurately model the interactions of participants is critical in situations where the prediction at one time step affects the behavior in the next time step. Furthermore, for use in driver assistance and autonomous driving system functionalities, predictions are required to be made early enough before the behavior occurs.

[0003] A simple generative model of the driving task can be constructed as a dynamic Bayesian network, in particular as a hidden Markov model, where the driving task is modeled as a hidden state that emits the lateral position and velocity at each time step.

[0004] The capabilities of such schemes are limited, since they do not explicitly model interactions and cannot capture all practical dependencies in complex real driving scenarios. Furthermore, the schemes are not able to learn from signal history, but rather act on what is sensed in the current time step. Summary of the invention

[0005] Advantages of the Invention In essence, it is proposed to replace the dynamic Bayesian network with a deep neural network model that can explicitly model the dependencies and interactions between traffic participants.

[0006] Deep neural network based models are able to learn complex dependencies in real driving scenarios from data. Deep neural network models excel in comparison with traditional models, especially when the complexity of interactions increases. This is due to the fact that deep neural network models are proficient in learning interactions, which is enhanced as they are exposed to more real-world driving data during training.

[0007] Another advantage gained by utilizing deep neural networks is the better use of signals extending further into the past. Therefore, more comprehensive information is available, based on which more precise predictions can be made.

[0008] In summary, the advantages of the present invention are obtained by a deep learning based model architecture for behavior prediction in a multi-agent setting by modeling their interactions. Thus, past driving data is taken into account and complex joint behaviors on multiple traffic participants can be learned.

[0009] Furthermore, the proposed invention provides a framework for developing data-efficient and robust AI-based multi-agent prediction models for safe and comfortable natural driving.

[0010] The proposed architecture can capture the dynamics of driving scenarios within a predefined time window. Therefore, the application can be used for L0-L3 driving assistance functions and L4-L5 autonomous driving functions.

[0011] The modular design of the architecture allows it to be adapted to multiple application scenarios and enables easy identification of the contribution of each component, thus simplifying architecture optimization.

[0012] In a first aspect, a computer-implemented method for predicting the dynamics of objects in the surroundings of a vehicle is presented. The dynamics of an object may be the behavior of the object, or preferably, the movement or trajectory of the object. The object may be a traffic participant like other motor vehicles (e.g., motorcycles or trucks) or pedestrians or cyclists. In the surroundings, the vicinity of the vehicle may be understood, in particular, the vicinity in which objects move and potentially may cross or become dangerously close to the ego trajectory of the vehicle.

[0013] The method begins with the following steps: receiving a first data set that characterizes the dynamics of the objects respectively. The first data set is a time series of features, wherein one of the features of each series characterizes the current dynamics and the other features of each series characterize the previous dynamics of the corresponding object. The current dynamics are the dynamics at the current time point, and the previous dynamics are the dynamics at the previous time point. It is worth noting that as one or more objects can appear or disappear in the surrounding environment of the vehicle, the number of objects (more precisely, the number of the first data set) can change over time. For example, the number of the first data set can be constant, or increase or decrease from one time point to its subsequent time point.

[0014] Next is the step of propagating each of the first data sets through an encoder configured to output a latent representation for each of the first data sets.

[0015] Next is the step of generating a graph based on the latent representation, wherein a node is the latent representation, in particular for each object, and the nodes are connected to each other in particular by edges. The step of generating the graph may be implemented by defining nodes and connecting each node with edges. In other words, the latent representation may be stored in a graph-based data format.

[0016] Next is the step of propagating the graph through a graph neural network that outputs an updated graph. The graph neural network has been trained to update the graph.

[0017] This is followed by the step of propagating the selected node of the updated graph through a decoder configured to output the predicted dynamics for at least the next subsequent time step of the current time step. Preferably, the decoder predicts the dynamics for a plurality of subsequent time steps. The method step of propagating may be carried out for a plurality or each of the updated graph nodes to predict the dynamics of the object corresponding to the node.

[0018] GNNs offer the following advantage: the number of objects can change over time. Therefore, the method is able to handle a time-varying number of different objects.

[0019] It is proposed that: each feature of the first data set is assigned to a time point among subsequent time points t=0, 1, 2, ... k. t=k may be the current time point for the current dynamics, and t=k+1 may be the immediately previous time point of the current time point for the previous dynamics. The features include the speed and position of the object at the corresponding time point, and preferably, the category of the object. The features may further include environmental features characterizing the environment of the corresponding object. The environmental features may be lane routes, traffic signs, and the like. It should be noted that the number of features may change (e.g., increase or decrease) over time.

[0020] Preferably, the encoder and the graph neural network and the decoder have been jointly trained end-to-end. For end-to-end training, a training dataset having a plurality of first datasets and assigned dynamics is provided, wherein parameters of the architecture are learned based on a loss function characterizing the difference between the output dynamics and the assigned dynamics.

[0021] Furthermore, it is proposed that a technical system is controlled according to the predicted dynamics. The technical system may be a robot or a vehicle. The predicted dynamics are used to provide actuator control signals for controlling actuators of the technical system. Preferably, the invention of the first aspect is utilized to control a vehicle, in particular for automatic cruise control or automatic lane change functions by predicting the driving behavior of traffic participants around the ego vehicle. Furthermore, the proposed invention may be used as a prediction component of a fully autonomous driving software architecture stack.

[0022] In a second aspect of the invention, a computer program and a machine readable storage according to the first aspect are proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. The drawings show: Figure 1 Schematic architecture; Figure 2 A flowchart of a method for utilizing the architecture; and Figure 3 Transportation. DETAILED DESCRIPTION

[0024] In driving scenarios, traffic participants need to understand the intentions of neighboring drivers to predict their future behavior. For example, predicting the next movement (overtaking, lane keeping, etc.) of all surrounding traffic participants enables drivers to make safe decisions related to their own movements. The actions of one driver at one time step affect the behavior of neighboring drivers in the next time step. Therefore, the ability to accurately model the interactions of participants is critical for autonomous driving. To this end, the following embodiments propose a data-efficient and robust AI-based multi-agent prediction model to achieve safe and comfortable natural driving.

[0025] according to Figure 1 The proposed AI-based model architecture 10 is a multi-agent deep neural network model consisting of three main modules, an encoder 11, an interaction module 12, and a decoder 13. The model in the proposed architecture can be trained by real driving data collected using a selected sensor setup. It should be noted that data pre-processing and preferably post-processing are performed to obtain the required information of objects in the surrounding environment of the sensor setup, considering the provided sensor signals of the selected sensor setup.

[0026] The data required to train the model may include a set of traffic participant characteristics (e.g., their estimated speed, acceleration, coordinate position) and a set of static environment characteristics (e.g., coordinate position of lane separators, lane existence probability, road curvature).

[0027] Given a set of available features as input, the proposed architecture 10 encodes the input feature vector into an efficient latent representation, uses a graph representation of the encoded latent representation as input to a graph neural network to learn interactions, and uses a recurrent neural network based decoder 13 to transform the learned interactions into a sequence of future predictions.

[0028] In a preferred embodiment of the model architecture 10, it comprises an encoder 11, an interaction module 12 and a decoder 13, wherein the encoder 11 is based on a temporal convolutional network (TCN). It can be a general 1-d convolutional network that modifies the classic 2-d convolutional neural network (CNN) for time series prediction. The encoder 11 maps the input history into a latent space, and the mapping is done independently for each agent. Therefore, for a given number of agents, the same number of latent representations is obtained. Preferably, each latent representation is assigned to its corresponding agent, for example by indexing.

[0029] The interaction module 12 models the interaction between any number of traffic participants and is preferably a graph neural network (GNN). It is known that the basic function of a GNN is to propagate information along a predefined graph to learn a meaningful representation. That is, given the encoded input vector for traffic agents, the GNN learns a new embedding for each agent by treating each agent as a node in the graph, and information exchange between agents occurs by sending messages along the edges of the predefined graph. As a GNN architecture, a graph attention network is a preferred choice because it can automatically learn which neighboring agents to pay attention to.

[0030] The decoder module 13 may be composed of a feed-forward network, preferably a recurrent neural network (RNN), which transforms the output of the interaction module into a sequence of future predictions. The decoder maps the latent state to a sequence of target predictions. RNNs are a class of feed-forward neural networks that can model such sequence data. The premise of RNNs is that they can learn the actual dependencies between past time steps to predict the future.

[0031] The proposed three-component model architecture 10 is trainable with respect to driving data collected in an end-to-end manner using various sensor settings. More precisely, one or more driving scenarios with multiple objects / agents are recorded within a given time span. For a predefined time point in the time span, a feature x for each agent / object is determined. These features are used as training data. A short sequence of subsequent features (e.g., up to 5 seconds for each object at a temporally subsequent time point) is used as a training input for the architecture 10, and the selected features immediately following the training input features (e.g., direction, speed) that characterize the behavior are used as labels for the output of the decoder 13. The content of the future prediction of the decoder module 13 can represent the same content as the content of the input, or can represent different content. For example, the prediction can be a category (e.g., the surrounding vehicles are keeping their current lanes or changing lanes), or the prediction can be the estimated coordinates of the surrounding vehicles. Preferably, the prediction for each of the objects is the category of "lane keeping" or "lane changing" for the corresponding object. Via supervised machine learning, the parameters of the three modules 11, 12, 13 of the architecture 10 are optimized to mimic the labels of the real data for the corresponding training inputs.

[0032] Figure 2 A flow chart 20 of a method for predicting the dynamics of objects in the surroundings of a vehicle 100 using a trained architecture 10 is shown.

[0033] The method starts with the following steps: receiving S21 a first data set that characterizes the dynamics of the object, wherein the first data set is a time series of features The subscript of feature x represents the corresponding object in the scene (e.g., vehicle). represents a time series, so the i-th object itself The features of an object include several time series data points (e.g., t=0...k). A time series data point can be defined by a superscript representing the time point t, where the time can start from t=0 to the current time t=k. For example, the feature of the i-th object Including the i-th object The current characteristics (x i k ) and previous features (x i k-1 , ..., x i 0 ). In other words, for each object i in the scene, the input to the encoder 11 is a series of features

[0034] Then, a propagation step S22 is performed. In this step, each of the first data sets is propagated through the encoder 11 which outputs a latent representation (v1, v2, v3) for each of the first data sets. In other words, a predefined number of features from the history are used to generate a vector v at time step k.

[0035] Then, a generation step S23 is performed. In this step, a graph is generated based on the potential representations (v1, v2, v3), where the nodes are the potential representations and the nodes are connected by edges.

[0036] Then, a propagation step S24 is performed. In this step, the graph is propagated by outputting the updated graph neural network 12. Preferably, step S24 is performed for L iterations until convergence.

[0037] Then, a propagation step (S25) is performed. In this step, the selected node of the updated graph is propagated through the decoder 13, which is configured to output the predicted dynamics for subsequent time steps of the object assigned to the selected node.

[0038] exist Figure 3 1 shows an embodiment of an actuator with a control system 40. The actuator and its environment will be collectively referred to as the actuator system. At preferably evenly spaced distances, the sensor 30 senses the condition of the actuator system. The sensor 30 may include several sensors. Preferably, the sensor 30 is an optical sensor that takes an image of the environment. The output signal S of the sensor 30 (or the output signal S for each of the sensors in the case where the sensor 30 includes multiple sensors) encoding the sensed condition is transmitted to the control system 40.

[0039] Thus, the control system 40 receives a stream of sensor signals S. It then calculates a series of actuator control commands A according to the stream of sensor signals S, which are then transmitted to the actuator unit, which converts the control commands A into a mechanical movement or change in a physical quantity. For example, the actuator unit can convert the control command A into an electrical, hydraulic, pneumatic, thermal, magnetic and / or mechanical movement or change. Specific, but non-limiting, examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servomechanisms, solenoids, stepper motors, etc.

[0040] The control system 40 receives a stream of sensor signals S from the sensor 30 in an optional receiving unit 50. The receiving unit 50 transforms the sensor signals S into input features x. The input features x are then passed on to the module 60, which contains Figure 1 Trained architecture 10.

[0041] The module 60 determines the prediction output by the trained architecture 10 and determined according to the prediction of the output signal y. The output signal y may be a control signal for controlling the actuator. Alternatively, the output signal y is transmitted to a conversion unit 80, which converts the output signal y into a control command A. The actuator control command A is then transmitted to the actuator unit for controlling the actuator unit accordingly. Alternatively, the output signal y may be directly regarded as the control command A.

[0042] The actuator unit receives the actuator control command A, is controlled accordingly and performs an action corresponding to the actuator control command A. The actuator unit may comprise a control logic which transforms the actuator control command A into a further control command which is then used to control the actuator.

[0043] In further embodiments, the control system 40 may include the sensor 30. In even further embodiments, the control system 40 may alternatively or additionally include an actuator.

[0044] In one embodiment, the module 60 can be designed to recognize lane changes of traffic participants. Likewise, when the movement of pedestrians and / or vehicles is recognized, the predicted future behavior of the pedestrians and / or vehicles is estimated, and based on the estimated future behavior, a trajectory can then be selected to avoid a collision with the pedestrians and / or vehicles, and the actuator control command A can be calculated to steer the vehicle in order to execute the trajectory.

[0045] In still further embodiments, it is envisioned that the control system 40 controls the display 10a instead of the actuator, wherein the display 10a can display control commands, etc. In other embodiments, the display 10a can be an output interface to a presentation device, such as a display, a light source, a speaker, a vibration motor, etc., which can be used to generate a sensory perceptible output signal that can be generated based on the output of the module 60.

[0046] Furthermore, the control system 40 may include a processor 45 (or multiple processors) and at least one machine-readable storage medium 46 storing thereon instructions that, when implemented, cause the control system 40 to implement a method according to an aspect of the present invention.

[0047] exist Figure 3 In a preferred embodiment of the invention, the control system 40 is used to control an actuator, which is an at least partially autonomous robot, such as an at least partially autonomous vehicle 100 .

[0048] The sensors 30 may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more position sensors (such as, for example, GPS). Some or all of these sensors are preferably, but not necessarily, integrated in the vehicle 100.

[0049] Alternatively or additionally, the sensor 30 may comprise an information system for determining the state of the actuator system. One example of such an information system is a weather information system that determines the current or future state of the weather in the environment.

[0050] The actuator unit, preferably integrated in the vehicle 100, may be given by the brakes, propulsion system, engine, drive train or steering device of the vehicle 100. The actuator control command A may be determined such that the actuator (or actuators) unit is controlled such that the vehicle 100 avoids a collision with said detected object. The detected objects may also be classified according to what the classifier 60 considers them to be most likely (e.g., pedestrians or trees), and the actuator control command A may be determined according to the classification.

[0051] In a further embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. Among other things, the mobile robot may be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above embodiments, the actuator command control A may be determined such that a propulsion unit and / or a steering device and / or a brake of the mobile robot are controlled such that the mobile robot may avoid a collision with the identified object.

Claims

1. A computer-implemented method (20) for predicting the dynamics of an object in the surrounding environment of a vehicle (100), comprising the following steps: Receive (S21) a first data set that characterizes the dynamics of the object (i), wherein the first data set is a time series of features. One of the features of each time series characterizes the current dynamics (x i k ), and other features characterize the previous dynamics (x i k-1 , x i k-2 ); propagating (S22) each of the first data sets through an encoder (11), the encoder (11) being configured to output a latent representation (v1, v2, v3) for each of the first data sets; generating (S23) a graph based on the latent representations (v1, v2, v3), wherein nodes are the latent representations and the nodes are connected by edges; Propagating (S24) the graph by outputting a graph neural network (12) of the updated graph; At least selected nodes of the updated graph are propagated (S25) through a decoder (13) configured to output the predicted dynamics.

2. The method according to claim 1, wherein the encoder (11) is a neural network, in particular a temporal convolutional network or a transformer, wherein the graph neural network is a graph attention network, and wherein the decoder (13) is a neural network, in particular a recurrent neural network or an LSTM or a transformer.

3. A method according to claim 1 or 2, wherein each feature of the first data set is assigned to a time point in subsequent time points (t=0, 1, 2, ... k), wherein the feature of the first data set at least characterizes the speed and position of the object at the corresponding time point, in particular the category of the object. The method of claim 3 , wherein the features further include environmental features characterizing an environment of the object.

5. The method according to any one of claims 1 to 4, wherein the encoder (11) and the graph neural network (12) and the decoder (13) have been jointly trained end-to-end. 6 . The method according to claim 1 , wherein the predicted dynamic is a lane change by another road user.

7. The method according to any of the preceding claims, wherein the technical system (100) is controlled in accordance with the predicted dynamics.

8. A computer program configured to cause a computer to carry out the method according to any one of claims 1 to 7 with all its steps, if the computer program is executed by the computer.

9. A machine-readable storage medium on which the computer program according to claim 8 is stored.

10. A system configured to implement the method according to any one of claims 1 to 7.

Images