Method and apparatus for determining a vehicle comfort metric for an autonomous vehicle
By using an attention mechanism to predict the lane change behavior of the target vehicle in autonomous vehicles, the lane change prediction problem in autonomous driving is solved, the accuracy and interpretability of prediction are improved, and driving safety is enhanced.
Patent Information
- Application Number
- CN201980035107.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-08
- Filing Date
- 2019-05-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-05-28
AI Technical Summary
In autonomous driving, the prior art is difficult to accurately predict the lane change behavior of the target vehicle, which affects the traffic situation adjustment and driving safety of the autonomous vehicle.
Using a machine learning network with attention mechanism, the driving maneuvering behavior of the target vehicle is predicted, including lane change, by obtaining the environmental characteristics of the target vehicle and using the attention layer selectively weighted input characteristics of the recurrent neural network, the driving maneuvering behavior of the target vehicle is predicted, including lane change, and thus determining vehicle comfort. Measure to adjust driving strategies.
Improves the accuracy and interpretability of lane change predictions, allowing autonomous vehicles to adjust their driving strategies earlier and smoother, reducing driving discomfort and accident risks.
Smart Images

Figure CN112203916B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a method and apparatus for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle.
[0002] Some embodiments relate to a method and apparatus for determining information related to a lane change of a target vehicle, and to a program storage device, and more particularly, but not exclusively, to predicting a lane change of a target vehicle using a machine learning network with an attention mechanism. Background Art
[0003] Autonomous driving is a major focus in automotive research. In autonomous driving, predicting how other vehicles are going to behave is a key feature. Based on these predictions, the autonomous vehicle can determine what traffic situation is forming around the autonomous vehicle, and the autonomous vehicle can be allowed to adjust its driving to the changing traffic situation. One important prediction is predicting lane changes of other vehicles. If a target vehicle in front of the autonomous vehicle changes lanes on a multi-lane road, such as going from a ramp onto a highway or changing lanes along multiple lanes of a highway, there is a high probability that the autonomous vehicle will need to adjust its speed and direction of travel, either because it is blocked by the vehicle that is changing lanes, or because it causes the other vehicle to change lanes. Summary of the invention
[0004] There may be a need to provide a method and apparatus for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle.
[0005] The present application provides a method for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle, the method comprising:
[0006] Obtaining a start time of a driving maneuver of a target vehicle in an environment of the autonomous vehicle;
[0007] Inputting a plurality of input feature groups of the target vehicle into corresponding plurality of long short-term memory network (LSTM) units of the recurrent neural network, the input feature groups corresponding to two or more of the following elements: driving parameters of the target vehicle, other vehicles in the vicinity of the target vehicle, or a static environment of the target vehicle;
[0008] selectively weighting the input feature set using an attention layer of a recurrent neural network;
[0009] obtaining, as an output from the recurrent neural network, a predicted start time of a predicted driving maneuver of the target vehicle; and
[0010] A vehicle comfort metric for an autonomous vehicle is determined for predicting a driving maneuver by comparing a start time of the driving maneuver to a predicted start time of the driving maneuver output from a recurrent neural network to determine a delay between the start time of the driving maneuver and the predicted start time of the driving maneuver, wherein a larger delay is associated with reduced vehicle comfort.
[0011] In some implementations, the method further includes obtaining a first duration of a driving maneuver of the target vehicle, wherein the vehicle comfort metric is further determined based on an overlap between the first duration of the driving maneuver and a predicted second duration of the driving maneuver.
[0012] In some embodiments, the vehicle comfort metric is further determined based on a plurality of discrete predictions within a duration of the driving maneuver.
[0013] In some embodiments, the vehicle comfort metric is further determined based on a prediction of a missed driving maneuver.
[0014] The present application also provides an apparatus for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle, the apparatus comprising:
[0015] At least one interface, the at least one interface configured to:
[0016] - obtaining an extent of a driving maneuver of a target vehicle in an environment of the autonomous vehicle, the extent corresponding to one or more of the following elements: a first duration of the driving maneuver of the target vehicle, or a start time of the driving maneuver of the target vehicle;
[0017] - inputting a plurality of input feature groups of the target vehicle into corresponding plurality of long short-term memory network (LSTM) units of the recurrent neural network, wherein the input feature groups correspond to two or more of the following elements: driving parameters of the target vehicle, other vehicles in the vicinity of the target vehicle, and a static environment of the target vehicle;
[0018] - selectively weighting the input feature set using an attention layer of a recurrent neural network;
[0019] - obtaining, as an output from the recurrent neural network, a prediction of a driving maneuver of the target vehicle;
[0020] and a computing module configured to:
[0021] - determining a vehicle comfort metric for the autonomous vehicle based on an overlap between a first duration of the driving maneuver and a predicted second duration of the driving maneuver for the prediction of the driving maneuver output from the recurrent neural network, wherein a greater overlap is associated with increased vehicle comfort.
[0022] In some embodiments, the vehicle comfort metric is further determined based on a delay between a start time of the driving maneuver and a predicted start time of the driving maneuver.
[0023] In some embodiments, the vehicle comfort metric is further determined based on a plurality of discrete predictions within a first duration of the driving maneuver.
[0024] In some embodiments, the vehicle comfort metric is further determined based on a prediction of a missed driving maneuver.
[0025] There may be a need to provide an improved scheme for predicting lane changes of a target vehicle.
[0026] At least some embodiments relate to a method for a vehicle, the method being adapted to determine information related to a lane change of a target vehicle, wherein the information related to the lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change, the method comprising:
[0027] Acquiring information related to an environment of the target vehicle, the information related to the environment relating to a plurality of features of the environment of the target vehicle, the plurality of features being divided into two or more feature groups;
[0028] determining two or more weighting factors for the two or more feature groups, wherein an attention mechanism is used to determine the two or more weighting factors, the attention mechanism being a mechanism for determining selective attention to a subset of features among the plurality of features, the attention mechanism employing at least one of “paying attention to previous time steps” and “paying attention to different feature groups”; and
[0029] Determine information related to a lane change of a target vehicle based on information related to an environment of the target vehicle using a machine learning network, wherein weighting of the plurality of features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
[0030] The method may be used to predict lane changes of a target vehicle utilizing a machine learning network. Within the machine learning network, an attention mechanism, such as an attention layer, is used to selectively focus on input features of the network that are deemed to have increased importance. For example, if the target vehicle approaches a road ramp, an increased focus may be set on mapped data input features to the machine learning network; in dense traffic situations, the increased focus may be on vehicles (directly) surrounding the target vehicle. Additionally, in at least some embodiments, the machine learning network may be a recurrent machine learning network, such as a recurrent neural network, such that lane changes may be predicted based not only on current input features of the machine learning network, but also on previous states of the machine learning network. To this effect, the machine learning network may, for example, include a long short-term memory network (LSTM) unit.
[0031] Some embodiments provide a method for a vehicle. The method is suitable for determining information related to a lane change of a target vehicle. The information related to the lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change. The method includes: obtaining information related to the environment of the target vehicle. The information related to the environment involves multiple features of the environment of the target vehicle. The multiple features are divided into two or more feature groups. The method also includes: determining two or more weighting factors for the two or more feature groups. An attention mechanism is used to determine the two or more weighting factors. The method also includes: using a machine learning network to determine the information related to the lane change of the target vehicle based on the information related to the environment of the target vehicle. The weighting of the multiple features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
[0032] Utilizing a machine learning network with an attention mechanism may not only allow for better quality of lane change predictions, but also allow for better interpretability of the predictions, since the evaluation of the predictions may show which input features have an increased impact.
[0033] For example, the two or more feature groups include at least two elements among a group of features related to driving parameters of the target vehicle, a group of features related to one or more other vehicles near the target vehicle, and a group of features related to the static environment of the target vehicle. Grouping the input features into multiple feature groups can allow the machine learning network to be separated according to the grouping, for example, using two or more LSTMs for the two or more feature groups.
[0034] In at least some embodiments, information related to the environment of the target vehicle is based on processed sensor data of the vehicle and / or based on mapping data. For example, features of a feature group related to driving parameters of the target vehicle can be based on processed sensor data of the vehicle. Features of a feature group related to one or more other vehicles in the vicinity of the target vehicle can be based on processed sensor data of the vehicle. Features of a feature group related to the static environment of the target vehicle can be based on mapping data. Utilizing input features from different sources, such as processed sensor data and mapping data of the vehicle, can improve the quality of the prediction and can allow selective focus among the input features.
[0035] In some embodiments, the machine learning network includes an attention layer. The attention layer can be used to determine the two or more weighting factors. The attention layer can be used to selectively focus on an input feature or input feature group of the machine learning network.
[0036] For example, the attention layer may be configured to determine the two or more weighting factors based on features of the two or more feature groups. Additionally or alternatively, the attention layer may be configured to determine the two or more weighting factors based on current information related to the environment and based on a previous state of the attention layer. Utilizing features that are in different spaces and / or have different morphologies may allow for improved selection of a point of interest. Furthermore, if one or more weighting factors are determined based on current information related to the environment and based on a previous state of the attention layer, the continuous nature of the lane change prediction is taken into account within the determination of the one or more weighting factors.
[0037] In at least some embodiments, the method includes: repeatedly determining information related to a lane change of the target vehicle. For each repeated determination of information related to a lane change of the target vehicle, the two or more weighting factors may be re-determined. This may allow the weighting conditions to be adjusted for the driving conditions of the target vehicle.
[0038] In some embodiments, the machine learning network is a recurrent machine learning network. The machine learning network can be configured to determine information related to a lane change of the target vehicle based on current information related to the environment of the target vehicle and based on a previous state of the machine learning network. This can take into account the continuous nature of lane change prediction.
[0039] For example, a machine learning network may include two or more long short-term memory network units. Information related to the lane change of the target vehicle may be determined using the two or more long short-term memory network units based on current information related to the environment of the target vehicle and based on the previous state of one or more long short-term memory network units. This may take into account the continuous nature of lane change predictions, as the LSTMs are based on their current input features as well as their previous states. Additionally or alternatively, each of the two or more long short-term memory network units may be equipped with a feature group among the two or more feature groups. This may facilitate weighting the two or more feature groups through an attention mechanism within the machine learning network.
[0040] In some embodiments, the method includes guiding the vehicle based on information related to the lane change of the target vehicle. For example, the vehicle can be guided by accelerating or decelerating the vehicle and / or by steering the vehicle. This can allow for reacting to the predicted lane change of the target vehicle.
[0041] For example, information related to a lane change of a target vehicle may be determined to predict the lane change of the target vehicle. The predicted lane change may be used to change the speed or heading of the vehicle.
[0042] The method includes determining information related to lane changes of target vehicles for a plurality of target vehicles in the vicinity of the vehicle. This may allow lane changes of a plurality of vehicles in the vicinity of the vehicle to be predicted.
[0043] Some embodiments also provide a method for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle. The method includes: obtaining information related to the extent of the driving maneuver of the target vehicle. The method also includes: obtaining information related to the prediction of the driving maneuver of the target vehicle. The method also includes: determining a vehicle comfort metric for predicting the driving maneuver based on a comparison of the information related to the extent of the driving maneuver and the information related to the prediction of the driving maneuver. The comfort metric is determined based on one or more elements of the following group: a delay between the start of the driving maneuver and the first prediction of the driving maneuver, an overlap between the driving maneuver and the prediction of the driving maneuver, multiple discontinuous predictions of the driving maneuver, and a missed prediction of the driving maneuver. The vehicle comfort metric can be used to evaluate and improve a prediction method for predicting the maneuver of the target vehicle.
[0044] Some embodiments further provide a computer program having a program code, wherein the computer program is configured to implement at least one of the methods described when the computer program is executed on a computer, a processor or programmable hardware.
[0045] Some embodiments also provide a device for a vehicle. The device is adapted to determine information related to a lane change of a target vehicle. The information related to the lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change. The device includes at least one interface for obtaining information related to an environment of the target vehicle. The information related to the environment relates to a plurality of features of the environment of the target vehicle. The plurality of features are divided into two or more feature groups. The device includes a computing module configured to determine two or more weighting factors for the two or more feature groups. An attention mechanism is used to determine the two or more weighting factors, the attention mechanism being a mechanism for determining selective attention to a subset of features among the plurality of features, the attention mechanism employing at least one of "paying attention to a previous time step" and "paying attention to a different feature group". The computing module is configured to determine information related to the lane change of the target vehicle based on the information related to the environment of the target vehicle using a machine learning network. The weighting of the plurality of features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
[0046] Some embodiments also provide a device for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle. The device includes at least one interface for obtaining information related to the extent of the driving maneuver of the target vehicle and for obtaining information related to the prediction of the driving maneuver of the target vehicle. The device also includes a computing module configured to determine a vehicle comfort metric for predicting the driving maneuver based on a comparison of the information related to the extent of the driving maneuver and the information related to the prediction of the driving maneuver. The comfort metric is determined based on one or more elements of the group consisting of: a delay between the start of the driving maneuver and the first prediction of the driving maneuver, an overlap between the driving maneuver and the prediction of the driving maneuver, multiple discontinuous predictions of the driving maneuver, and a missed prediction of the driving maneuver. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Some examples of the apparatus and / or method are described below by way of example only and with reference to the accompanying drawings, in which:
[0048] Figure 1a and 1b A flow chart showing an embodiment of a method for a vehicle;
[0049] Figure 1c A block diagram showing an embodiment of an apparatus for a vehicle;
[0050] Figure 1d A flow chart showing an embodiment of a method for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle;
[0051] Figure 1e A block diagram showing an embodiment of an apparatus for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle;
[0052] Figure 2a Block diagram showing a distributed LSTM network with an attention layer.
[0053] Figure 2b Schematic diagram showing a prediction of a lane change and a corresponding evaluation based on comfort-related metrics;
[0054] Figure 3a Sample diagram showing how attention can understand context;
[0055] Figure 3b A diagram showing a sample graph from fleet data;
[0056] Figure 3c Visualizations showing dynamic environment features;
[0057] Figure 3d Visualization showing a long short-term memory network;
[0058] Figure 3e Example showing attention calculation;
[0059] Figure 3f An overview of a dataset showing a highway (US-101) with five lanes;
[0060] Figure 3g A sequence of events showing a lane change maneuver in progress;
[0061] Figure 3h A time series showing a lane change maneuver and a corresponding prediction of the lane change maneuver;
[0062] Figure 3i a table showing the markings of the lane change maneuver performed by the assessee;
[0063] Figure 4 Shows comparison and ranking of different prediction methods on different metrics;
[0064] Figures 5a to 5e Show the influence of attention on the network’s decision making, highlighting its correct and intuitive contributions;
[0065] Figure 6a-1 and 6a-2 Displaying a visualization of a lane change to the right as recorded by a series of cars; and
[0066] Figure 6b-1 , 6b-2and 6b-3 show the visualization of a “fake” left lane change. DETAILED DESCRIPTION
[0067] Some examples will now be described more fully with reference to the accompanying drawings, in which some examples are described. In the drawings, the thickness of lines, layers and / or regions may be exaggerated for clarity.
[0068] Accordingly, other examples can be different modifications and alternative forms, and some specific examples of them are shown in the accompanying drawings and are subsequently specifically described. However, this specific description does not limit other examples to the specific forms described. Other examples can cover all variations, equivalent forms and alternative forms that fall within the scope of the disclosure. The same or similar numbers represent the same or similar elements throughout the description of the accompanying drawings, and they can be implemented in the same or modified form when compared to each other, but provide the same or similar functionality.
[0069] It is understood that if an element is represented as being "connected" or "coupled" to another element, these elements may be directly connected or coupled, or connected or coupled via one or more intermediate elements. If two elements A and B are combined using "or", this is to be understood as disclosing all possible combinations, i.e., only A, only B, and A and B, unless otherwise defined explicitly or implicitly. Alternative expressions for these same combinations are "at least one of A and B" or "A and / or B". The same provisions apply to combinations of more than two elements.
[0070] The terms used herein to illustrate the purpose of specific examples are not intended to be limiting for other examples. Whenever singular forms such as "one" and "the one" are used and only a single element is not explicitly or implicitly defined as mandatory, other examples may also use plural elements to implement the same functionality. Similarly, if a functionality is subsequently described as being implemented using multiple elements, other examples may implement the same functionality using a single element or processing entity. It is further understood that the terms "include", "comprising" indicate the presence of stated features, integers, steps, operations, processes, actions, elements and / or components when used, without excluding the presence or addition of one or more other features, integers, steps, operations, processes, actions, elements, components and / or any of their sets.
[0071] Unless otherwise defined, herein, all terms (including technical and scientific terms) are used according to their common meanings in the art to which the examples belong.
[0072] Figure 1a and 1bA flow chart showing an embodiment of a method for a vehicle 100 is shown. The method is suitable for determining information related to a lane change of a target vehicle. The information related to the lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change. The method includes: obtaining 110 information related to an environment of the target vehicle. The information related to the environment relates to multiple features of the environment of the target vehicle. The multiple features are divided into two or more feature groups. The method also includes: determining 120 two or more weighting factors for the two or more feature groups. An attention mechanism is used to determine the two or more weighting factors. The method also includes: determining 130 information related to the lane change of the target vehicle based on the information related to the environment of the target vehicle using a machine learning network. The weighting of the multiple features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
[0073] Figure 1c A block diagram showing an embodiment of a (corresponding) device 10 for a vehicle 100 is shown. The device 10 is suitable for determining information related to a lane change of a target vehicle. The information related to the lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change. The device 10 includes at least one interface 12 for obtaining information related to the environment of the target vehicle. The information related to the environment relates to multiple features of the environment of the target vehicle. The multiple features are divided into two or more feature groups. The device 10 includes a computing module 14 for determining two or more weighting factors for the two or more feature groups. An attention mechanism is used to determine the two or more weighting factors. The computing module 14 is configured to determine information related to the lane change of the target vehicle based on information related to the environment of the target vehicle using a machine learning network. The weighting of the multiple features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups. The at least one interface 12 is connected to the computing module 14. Figure 1c The vehicle 100 is also shown to include the device 10 .
[0074] For example, combined with Figure 1a and 1b The method features of the described method can be executed by the vehicle 100. The computing module 14 can be configured to execute the combined Figure 1a and / or method features / steps described in 1 b, such as the calculation module being combined with the at least one interface 12 .
[0075] The following instructions not only cover Figure 1a and / or 1b, and involves Figure 1c Device 10.
[0076] Some embodiments of the method and / or device are suitable for determining information related to a lane change of a target vehicle. In other words, the method and / or device can be used to determine whether a target vehicle (which can be a vehicle in front of the vehicle 100) is likely to perform a lane change, such as whether it is possible to move from a first lane to a second lane of the same highway. For example, information related to the lane change of the target vehicle can be determined to predict the lane change of the target vehicle. The information related to the lane change of the target vehicle may include: information related to the possibility of the target vehicle changing lanes at a predetermined time point or time stage and / or information related to the predicted start and / or end of the lane change maneuver. In other words, the information related to the lane change of the target vehicle may include the following information related to the lane change, which can be used by the autonomous driving module or semi-autonomous driving module of the vehicle to adjust the vehicle driving. For example, the method may include: providing information related to the lane change of the target vehicle to a semi-autonomous or autonomous driving module of the vehicle 100, such as via the at least one interface 12.
[0077] Some embodiments of the method and / or apparatus are suitable for determining information related to a lane change of a target vehicle. For example, the target vehicle is a vehicle on the same road as the vehicle 100, such as a vehicle in front of the vehicle 100. The vehicle 100 may determine information related to a lane change of a plurality of vehicles located near the vehicle 100, such as for predicting a lane change of the plurality of vehicles located near the vehicle 100. For example, the method may include: determining 160 information related to a lane change of a target vehicle of a plurality of target vehicles located near the vehicle 100, such as for each of the plurality of target vehicles respectively. The method may include sequentially determining 160 information related to a lane change of the plurality of target vehicles located near the vehicle 100. For example, the plurality of (target) vehicles may be located on the same highway as the vehicle 100, such as including adjacent road structures such as highway ramps. For example, the target vehicle may be a vehicle among a plurality of vehicles located near the vehicle 100.
[0078] The method comprises: acquiring 110 information related to the environment of the target vehicle. The information related to the environment relates to a plurality of features of the environment of the target vehicle. The plurality of features are divided into two or more feature groups. For example, the two or more feature groups may include a feature group related to the driving parameters of the target vehicle, a feature group related to one or more other vehicles near the target vehicle, and / or a feature group related to the static environment of the target vehicle. The feature group related to the driving parameters of the target vehicle may include the following group: information related to the longitudinal (e.g., in the direction of travel) speed of the target vehicle, information related to the lateral (e.g., orthogonal to the direction of travel) speed of the target vehicle, information related to the longitudinal acceleration of the target vehicle, information related to the lateral acceleration of the target vehicle, and at least one element of information related to the forward angle of the target vehicle. The feature group associated with one or more other vehicles near the target vehicle may include the following group: the (temporary) distance of the vehicle driving in front of the target vehicle in the same lane, the (temporary) distance of the vehicle driving in front of the target vehicle in the adjacent lane (left or right), the (temporary) distance of the vehicle driving behind the target vehicle in the same lane, and the (temporary) distance of the vehicle driving behind the target vehicle in the adjacent lane (left or right). One or more elements. The temporary distance can be the distance between the vehicles divided by the speed of the tracking vehicle. The feature group associated with the static environment of the target vehicle may include one or more features associated with the static environment of the target vehicle, such as the features of the road on which the target vehicle is traveling. For example, the feature group associated with the static environment of the target vehicle may include information related to the distance of the target vehicle from the ramp, information related to the number of lanes of the road on which the target vehicle is traveling, and / or information related to the change in the number of lanes of the road on which the target vehicle is traveling. The term "feature" used herein relates to input data or input data sets for machine learning networks. For example, a feature can be a specific measurable attribute or characteristic of the phenomenon being observed, such as an attribute or characteristic of a driving parameter of the target vehicle, an attribute or characteristic of one or more other vehicles in the vicinity of the target vehicle, or an attribute or characteristic of the static environment of the target vehicle.
[0079] In at least some embodiments, the information related to the environment of the target vehicle is based on processed sensor data of the vehicle 100 and / or based on mapping data. For example, the processed sensor data may not be "raw" sensor data; but rather sensor data in which features are separated, for example, to facilitate processing by a machine learning network. The processed sensor data may be processed perception sensor data. The vehicle 100 may include one or more perception sensors, such as at least one LIDAR (laser radar) sensor, radar sensor, time of flight (TOF) sensor, and camera sensor, which may be configured to provide sensor data. The vehicle may also include one or more computing modules configured to process sensor data of the LIDAR / radar / TOF / camera sensors to provide processed sensor data. The method may include: obtaining processed sensor data from the one or more computing modules. For example, features of a feature group related to driving parameters of the target vehicle may be based on processed sensor data of the vehicle 100. Features of a feature group related to one or more other vehicles in the vicinity of the target vehicle may be based on processed sensor data of the vehicle 100. Features of a feature group related to the static environment of the target vehicle may be based on mapping data, for example, based on map information.
[0080] The terms "previous" and "current" may be used hereinafter, for example to refer to "previous" or "current" information related to the environment of the target vehicle, a "previous" state of a machine learning network, a "previous" state of an attention layer, etc. The terms "previous" and "current" may relate to time steps at which the method is implemented. For example, a current time step may be a time step for which (current) information related to the environment of the target vehicle is used to determine information related to a lane change of the target vehicle, and a previous time step may be a time step that (directly) precedes the current time step. In at least some embodiments, as Figure 1b As shown, the method includes: repeating 140 to determine information related to the lane change of the target vehicle. For example, at t=0, the information related to the lane change may be determined for the first time. The information related to the environment of the target vehicle at t=0 may be current information related to the environment of the target vehicle, and there may be no previous information related to the environment of the target vehicle or no previous state of the machine learning network. At t=1, 140 to determine the information related to the lane change may be repeated. In this case, the information related to the environment of the target vehicle at t=1 may be current information related to the environment of the target vehicle, the information related to the environment of the target vehicle at t=0 may be previous information related to the environment of the target vehicle, and the state of the machine learning network at t=0 may be a previous state of the machine learning network.
[0081] The method includes determining 120 two or more weighting factors for the two or more feature groups. The weighting of the plurality of features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups. For example, the two or more weighting factors may represent how much weight is applied to a feature group among the two or more feature groups in determining information related to the lane change of the target vehicle. An attention mechanism is used to determine the two or more weighting factors. An attention mechanism may be a mechanism for determining selective attention to a subset of features among the plurality of input features. For example, an attention mechanism may be applicable to assigning weights to a feature group among the two or more feature groups. For example, the machine learning network may include an attention layer. The attention layer may be used to determine the two or more weighting factors. The attention layer may implement an attention mechanism. The attention layer may be trained to assign weights to a feature group among the two or more feature groups, for example using features of the two or more feature groups and / or a previous state of the attention layer as a training input. For example, a previous state of the attention layer may include previous information related to the environment and / or a previous weight implied by the name of the attention layer. The attention layer may be configured to determine the two or more weighting factors based on features of the two or more feature groups. The two or more feature groups may be input features for the attention layer. In at least one embodiment, the attention layer is configured to determine the two or more weighting factors based on current information related to the environment and based on a previous state of the attention layer (e.g., based on previous information related to the environment). Together with the two or more feature groups, the previous state of the attention layer may be an input feature of the attention layer. For each repeated determination of information related to the lane change of the target vehicle, the two or more weighting factors may be re-determined 142.
[0082] The method includes: using a machine learning network, based on information related to the environment of the target vehicle, determining 130 information related to the lane change of the target vehicle. For example, the information related to the environment of the target vehicle, such as the multiple features and / or the two or more feature groups, can be input features of the machine learning network. For example, the machine learning network can be a recurrent machine learning network, such as a recurrent neural network. In other words, the machine learning network can be configured to determine information related to the lane change of the target vehicle based on current information related to the environment of the target vehicle and based on a previous state of the machine learning network. For example, the previous state of the machine learning network can include previous information related to the environment of the target vehicle and / or previously determined information related to the lane change of the target vehicle.
[0083] In an exemplary embodiment, the machine learning network includes two or more long short-term memory network (LSTM) units for the two or more feature groups, for example, three LSTM units for three feature groups. Information related to the lane change of the target vehicle can be determined 130 using two or more long short-term memory network units based on current information related to the environment of the target vehicle and based on the previous state of one or more long short-term memory network units. Each of the two or more long short-term memory network units can be configured for a feature group among the two or more feature groups. For example, each LSTM unit can process (just) one feature group among the two or more feature groups as an input feature.
[0084] In some embodiments, as in Figure 1b As further shown in FIG. 1 , the method includes: directing 150 (e.g., autonomously or semi-autonomously driving) the vehicle 100 based on the information related to the lane change of the target vehicle. For example, the vehicle 100 may accelerate 152 or decelerate 154 based on the information related to the lane change of the target vehicle. Alternatively or additionally, the method may include: steering 156 the vehicle 100 based on the information related to the lane change of the target vehicle.
[0085] For example, vehicle 100 and / or target vehicle may be a land vehicle, a road vehicle, a sedan, an automobile, an off-road vehicle, a motor vehicle, a truck, or a van. In at least some embodiments, vehicle 100 and / or target vehicle may be an autonomously or semi-autonomously operated vehicle.
[0086] The at least one interface 12 (and / or as combined Figure 1e The at least one interface 22) introduced may correspond to one or more input terminals and / or output terminals for receiving and / or transmitting information, which may be in the form of digital (binary) values according to a specific code within a module, between modules or between modules of different entities.
[0087] In some embodiments, the computing module 14 (and / or as combined with Figure 1e The described computing module 24) can be implemented using one or more computing units, one or more computing devices, any device for computing, one or more processing units, one or more processing devices, any device for processing, such as a processor, a computer or a programmable hardware component that can be run with correspondingly adapted software. In other words, the functions of the described computing modules 14, 24 can also be implemented in software, which is then executed in one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc.
[0088] Further details and aspects of the method and / or device 10 may be combined with the proposed concept or one or more examples described above or below (e.g. Figure 1d The method and / or device 10 may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.
[0089] Figure 1d A flow chart of an embodiment of a method for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle is shown. The method comprises: obtaining 210 information related to an extent of a driving maneuver of the target vehicle. The method further comprises: obtaining 220 information related to a prediction of a driving maneuver of the target vehicle. The method further comprises: determining 230 a vehicle comfort metric for predicting a driving maneuver based on a comparison of the information related to the extent of the driving maneuver and the information related to the prediction of the driving maneuver. The comfort metric is determined based on one or more elements of the group consisting of: a delay between the start of the driving maneuver and a first prediction of the driving maneuver, an overlap between the driving maneuver and a prediction of the driving maneuver, a number of discontinuous predictions of the driving maneuver, and a prediction of a measure of the driving maneuver.
[0090] Figure 1c A block diagram of an embodiment of a (corresponding) device 20 for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle is shown. The device 20 includes at least one interface 22 for obtaining information related to the extent of the driving maneuver of the target vehicle and for obtaining information related to the prediction of the driving maneuver of the target vehicle. The device 20 includes a calculation module 24, which is configured to determine a vehicle comfort metric for predicting a driving maneuver based on a comparison of the information related to the extent of the driving maneuver and the information related to the prediction of the driving maneuver. The vehicle comfort metric is determined based on one or more elements of the following group: a delay between the start of the driving maneuver and the first prediction of the driving maneuver, an overlap between the driving maneuver and the prediction of the driving maneuver, the number of discontinuous predictions of the driving maneuver and the number of missed predictions of the driving maneuver. The at least one interface 22 is coupled to the calculation module 24. The calculation module 24 can be configured to perform Figure 1d Method features / steps of the method, for example in combination with the at least one interface 22 .
[0091] The following instructions involve Figure 1d Methods and Figure 1e Device 20.
[0092] The method and / or device is suitable for determining a vehicle comfort metric for predicting a driving maneuver of a target vehicle. Figures 1a to 1c As introduced, the driving maneuver may be or may include a lane change maneuver. For example, the driving maneuver may be an overtaking maneuver or a merging maneuver, which includes a lane change maneuver. The vehicle comfort metric may represent the comfort level allowed by the prediction of the driving maneuver of the target vehicle. For example, the vehicle comfort metric may be used to evaluate one or more driving maneuver prediction methods, such as for selecting a driving maneuver prediction method that allows improved comfort in a vehicle (e.g., not the target vehicle) utilizing the prediction method. In at least some embodiments, the method may be performed offline, such as outside the vehicle. The determination of the vehicle body comfort metric may be based on offline information related to the degree of the driving maneuver of the target vehicle (e.g., recorded or simulated information related to the degree of the driving maneuver of the target vehicle), and / or based on offline information related to the prediction of the driving maneuver of the target vehicle, information related to the prediction of the driving maneuver of the target vehicle simulated outside the vehicle. The target vehicle may be an "actual" target vehicle, e.g., a driving maneuver may be performed by an actual target vehicle and information related to the extent of the driving maneuver may be based on the driving maneuver as performed by the actual target vehicle; or the target vehicle may be a simulated target vehicle, e.g., a driving maneuver may be performed in a computer simulation. For example, information related to the extent of the driving maneuver of the simulated target vehicle may be based on the simulated driving maneuver. In at least some embodiments, the extent of the driving maneuver may be determined automatically, e.g., based on the lateral acceleration of the target vehicle or based on the lateral velocity of the target vehicle. Alternatively, the extent of the driving maneuver may be determined manually. Information related to the extent of the driving maneuver may include or represent the following group: one or more elements of the duration of the driving maneuver, the start time of the driving maneuver, and the end time of the driving maneuver.
[0093] The method comprises: obtaining 220 information related to the prediction of a driving maneuver of the target vehicle. For example, the information related to the prediction of the driving maneuver may include or represent the following group: one or more predicted durations of the driving maneuver (within an actual driving maneuver, the prediction may determine one or more predicted driving maneuvers) and one or more elements of the first predicted start of the driving maneuver (the first predicted driving maneuver of the one or more predicted driving maneuvers).
[0094] The method comprises determining 230 a comfort measure for predicting the driving maneuver based on a comparison of information related to the extent of the driving maneuver and information related to the prediction of the driving maneuver.
[0095] For example, the method may include evaluating a prediction method (eg, a method for determining information related to a lane change of a target vehicle) based on the vehicle comfort metric determined 230 .
[0096] The comfort metric may be determined based on a delay between the start of a driving maneuver and a first prediction of the driving maneuver, for example based on a delay between the start time of the driving maneuver and the start of the first prediction of the driving maneuver. The earlier the prediction, the more time the vehicle has to adjust its driving using the prediction. The comfort metric may be determined based on an overlap between a driving maneuver and a prediction of the driving maneuver, for example based on a comparison of the entire duration of the driving maneuver with a duration during which the driving maneuver has been predicted. The comfort metric may be determined based on the number of discontinuous predictions of the driving maneuver, for example based on how many predictions are made within a driving maneuver degree that results in gaps between individual predictions. The comfort metric may be determined based on a missed prediction of a driving maneuver, for example based on whether the driving maneuver is missed / not recognized at all.
[0097] Further details and aspects of the method and / or device 10 may be combined with the proposed concept or one or more examples described above or below (e.g. Figure 1d The method and / or device 10 may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.
[0098] At least some embodiments provide attention-based lane change prediction, which may be based on comfort-related metrics.
[0099] Prediction is an important factor in implementing autonomous driving. Lane change prediction is particularly important because lane changes are often performed at high speeds. In addition, lane changes are responsible for 18% of accidents.
[0100] In some systems, "classical" machine learning methods are used, such as so-called support vector machines or random forests, or simple recurrent neural networks are used. In these approaches, input features (such as speed, lateral acceleration, etc.) are concatenated and passed to the method.
[0101] There are a large number of evaluation criteria to evaluate the quality of lane change prediction in autonomous driving. The evaluation criteria can be facilitated by information collection so that the evaluation criteria for each time step can be determined independently of other time steps.
[0102] Unfortunately, in many cases, the lane change maneuver cannot be considered as a single time step, but rather as a continuous process consisting of multiple time steps.
[0103] If this property is discarded, important information is lost, which is information relevant to driving comfort. For example, the prediction may be discontinuous, which may lead to multiple braking and acceleration maneuvers in one lane change maneuver. Another scenario where driving comfort is affected is one in which the prediction is made at a late point in time, which may lead to abrupt acceleration or deceleration. Evaluation criteria used in other systems may not take these factors into account.
[0104] Additionally, in some systems, all factors may be considered simultaneously, which may lead to errors in extreme cases. Driving comfort may not be exempt from the evaluation criteria. Each time step may be considered separately, not as a continuous event.
[0105] In at least some embodiments, a recurrent neural network can be used to predict (e.g., in combination with Figures 1a to 1c The machine learning network introduced in , which includes so-called LSTM units (Long Short-Term Memory Network units). Figure 2a A block diagram of a distributed LSTM with an attention layer 208 (e.g., an attention mechanism) is shown. Input features 210 (e.g., multiple features) are provided to the LSTM 202; 204; 206 and the attention layer 208, and a prediction 212 is determined based on the LSTM 202; 204; 206 and based on the attention layer 208. The network may, for example, include three separate LSTM units that each cover predefined attributes, such as characteristics of the target vehicle, map information, etc. ("decoupling"). In addition, in at least some embodiments, the network may be extended with an "attention layer" (e.g., an attention mechanism). Some embodiments may provide a mechanism that allows the network to focus on specific attributes based on the current situation. An example is passing a highway ramp. In this scenario, there is a very high probability that the vehicle on the ramp will perform a lane change. In this case, the system may focus on static attributes of the environment (e.g., street data, a feature set related to the static environment of the target vehicle) and perform predictions based on this focus. This can avoid misinterpretation of other attributes. Another positive side effect of utilizing the attention mechanism can be the interpretability of the prediction results: by explaining the results, it is possible to reconstruct and evaluate, which features / attributes had an improved impact on the prediction.
[0106] In addition, at least some embodiments may provide an evaluation criterion (leaving vehicle comfort metric) that interprets the lane change as an integrated, continuous event and which represents driver comfort / driving comfort (e.g., a "comfort-related metric"). For example, at least some of the following four evaluation criteria may be utilized. The higher the delay between the actual event and the prediction, the less time is left for braking or acceleration; and the more jumpy the reaction may become. Overlap may represent the relative proportion of the first correct prediction of the lane change to the actual lane change maneuver. The higher the overlap, the smoother or less jumpy the reaction may be. Frequency may represent how often a certain prediction is made during a single lane change. The ideal situation is a frequency of 1, as it may only result in a single acceleration or deceleration. The fourth criterion may be denoted as "misses." Misses may represent the number of lane changes that were not recognized. The more lane changes are not recognized, the more frequently the driver may need to intervene and the more uncomfortable driving may become. In Figure 2b An example of a prediction of a lane change and a corresponding evaluation based on comfort-related metrics is shown in FIG.
[0107] Figure 2b The display segments the driving into multiple different time steps. "F" can mean "follow," and "LC" can mean "lane change." Bar 220 shows the time span of the actual lane change maneuver ("ground truth"). In bar partitions 222 and 226, the target vehicle follows, e.g., stays within its lane, and in bar partition 224, the target vehicle performs a lane change. Bars 230 and 240 show possible predictions, where prediction 230 is advantageous because prediction 234 is discontinuous, and predictions 244 and 246 are in the sharp prediction 240. Within predictions 230 and 240, bar partitions 232, 238, 242, and 248 show accurate predictions of the target vehicle's "follow," bars 234, 244, and 246 show accurate lane change predictions for a portion of the actual lane change maneuver 224, and bar partitions show incorrect predictions of the target vehicle's "follow."
[0108] In some embodiments, erroneous interpretations may be reduced by a targeted focus on relevant attributes. The interpretability of the predictions may be improved. The evaluation criteria may allow conclusions to be drawn about driving comfort / driver comfort. The lane change maneuver may be considered as a continuous process.
[0109] Further details and aspects of the prediction concepts and / or comfort-related metrics may be provided in conjunction with the proposed concepts or one or more of the examples described above or below (e.g. Figures 1a to 1e The prediction concept and / or the comfort-related metric may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.
[0110] Lane change prediction of surrounding vehicles can be a key building block for path planning. It is often focused on improving prediction accuracy in a way that makes it purely a function of the estimation problem, at the expense of model comprehensibility. However, the efficiency of any lane change prediction model may be improved if not only extreme cases but also failure cases are human comprehensible. Attention based on recurrent models (e.g. implemented in machine learning networks) is suggested to address not only comprehensibility but also prediction quality. Furthermore, some metrics are presented, which reflect the discomfort felt by the driver. Encouraging results on publicly available datasets and proprietary fleet data are presented in at least some embodiments.
[0111] In many cases, artificial intelligence is seen as a key enabler for fully autonomous driving. Sensing and mapping, perception and (path) planning are generally seen as basic building blocks for any autonomous system that is not end-to-end. The rise of deep learning has led to unprecedented progress in mapping and perception. However, path planning has a hybrid nature, which tends to be model-driven with some subcomponents learned using deep learning. This may be mainly due to the very complex interaction of different agents (static and dynamic) and existing knowledge (maps and traffic information). Data death, including various extreme cases, further limits planning that is completely based on data-driven.
[0112] Predictions can be considered an important part of autonomous driving as a building block for tasks such as path planning, adaptive cruise control, side collision warning, etc. At least some embodiments can address the problem of vehicle lane change prediction. This can be extremely important because approximately 18% of all accidents occur during lane change maneuvers, and lane changes are often performed at high speeds, such as on highways. Therefore, accurate predictions can reduce risks and can allow safer driving. This safety gain derived from sensitive predictions is one side of the coin. On the other hand, though, incorrect predictions may need to be avoided because they can have a negative impact on driver comfort. Each incorrect prediction may result in unnecessary braking or acceleration.
[0113] To predict lane changes, several "classical" models can be utilized, such as Support Vector Machines (SVM) or Random Forests. These classical approaches, although good in theory, may treat maneuver prediction as function estimation. Although the weights on different features may hint at what the function considers important, understanding these models may lack clarity in the analysis when taking prior instructions as input as well. The question to be answered may be: does the system see / can it see what the human observes? For example, if the driver approaches a highway entrance ramp, the probability that the vehicle will change lanes on the ramp is high, and the human driver uses this prior knowledge to slow down (see Figure 3a).
[0114] Figure 3a A diagram showing how the attention mechanism may perceive a scene: In predicting the upcoming lane change of the target vehicle 302, a strong weight may be given to ending up on a ramp 304. In addition, intrinsic features of the target, such as lateral velocity, may also be given high weights, as they are generally good indicators. Small weights may be given to neighboring vehicles 308 in adjacent lanes, gaps that may be determined to be non-critical for the merge maneuver. The ego vehicle 306 may therefore decelerate smoothly.
[0115] To answer the interesting questions above, some implementations may provide:
[0116] a) Recurrent Neural Networks (e.g., machine learning networks) that utilize attention mechanisms on different features and time steps. The model can be designed to understand complex situations and also explain its decisions. Like humans, it can shift its focus to certain important aspects of the current scene.
[0117] b) Introducing metrics (e.g. vehicle comfort metrics) that can indirectly reflect the driver's comfort and thus potentially allow a meaningful quantification of prediction quality.
[0118] c) Provide a comprehensive evaluation of multiple models for the same task on the same benchmark, and analyze key corner cases and explain them graphically.
[0119] d) Using the publicly available NGSIM (Next Generation Simulation) dataset and attributed fleet data ( Figure 3b ) to show encouraging results related to other methods.
[0120] Figure 3b A diagram showing a sample image from fleet data. Bounding box 312 highlights the vehicle that is making a lane change, and bounding box 314 shows two additional vehicles.
[0121] Lane change prediction is a fundamental building block for autonomous driving tasks, is a hot research problem and has been invested for several years. According to the criterion of taking the most informative features and then using "classical" methods such as SVM or random forest, this can contribute to the core of research in lane change prediction. Schlechtriemen et al. ("A lane change detection approach using feature ranking with maximized predictive power", Intelligent Vehicle Conference (IV), 2014) analyzed the expressiveness of multiple features and concluded that the lateral distance to the lane centerline, the lateral speed and the relative speed to the preceding vehicle are the most discriminative features. They introduced two models: a naive Bayesian approach and a hidden Markov model on top of the naive Bayesian model, with the vanilla naive Bayesian approach performing better. In another work, Schlechtriemen et al. ("When will it change the lane? A probabilistic regression approach for rarely occurring events", Intelligent Vehicle Conference (IV), 2015) dealt with the problem of predicting trajectories, where they considered lane change prediction as a helpful subtask. To achieve better summarization, they input all available features into a random forest.
[0122] Woo et al. (“Dynamic potential-model-based feature for lane change prediction”, Systems, Man, and Cybernetics (SMC) International Conference, 2016) proposed a hand-crafted energy field to model the vehicle's surroundings for prediction using the usual SVM model. Weidl et al. (“Situation awareness and early recognition of traffic maneuvers”, EUROSIM Modeling and Simulation Conference, 2016) introduced a dynamic Bayesian network for driving maneuvers that utilizes input features from different sensors and a safe distance from surrounding vehicles.
[0123] A potential drawback of the above approach is the inappropriate treatment of temporal features. Simple feature concatenation across time can lose expressiveness in the temporal domain, mainly due to the high correlation among the features. Patel et al. (“Predicting feature lane changes of other highway vehicles using rnn-based deep models”, arXiv preprint arXiv:1801.04340.2018) introduce a structured recurrent neural network for this problem. Three long short-term memory (LSTM) units process driving and adjacent lanes, where the input is the features of surrounding vehicles in the corresponding lanes and the features of the target. Zeisler et al. (“Vision based lane change detection using true flow features”, International Conference on Intelligent Transportation Systems (ITSC), 2017) then proposed a different scheme by utilizing raw video data instead of high-level features. Using optical flow, lane changes of the observed vehicle are predicted. General intention prediction is closely related to driving maneuvers. Jain et al. (“Recurrent neutral networks for driver activity anticipation via sensory-fusion architecture”, International Conference on Robotics and Automation (ICRA), 2016) presented results on predicting driver intentions.
[0124] Attention mechanisms can be used in vision and translation tasks. The key idea can be to guide the model towards specific input points, such as important image regions for vision tasks and words that are particularly relevant in translation. At least some embodiments integrate a temporal attention mechanism into the model, which can select relevant features from a sequence.
[0125] At least some embodiments relate to predicting lane change maneuvers of vehicles (e.g., target vehicles) around an autonomous vehicle (e.g., vehicle 100). In some embodiments and examples, the following notation may be used. t It can represent a snapshot of a scene containing N vehicles at time step t. The prediction algorithm assigns maneuver labels (left: L, follow: F, right: R) to the vehicles at F. t Each of the N vehicles present. A prediction of L or R may express the algorithm's belief that the vehicle has begun a lane change maneuver to the corresponding side. Conversely, a prediction of F may imply that the vehicle maintains its current lane. To obtain the prediction, for example, the following features may be used in F t For each of the N vehicles (considered as the target vehicle):
[0126] Target vehicle features (e.g., a set of features related to driving parameters of the target vehicle): G Z =(m,v lat , v long , a lat , h). m: lateral distance from the target to the center line of its lane, v lat : lateral velocity, v long : longitudinal velocity, a lat : lateral acceleration, h: heading angle. These features can be calculated in Frenet coordinates. The coordinate axes are along the lane centerline of the target object.
[0127] Dynamic environmental features, i.e., features of vehicles around the target (e.g., a set of features associated with one or more other vehicles near the target vehicle): G E =(dt X for X∈PV,RV,PLV L , PIV R , PFV L , PFV R ), according to the definition of Nie et al. (“Modeling of decision making behavior for discretionary lane-changing execution”, International Conference on Intelligent Transport Systems (ITSC), 2016) (see Figure 3c ). Here, dt x It represents the temporary distance between the target and vehicle X, that is, the distance divided by the speed of the following vehicle.
[0128] Static environment data (e.g., a set of features related to the static environment of the target vehicle): Static features describe the type of environment, such as map-based features. In the NGSIM dataset, there are entry / exit ramps, which are integrated into the G M =(d on , d off , lane). on , d off Respectively represents the distance to the nearest on / off ramp. lane is the one-hot encoding of the lane identifier.
[0129] Figure 3c Visualization showing dynamic environmental features, with direction of travel to the right. Figure 3cTarget vehicle 321 is shown moving to the right, vehicle RV 322 is behind the target vehicle, vehicle PV 323 is in front of the target vehicle, vehicle PFVL 324 is following and PLVL 325 is ahead of the target vehicle in a lane to the left of the center lane, and vehicle PFVR 326 is following and vehicle PLVR 327 is ahead of the target vehicle in a lane to the right of the center lane.
[0130] At least some embodiments utilize two recurrent networks for maneuver prediction:
[0131] (a) includes or is composed of multiple LSTM units, and
[0132] (b) Attention layer on top of the network.
[0133] The model can be trained in a sequential-to-sequential manner, that is, at each time step t, it can produce an output y t ∈{L, F, R]. The input features (T, E and M) used for this method are described in more detail later.
[0134] At least some embodiments utilize a LSTM unit having three different z , LSTM E , LSTM M ) to process the feature groups (G 2 , G E , G M ). This decoupling into multiple separate LSTMs can provide that the correlation within a group is high, but the correlation between groups is low. The following simplified notation can be used for LSTM units.
[0135]
[0136] Among them, X∈{G Z , G E E, G M} is the input, h represents the hidden state, and Represents a memory unit. The entire network is Figure 3d Visible in. Figure 3d Visualization showing the LSTM network used. Each feature class is processed by a different LSTM unit (LSTM z For t-1 331a, LSTM z For t 331b, LSTM E For t-1332a, LSTM E For t 332b, LSTM M For t-1 333a, LSTM MFor t 333b). Each LSTM receives the corresponding input Each result then passes through layer u t-1 ,u t 335 and o t-1 , o t 336 Fusion. The predicted output y (eg, information related to the lane change of the target vehicle) is obtained by applying a softmax function 337. FC 334 represents a fully connected layer.
[0137] Mathematically, the fusion of the three LSTMs can be expressed as equation (1):
[0138]
[0139] o t =tanh(W u ·u t +b u )
[0140] y t =soffmax(W o o t +b o )
[0141] Among them, the subscript W is the weight matrix, the subscript b is the bias vector, u is the fusion layer, and y is the output layer.
[0142] At least some embodiments may utilize an attention network (e.g., an attention mechanism). The idea behind the attention mechanism is to model selective focus, such as specific parts of the input. It mainly consists of or is composed of a function that processes a key (K) and a query (Q) to obtain a context vector. The context vector is an accumulation of multiple keys, which are weighted by their importance with respect to the query. Some embodiments may employ two attention mechanisms:
[0143] (a) attending to the previous time step, i.e., self-attention; and
[0144] (b) Note the different feature groups.
[0145] As opposed to some other attention methods, the features implemented can be in different spaces and have different shapes. They may not be accumulated, but their magnitude may be changed (only) according to the weighting, and these feature vectors can be accumulated over time steps, see for intuitive visualization Figure 3e .
[0146] Figure 3eAn example of attention calculation for a frame at time t is shown: for this, time steps tN, ..., t are considered. The scene 340a for time t and the scene 340b for time t-1 are drawn in vertical order. t , E t-1 , 341a, 341b are displayed thereafter. t 342a, 342b are used as keys to calculate the attention weight β t , β t-1 The embeddings are then scaled by type using them. Again using u t As the key (K), calculate the attention weight Y t 343 (e.g., two or more weighting factors) indicating the importance of different time steps. t The weighted sum of the scaled embeddings of composes the context vector c t 344. Vehicles 345a / b, 346a / b, 347a / b and 348a / b are translated into corresponding inserts.
[0147] In at least some embodiments, features may be divided into categories, but with finer granularity than in the previous Section III, i.e., Z =G Z , H S =[dt PV , dt RV ], and Attention function Ψ: It is given by equation (2).
[0148] Ψ(W, v, Q, K) = v T tanh(W[Q;K])
[0149] For all calls to Ψ at time step t, layer u t as the key. Let T = {tN, ..., t} be the time step for self-attention. In some examples, N = 20 is used. For each i∈T, the feature category can be embedded into a higher dimensional space, and the importance of each feature category, And as a whole, at each time step, γ i , can be determined. Given equation (3):
[0150] C={H Z , H L , H S , H R , H M}
[0151]
[0152] Among them, X∈C, i∈T. Finally, the feature type is used The resulting context vector is added to the fusion layer and calculated as per Eq. (1).
[0153] γ t =softmax([concat(γ i )])
[0154]
[0155] u t =[u t ;c t ]
[0156] At least some embodiments may provide for visualization of attention. In addition to improved performance, another major advantage of attention is its interpretability. In at least some embodiments, only the magnitude of the attention weights is displayed, which are used in the calculation of the weighted average. However, due to the different scales and dimensions of the feature types, this may not necessarily lead to the desired results. Instead, in some embodiments, the derivative of the predicted classification can be obtained by the attention weights β. i X and γ i , which are summed over all time steps. The derivative represents the contribution of category X to the resulting prediction, and even provides information on whether the contribution is positive or negative.
[0157] At least some embodiments may use an exponential growth loss to encourage early predictions. The Softmax loss used can be expressed as t exp(-T), where at time t, the lane change is expected to occur in the next T seconds. Exponential weighting of the loss function may not be used for fleet data, since human labels are error-free. α can be chosen so that the average value of α·exp(-T) over all frames of each lane change maneuver is equal to 1. For a given maneuver at time t, w t It may be inversely proportional to the global size of the driving maneuvers in the training data.
[0158] As mentioned by Schlechtriemen et al., simple scenarios cover most lane changes, and relatively good predictions can already be obtained by using the information from G Z To handle this imbalance and induce meaningful gradient flow for attention in all cases, some embodiments introduce a dropout layer between layers u and o, i.e.:
[0159] Dropout=[W Drop,Fusion ; W Drop,c ]·[u t ;c t ]+b drop
[0160] With probability p = 0.33, W Drop,Fusion and W Drop,c Can be independently set to 0, forcing the model to rely solely on its recurrent architecture or attention.
[0161] Some embodiments have been evaluated using different datasets. The Next Generation Simulation (NNGSIM) project consists of four publicly available traffic datasets. Figure 3f , the US Highway 101 dataset (US-101) and the Interstate 80 highway kilometers dataset (I-80) have been utilized. Figure 3f A sketch (not to scale) of the US-101 dataset is shown, with five lanes 352 and one lighted lane 354. Note that the I-80 dataset has no off-ramps. The data is acquired from a bird's-eye view of the highway using a static camera, and vehicle-related high-level features are extracted from it. The dataset contains measurements at 10Hz. After removing noisy trajectories, 3184 lane changes were observed. Another dataset is derived from fleet data: The fleet data is derived from an environment model for a fleet of vehicles. The data is acquired at 25Hz for moving autonomous vehicles equipped with several cameras and radar sensors to give a full 360° view. 830 lane changes were recorded.
[0162] Various metrics are used to measure the performance of lane change prediction algorithms. Mainly, they are influenced by information retrieval and are calculated at each time step independently from other algorithms.
[0163] ■ Accuracy: The percentage of time steps that are correctly classified.
[0164] The first approach based on maneuverability metrics was introduced by Jain et al. as follows.
[0165] ■Precision: The percentage of true predictions relative to the total number of maneuver predictions.
[0166] ■Recall: The percentage of true predictions with respect to the total number of maneuvers.
[0167] ■ Time to Maneuver (TTM): The interval between the predicted time and the actual start of the maneuver in ground truth.
[0168] The ground truth labels are continuous in an event-wise manner (see Figure 3g ). Maneuver tags are continuous events. Figure 3gIn FIG. 5 , F represents the following at t=[0, t1] and t=[t2, T], and label LC represents the lane change event at t=[t1, t2].
[0169] However, the information extraction metric may not reflect the nature of the event-by-event approach or what the driver experiences in the vehicle. The vehicle controller may react to the first predicted event (see Figure 3h ). If the prediction is discontinuous, this causes discomfort to the driver (stop-and-go functionality). In addition, the predicted events can be as early as possible with respect to the ground truth, the earlier the prediction, the higher the comfort. To reflect this comfort-related behavior, some embodiments may utilize the following per-event metrics:
[0170] ■ Latency: The delay in the prediction with respect to the ground truth label (measured in seconds). If the prediction lines up exactly with the ground truth, the latency is 0.
[0171] ■ Overlap: The percentage of overlap for the earliest predicted maneuver for a given ground truth event. The higher the overlap, the smoother the controller's response.
[0172] ■ Frequency: The number of maneuver events predicted for each ground truth event. For follow-up events, this represents the False Positive Rate (FPR).
[0173] ■ Misses: The number of lane changes that were completely missed. The higher the number of misses, the more uncomfortable it is because the driver needs to intervene.
[0174] exist Figure 3h In FIG. 3 , if a given ground truth event 372 has multiple corresponding predicted events, then for overlapping LCIs 374 and 376 in sharp predictions 378, the Figure 3h The comfort-related metrics for events are: delay=0, overlap=20%, frequency=2, miss=0.
[0175] The perception of the exact moment when the lane change begins varies from person to person, see Figure 3i .exist Figure 3i Comparing person-to-person labels in Figure 2: Precision and recall of ground truth lane changes on proprietary fleet data. Each trace is labeled by three different people. Metrics are calculated in a 1-vs-rest manner and then averaged. This shows that people agree on almost all lane changes, but have small disagreements on when to actually start the maneuver, i.e., opinions on maneuver intensity vary from person to person.
[0176] Therefore, manually labeled lane changes in fleet data may imply intent. However, automatic labeling may be advantageous in the NGSIM case because of the similar time span of lane changes. Therefore, a 3-second criterion may be adopted to label lane changes before the target's lane assignment changes. Although human labeling is accurate and error-free, it can be time consuming and expensive. Intelligent automatic labeling may be slightly inaccurate, but may instead be faster and may prove better for deep models, which may extract good cues that are imperceptible to humans.
[0177] Some implementations are evaluated. Two proposed recurrent methods are denoted as LSTM-E (extended LSTM) and LSTM-A (extended LSTM with attention). For both, a hidden size of 128 is used. State-of-the-art baselines are used to demonstrate the better performance of the implementations.
[0178] A. Baseline Method
[0179] Frame-based: uses features from a single time step.
[0180] ■ Random Forest (RF): Connected features (G Z , G E , G M ) is used as input.
[0181] ■Naive Bayes (NB): Use features m, v lat and the relative speed to the vehicle ahead.
[0182] Based on order:
[0183] ■Structural RNN (SRNN): SRNN includes three different LSTM units, which cover the target, left lane and right lane respectively. For each LSTM unit, a feature Q of three features is given, namely the features of the two neighboring vehicles of the target vehicle (PV-RV / PLV L -PLV R / PLV R -PFV R ) and the characteristics of the target vehicle itself. Q can consist of the absolute world coordinates, the lateral and longitudinal speeds, the heading angle, and the number of lanes to the left and right. The outputs of the three LSTM units are passed to another LSTM unit, which finally outputs the prediction.
[0184] ■ Vanilla LSTM (LSTM): The vanilla LSTM consists of a single unit with concatenated features (GZ, GE, GM).
[0185] Figure 4Results of all tested methods on NGSIM and Fleet datasets are shown with respect to all metrics. Figure 4 Comparison and ranking of different methods on different metrics are shown for NGSIM (top table) and fleet data (bottom table). The global ranking can be the average, lower, better of the algorithm rankings for each column. The ranking of the algorithm for a specific column is the index, which occupies that column after the ranking.
[0186] As can be seen, due to the diversity of evaluation metrics, some methods excel or fail in different categories. Sequence-based methods can outperform frame-based methods because the latter do not carry information about sequence history. Among the sequence-based methods, three recurrent methods, LSTM, LSTM-E, and LSTM-A of some embodiments stand out (see the "Rank" column in the table).
[0187] On the NGSIM dataset, the LSTM network with attention is the best performing method. It has the lowest latency, while predicting lane changes with low false positives during "following" and good sequential predictions shown by "overlap". On the convoy data, LSTM-A comes in second. A plausible reason for this is the simplistic structure of the dataset, as scenes tend to be less crowded and therefore do not exploit the full capabilities of the complex attention mechanism.
[0188] If Figure 4 As can be seen, the performance of some methods is relatively similar. Analyzing and explaining a few key corner cases helps evaluate the performance. This can provide clarity on the advantages of the attention mechanism. These key corner cases do not appear in the data. They are created by translating the existing trajectories around their position in the scene and thus remain realistic.
[0189] For fleet data no static data records are available, but rather moving autonomous vehicles (e.g. Figure 6a-1 and 6a-2 616), measurements of the scene are acquired from the autonomous vehicle. Two visualizations are used: (a) a snapshot visualization of a single frame and a visualization of the temporal evolution of the scene. The first can include a single image showing the ground truth for the predictions of a single algorithm for that frame, and attention visualizations for the five feature categories. For better readability of the categories, HZ, HS, HL, HR, and HM are represented by target, same, left, right, and map / street. (b) A concatenation of several frames spanning a certain amount of time, together with the predictions of the different algorithms.
[0190] Figures 5a to 5e Showing the impact of attention on the network’s decision making, highlighting its correct and intuitive contribution.
[0191] exist Figure 5e In FIG. 5 , target vehicle 502 is performing a lane change from the auxiliary lane to the left. Since lane changes are common and expected behavior after entering a ramp, street 504 has a high positive contribution to predicting L.
[0192] exist Figures 5b to 5e Two scenarios for changing lanes to the left are shown in Figure 2. Figure 5b and 5c Scenario (a) and Figure 5d and 5e The scenes (b) are derived from the same actual scene and differ only in the position of PV 512, which is close to the target in scene (a) and disappears in scene (b). Figure 5b , Figure 5d ) respectively, F is predicted, in each second figure ( Figure 5c , Figure 5e ) respectively, L is predicted.
[0193] exist Figure 5b In / 5c, the target (514) is approaching PV 512 quickly and the human is able to anticipate the upcoming lane change. The same reasoning is performed by the attention mechanism. The same 516 has a strong negative contribution to the prediction F and a strong positive contribution to the possible prediction L. Figure 5d In / 5e, the same contribution is exactly the opposite of a.
[0194] Figure 6a-1 , 6a-2 6b-1, 6b-2, 6b-3 show the temporal development of the two scenarios, while plotting the outputs of the three algorithms RF 610, LSTM-E 612, and LSTM-A 614. Overall, it can be observed that the recurrent models, especially LSTM-A, have excellent performance.
[0195] Figure 6a-1 and 6a-2 Displays a visualization of right lane changes as recorded from a series of vehicles ( Figure 3b Front camera image is shown). LSTM-A 614 predicts first, followed by Random Forest 610 and LSTM-E 612. An autonomous vehicle is shown in the depicted scene by reference numeral 616, and the vehicle is shown performing a lane change by reference numeral 618.
[0196] Figure 6b-1 , 6b-2and 6b-3 show a "fake" left lane change. The target vehicle (620) begins to move towards its left lane boundary. A lane change is not possible because this is blocked by a neighboring vehicle (622). Despite this, the random forest is quickly "fooled" and predicts a "left" lane change, whereas the recurrent network correctly predicts "follow" and only has an incorrect prediction near the end. Note that the attention mechanism may not completely prevent incorrect lane change behavior because the relationship with surrounding vehicles is learned and not hard-coded. Once strong lateral motion is observed, the possibility of an upcoming lane change can be considered, regardless of the current road situation. In the vast majority of these cases, though, the attention mechanism can prevent incorrect lane change predictions.
[0197] Some embodiments provide LSTM networks with attention mechanisms for lane change prediction, which can perform better than existing methods on different evaluation schemes. Attention mechanisms can address prediction quality and understandability. At least some embodiments can provide new event-based metrics for driver comfort. Results on public datasets as well as fleet data show that using the methods of embodiments for drivers, there is a high level of comfort in terms of earlyness in prediction, false positive rate, and miss rate. In addition, visual analysis of key situations is used to demonstrate the efficiency of using attention.
[0198] Aspects and features mentioned and described in conjunction with one or more previously detailed examples and figures may further be combined with one or more other examples in order to replace similar features of the other examples or in order to additionally introduce features into the other examples.
[0199] Some examples may further be or relate to a computer program with a program code, which, when executed on a computer or processor, is used to perform one or more of the above methods. The steps, operations or processes of the various methods described above may be performed by a programmed computer or processor. Some examples may also cover program storage devices such as digital data storage media, which are machine, processor or computer readable and encode machine-executable, processor-executable or computer-executable instruction programs. The instructions execute or cause the execution of some or all actions of the above methods. The program storage device may include or be a digital data storage medium such as a digital memory, a magnetic storage medium such as a disk and a tape, a hard disk or an optical scale. Other examples may also cover a computer, a processor or a control unit programmed to perform the actions of the above methods; or a (field) programmable logic array (FPLA, PLA) or a (field) programmable gate array (FPGA, PGA), which is programmed to perform the actions of the above methods.
[0200] The description and drawings merely describe the principles of the disclosure. In addition, all examples recorded herein are clearly intended to be merely descriptive in order to help the reader understand the principles of the disclosure and conception of the invention as contributed by the inventor to the technical improvement. The description of principles, statements of aspects, examples disclosed, and their specific examples herein are intended to include their equivalents.
[0201] A functional block that performs a certain function and is represented as a "means for..." may refer to a circuit that is configured to perform a certain function. Therefore, a "means for something" may be implemented as a "means configured to be used for something or suitable for something", such as a device or circuit configured or suitable for a corresponding task.
[0202] The functions of the various elements shown in the drawings, including any functional blocks labeled as "means", "means for providing a signal", "means for generating a signal", etc., may be implemented in the form of dedicated hardware such as "signal providers", "signal processing units", "processors", "controllers", etc., as well as hardware capable of executing software in association with appropriate software. If provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some or all of which may be shared. However, the term "processor" or "controller" is not limited to hardware capable of executing software alone, but may include digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memories (ROMs), random access memories (RAMs), and non-volatile memories for storing software. Other hardware, conventional and / or custom, may also be included.
[0203] Block diagrams may, for example, be high-level circuit diagrams that depict the principles of the disclosed invention. Similarly, flow charts, flow diagrams, state transfer diagrams, pseudocodes, and the like may represent different processes, operations, or steps that may, for example, be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown. Methods disclosed in the specification or claims may be implemented by a device having means for performing each of the corresponding actions of these methods.
[0204] It is understood that multiple actions, processes, operations, steps or functions disclosed in the specification or claims may not be constructed to be in a specific order, unless otherwise explicitly or implicitly indicated, for example, due to technical reasons. Therefore, the disclosure of multiple actions or functions does not limit them to a specific order, unless such actions or functions are not interchangeable due to technical reasons. In addition, in some examples, a single action, function, process, operation or step may include or may be decomposed into multiple sub-actions, sub-functions, sub-processes, sub-operations or sub-steps, respectively. Such sub-actions may be included and are part of the disclosure of the single action, unless explicitly excluded.
[0205] Furthermore, the following claims are hereby incorporated into the specific description, each claim being able to serve as a separate example on its own. Although each claim is able to serve as a separate example on its own, it is noted that, although a dependent claim may refer to a specific combination with one or more other features in the claim, other examples may also include combinations of the dependent claim with any other dependent claim and the subject matter of the independent claim. Such combinations are expressly suggested herein, unless it is indicated that a specific combination is not desired. It is also intended that features of a claim be included in any other independent claim, even if the claim is not directly dependent on the independent claim.
Claims
1. A method for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle, the method comprising: Obtaining a start time of a driving maneuver of a target vehicle in an environment of the autonomous vehicle; Inputting a plurality of input feature groups of the target vehicle into corresponding plurality of long short-term memory network (LSTM) units of the recurrent neural network, the input feature groups corresponding to two or more of the following elements: driving parameters of the target vehicle, other vehicles in the vicinity of the target vehicle, or a static environment of the target vehicle; selectively weighting the input feature set using an attention layer of a recurrent neural network; obtaining, as an output from the recurrent neural network, a predicted start time of a predicted driving maneuver of the target vehicle; and A vehicle comfort metric for an autonomous vehicle is determined for predicting a driving maneuver by comparing a start time of the driving maneuver to a predicted start time of the driving maneuver output from a recurrent neural network to determine a delay between the start time of the driving maneuver and the predicted start time of the driving maneuver, wherein a larger delay is associated with reduced vehicle comfort.
2. The method according to claim 1, wherein: The method further includes obtaining a first duration of a driving maneuver of the target vehicle, wherein the vehicle comfort metric is further determined based on an overlap between the first duration of the driving maneuver and a predicted second duration of the driving maneuver.
3. The method according to claim 1, wherein: The vehicle comfort metric is also determined based on a plurality of discrete predictions within a duration of the driving maneuver.
4. The method according to claim 1, wherein: The vehicle comfort metric is also determined based on a prediction of a missed driving maneuver.
5. The method according to any one of claims 1 to 4, wherein: The method is also adapted to determine information related to a lane change of a target vehicle, wherein the information related to a lane change of the target vehicle indicates whether the target vehicle is expected to perform a lane change, the method comprising: Acquiring (110) information related to an environment of the target vehicle, the information related to the environment relating to a plurality of features of the environment of the target vehicle, the plurality of features being divided into two or more feature groups; determining (120) two or more weighting factors for the two or more feature groups, wherein an attention mechanism is used to determine the two or more weighting factors, the attention mechanism being a mechanism for determining selective attention to a subset of features among the plurality of features, the attention mechanism employing at least one of “paying attention to previous time steps” and “paying attention to different feature groups”; and Information related to a lane change of a target vehicle is determined (130) based on information related to an environment of the target vehicle using a machine learning network, wherein weighting of the plurality of features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
6. The method according to claim 5, wherein: The two or more feature groups include at least two elements among a group of features related to driving parameters of the target vehicle, a group of features related to one or more other vehicles near the target vehicle, and a group of features related to a static environment of the target vehicle.
7. The method according to claim 6, wherein: The information about the environment of the target vehicle is based on processed sensor data of the vehicle (100) and / or on mapping data; and / or features of a feature set related to driving parameters of a target vehicle based on processed sensor data of the vehicle (100); and / or features of a set of features associated with one or more other vehicles in the vicinity of the target vehicle based on processed sensor data of the vehicle (100); And / or features of the feature set related to the static environment of the target vehicle are based on mapping data.
8. The method according to claim 5, wherein: The machine learning network includes an attention layer, which is used to determine the two or more weighting factors.
9. The method according to claim 8, wherein: The attention layer is configured to determine the two or more weighting factors based on features of the two or more feature groups; And / or the attention layer is configured to determine the two or more weighting factors based on current information related to the environment and based on a previous state of the attention layer.
10. The method according to claim 8, wherein: The method includes repeatedly (140) determining information related to a lane change of a target vehicle, wherein for each repeated determination of information related to a lane change of a target vehicle, the two or more weighting factors are re-determined (142).
11. The method according to claim 5, wherein: The machine learning network is a recurrent machine learning network and / or the machine learning network is configured to determine information related to a lane change of the target vehicle based on current information related to an environment of the target vehicle and based on a previous state of the machine learning network.
12. The method according to claim 11, wherein: The machine learning network includes two or more long short-term memory network units, wherein information related to the lane change of the target vehicle is determined using the two or more long short-term memory network units based on current information related to the environment of the target vehicle and based on a previous state of one or more long short-term memory network units (130); And / or each of the two or more long short-term memory network units is equipped with a feature group among the two or more feature groups.
13. The method according to claim 5, wherein: The method includes guiding (150) a vehicle (100) based on information related to a lane change of a target vehicle, wherein the vehicle (100) is guided by accelerating (152) or decelerating (154) the vehicle (100) and / or by steering (156) the vehicle (100).
14. The method according to claim 5, wherein: Information related to a lane change of the target vehicle is determined to predict the lane change of the target vehicle.
15. The method according to claim 5, wherein: The method includes determining (160) information related to lane changes of target vehicles for a plurality of target vehicles in the vicinity of the vehicle (100). 16 . A program storage device storing a computer program having a program code for implementing the method according to claim 1 when the computer program is executed on a computer, a processor or programmable hardware.
17. An apparatus for determining a vehicle comfort metric of an autonomous vehicle for predicting a driving maneuver of a target vehicle, the apparatus comprising: At least one interface, the at least one interface configured to: - obtaining an extent of a driving maneuver of a target vehicle in an environment of the autonomous vehicle, the extent corresponding to one or more of the following elements: a first duration of the driving maneuver of the target vehicle, or a start time of the driving maneuver of the target vehicle; - inputting a plurality of input feature groups of the target vehicle into corresponding plurality of long short-term memory network (LSTM) units of the recurrent neural network, wherein the input feature groups correspond to two or more of the following elements: driving parameters of the target vehicle, other vehicles in the vicinity of the target vehicle, and a static environment of the target vehicle; - selectively weighting the input feature set using an attention layer of a recurrent neural network; - obtaining, as an output from the recurrent neural network, a prediction of a driving maneuver of the target vehicle; and a computing module configured to: - determining a vehicle comfort metric for the autonomous vehicle based on an overlap between a first duration of the driving maneuver and a predicted second duration of the driving maneuver for the prediction of the driving maneuver output from the recurrent neural network, wherein a greater overlap is associated with increased vehicle comfort.
18. The apparatus according to claim 17, wherein: The vehicle comfort metric is also determined based on a delay between a start time of the driving maneuver and a predicted start time of the driving maneuver.
19. The apparatus according to claim 17, wherein: The vehicle comfort metric is also determined based on a plurality of discrete predictions within a first duration of the driving maneuver.
20. The apparatus of claim 17, wherein: The vehicle comfort metric is also determined based on a prediction of a missed driving maneuver.
21. The apparatus according to any one of claims 17 to 20, further adapted to determine information related to a lane change of a target vehicle, the information related to a lane change of a target vehicle indicating whether the target vehicle is expected to perform a lane change, the apparatus comprising: at least one interface (12) for acquiring information related to an environment of the target vehicle, wherein the information related to the environment relates to a plurality of features of the environment of the target vehicle, the plurality of features being divided into two or more feature groups; and A calculation module (14), the calculation module being configured to: - determining two or more weighting factors for the two or more feature groups, an attention mechanism is used to determine the two or more weighting factors, the attention mechanism is a mechanism for determining selective attention to a feature subset among the plurality of features, the attention mechanism adopts at least one of "paying attention to previous time steps" and "paying attention to different feature groups"; and - using a machine learning network, based on information related to the environment of the target vehicle, determining information related to the lane change of the target vehicle, wherein the weighting of the multiple features of the environment of the target vehicle within the machine learning network is based on the two or more weighting factors for the two or more feature groups.
Citation Information
Patent Citations
Cooperative self-adaptive cruise control system algorithm based on traffic information
CN106476806A
Method and system for implementing attention mechanism in artificial neural network
CN107909151A
Method and vehicle communication system for determining a driving intention for a vehicle
CN108352112A
Trained navigational system with imposed constraints
CN108431549A
Motor vehicle driving maneuver e.g. lane change, detecting method, involves defining bayes networks for specifying probability of presence of maneuver based on preset state variables, and calculating probability of presence of maneuver
DE102005043838A1
