METHODS AND SYSTEMS FOR PREDICTING THE PROPERTIES OF A MULTIPLE OF OBJECTS IN THE NEAR A VEHICLE

Description

FIELD

[0001] The present disclosure relates to methods and systems for predicting properties of a plurality of objects in a vicinity of a vehicle.BACKGROUND

[0002] Predicting future locations of an object is an important task in various applications, for example in (at least partially) autonomous driving.

[0003] Accordingly, there is a need to provide efficient and reliable methods for predicting future locations of one or more objects in a scene.

[0004] US 2019 / 113927 A1 discloses a computer-readable medium which stores instructions executable by one or more processors to implement a self-driving control architecture for controlling an autonomous vehicle. A perception and prediction component receives sensor data, and generates an observed occupancy grid indicating which cells are currently occupied in a two-dimensional representation of the environment, and predicted occupancy grids indicating which cells are expected to be occupied later. A mapping component provides navigation data for guiding the vehicle toward a destination, and a cost map generation component is configured to generate, based on the observed occupancy grid, the predicted occupancy grid(s), and the navigation data, cost maps that each specify numerical values representing a cost, at a respective instance of time, of occupying certain cells in a two-dimensional representation of the environment.

[0005] A motion planner generates a grid path through the environment based on the cost maps, and corresponding decisions for maneuvering the vehicle.

[0006] US 2021 / 101624 A1 discloses a collision avoidance system which may validate, reject, or replace a trajectory generated to control a vehicle. The collision avoidance system may comprise a secondary perception component that may comprise one or more machine learned models, each of which may be trained to output one or more occupancy maps based at least in part on sensor data of different types. The occupancy maps may include a prediction of whether at least a portion of an environment is occupied at a future time by any one of multiple object types. Occupancy maps associated with a same time may be aggregated into a data structure that may be used to validate, reject, or replace the trajectory.

[0007] US 2020 / 148215 A1 discloses methods and systems for generating a predicted occupancy grid map (OGM) over at least one future time step are described. The system include a first encoder for extracting OGM features from an input OGM in a current time step. The system also includes a recurrent neural network for generating a corrective term from at least the OGM features, wherein the corrective term represents predicted change to the input OGM, and wherein the corrective term is applied to the input OGM to generate a corrected OGM. The corrected OGM represents features corresponding to occupancy of the environment in a first future time step. The system also includes a classifier for converting the corrected OGM to the predicted OGM for the first future time step. The predicted OGM is fed back as input for performing generating a predicted OGM for a second future time step.SUMMARY

[0008] The present disclosure provides a computer implemented method, a computer system and a non-transitory computer readable medium according to the independent claims. Embodiments are given in the subclaims, the description and the drawings.

[0009] In one aspect, the present disclosure is directed at a computer implemented method for predicting properties of a plurality of objects in a vicinity of a vehicle, the method comprising the following steps carried out by computer hardware components: determining a grid map representation of road users perception data, the road users perception data comprising tracked perception results and / or untracked sensor intermediate detections; determining a grid map representation of static environment data based on data obtained from a perception system and / or a pre-determined map; and determining the properties of the plurality of objects based on the grid map representation of road users perception data and the grid map representation of static environment data.

[0010] Untracked sensor intermediate detections may for example include radar data or camera data. For radar data, radar detections may include points, each point being a reflection of the radio wave sent out by radar, wherein metal objects have stronger reflection, and thus result in more detection points, so that these points can be located at any random position of the vehicle, and the number of detected points is also variable and random. For camera data as intermediate detection, methods may work on images and may normally detect multiple bounding boxes around a target, e.g. a vehicle, wherein these boxes may only differ from each other slightly, but they all come from the same object. In both radar data and camera data, these detections may be tracked, for example through Kalman Filter, and only after tracking, a track may be obtained, which corresponds to the vehicle, with some physical properties estimated, such as position or speed. The untracked sensor intermediate data may include data that has not gone through such tracking.

[0011] According to another aspect, the properties of the plurality of objects are determined as a function of at least one variable point in time for which the properties are determined.

[0012] According to another aspect, the properties of the plurality of objects are determined in the form of Y(t) = At 2< + Bt 1< + C, wherein Y(t) denotes the properties (as a function of t), t denotes at least one point in time for which the properties are determined, and A, B, and C are constants, which are preferably used for sampling or interpolation or extrapolation at the variable time t, and which are preferably determined as variable output of a network.

[0013] According to another aspect, the properties can also be determined for discrete times (so that A and B are canceled out), wherein t may be a time taken from a point set T. In the discrete form, the output may be a set of properties predicted (for each such discrete and fixed time) by the network, and may be denoted as Y = {C t }, t ∈ T.

[0014] According to another aspect, the properties are determined for a past point in time and / or a present point in time and / or a future point in time.

[0015] According to another aspect, the properties of the plurality of objects comprises a probability matrix, wherein each entry of the probability matrix is related to a respective pre-determined region in the vicinity of the vehicle and indicates a probability of an object being present in the respective region at a time different from the pre-determined time, wherein the computer implemented method further comprises determining an offset to the center of the region for each of the entries of the probability matrix, wherein preferably the offsets are determined using a regression method.

[0016] According to another aspect, the grid map representation of road users perception data is determined further based on data from a perception system, and / or the grid map representation of static environment data is determined further based on a pre-determined map.

[0017] According to another aspect, the grid map representation of road users perception data is encoded using a dynamic context encoder, wherein the grid map representation of static environment data is encoded using a static context encoder, wherein the properties of the plurality of objects is decoded using a decoder. For example, a recurrent neural network is provided between the dynamic context encoder and the static context encoder, and a dilated neural network is provided between the static context encoder and the decoder; or a recurrent neural network and a dilated neural network are provided between the static context encoder and the decoder.

[0018] According to another aspect, the classification method comprises a machine learning method, preferably an artificial neural network, preferably comprising a plurality of skip-connections with temporal filtering in a pyramid neural network architecture, wherein preferable the machine learning method is trained based on training data, the training data comprising traffic situations of a plurality of objects moving over a plurality of time frames.

[0019] According to another aspect, the properties of the plurality of objects are determined for a plurality of classes, wherein the plurality of object classes comprise at least two of an ego vehicle class, a pedestrian class, a bicyclist class, a vehicle class, a truck class, a lorry class, a fast moving object class, or a slow moving object class.

[0020] According to another aspect, the properties of the plurality of objects comprise at least one of positions, speeds, heading angles, intentions, or collision risk values.

[0021] According to another aspect, the computer implemented method is used for fusion or for prediction. For example, the input may come from various sensors, and then the method may be used as a fusion method (for example in a fusion system).

[0022] In a further aspect, the present disclosure is directed at a computer implemented method for predicting properties (for example positions, velocities, heading angles, driving intentions, or collision risk values) of a plurality of objects (which may, for example, be of a plurality of classes) in a vicinity of a vehicle, the method comprising the following steps performed (in other words: carried out) by computer hardware components: determining at least one map comprising a representation of the vicinity of the vehicle at a pre-determined time; and based on the at least one input map, determining a function in time indicating a probability of an object being present in the respective region at a time t different from the pre-determined time, wherein the function in time comprises a constant portion, wherein the constant portion is determined by applying a classification method to the at least one input map, the classification method providing as an output a probability matrix, wherein each entry of the probability matrix is related to a respective pre-determined region in the vicinity of the vehicle and indicates a probability of an object being present in the respective region at a time t different from the pre-determined time, wherein the classification method determines a plurality of functions in time and a plurality of probability matrices, each function in time and each probability matrix related to one of a plurality of object classes, each class comprising a plurality of objects.

[0023] In other words, a multimodal future position prediction (for example of multiple agents) based on given context information may be provided. It will be understood that "multimodal" future position prediction may mean that multiple future positions are feasible for a single agent. A technique for predicting future positions of multiple objects for different future points in time may be provided.

[0024] According to another aspect, the at least one map comprises a map indicating a static context of the vicinity. For example, the static context may represent objects which do not move (for example roads, buildings, or trees). The static context may represent the environment which restricts the movement of the road users and is not affected by the road users. For example, the static context may include objects which do not move (for example roads, buildings, or trees). The static context may also include traffic rules. The static context may include objects which are dynamic, for example standard traffic lights or weather conditions. All of this static context affects the road users, and road users' behavior does not affect the static context.

[0025] According to another aspect, the at least one map comprises a map indicating a dynamic context of the vicinity. The dynamic context may represent objects which move, for example other traffic participants.

[0026] The dynamic context may include the whole road users' state, which may also include the ego vehicle. If one specific road user's future trajectory is to be predicted, then all other road users may be considered as his dynamic context.

[0027] According to various embodiments, all the road users may be predicted jointly, thus their overall trajectories are considered as dynamic context of the scene.

[0028] The ego vehicle's movement may be part of the dynamic context, and may be treated equally as other road users in the dynamic context input grid map.

[0029] Furthermore, the ego vehicle's movement may also be used indirectly for data preprocessing. From the ego vehicle perception system, the other road users' trajectories may be provided in the ego vehicle coordinate system (VCS). The ego vehicle may be moving, and thus, the perception at each frame may be represented in coordinate systems which itself are moving. According to various embodiments, the ego dynamic (which may be or may include the ego vehicle's movement) may be used to compensate the other users' trajectories and transform them into a world coordinate system (WCS); in this case, the trajectory data from multiple past time frames may have the same fixed coordinate system. This way, the ego dynamics may indirectly be used as input (for example for preprocessing of the perception result of other road users).

[0030] According to another aspect, the classification method comprises a machine learning method, preferably an artificial neural network. The machine learning method may be trained using recorded movement scenes.

[0031] According to another aspect, the machine learning method is trained based on training data, the training data comprising traffic situations of a plurality of objects moving over a plurality of time frames.

[0032] According to another aspect, the machine learning method is trained based on a loss function, wherein the loss function preferably comprises a pixel-wise distance aware cross entropy.

[0033] According to another aspect, the classification method determines a plurality of probability matrices, each probability matrix related to one of a plurality of object classes.

[0034] According to another aspect, the plurality of object classes comprise at least two of an ego-vehicle class, a pedestrian class, a bicyclist class, a vehicle class, a truck class, a lorry class, a fast moving object class, or a slow moving object class.

[0035] According to another aspect, the probability matrix is related to a center of the respective pre-determined region, and the computer implemented method further comprises the following step carried out by the computer hardware components: determining an offset to the center of the region for each of the entries of the probability matrix.

[0036] According to another aspect, the offset is determined based on a regression loss. The determination of the offset (based on the regression loss) may allow minimizing a difference between a predicted location and an actual location, by allowing for a higher spatial resolution than provided by the cells (in other words: pixels) of the images or matrices used in the classification method.

[0037] According to another aspect, a plurality of probability matrices are determined related to a plurality of times different from the pre-determined time.

[0038] According to another aspect, the computer implemented method further comprises the following step carried out by the computer hardware components: determining an occupation matrix, wherein each entry of the occupation matrix is related to a respective pre-determined region in the vicinity of the vehicle and indicates whether or not an object is present at the region, and wherein a corresponding entry of the probability matrix is only determined if the entry of the occupation matrix indicates that an object is present at the region. It has been found that this increases efficiency.

[0039] The method may support trajectory prediction of an arbitrary number of road users in one general approach. The computing effort does not depend on the number of road users which may be a considerable advantage to state-of-the art approaches. The method may support the prediction of various types of traffic participants, e.g. cars, pedestrians, bicycles etc. and may take multimodality (prediction of many possible future positions) into account.

[0040] In another aspect, the present disclosure is directed at a computer system, said computer system comprising a plurality of computer hardware components configured to carry out several or all steps of the computer implemented method described herein. The computer system can be part of a vehicle.

[0041] The computer system may comprise a plurality of computer hardware components (for example a processor, for example processing unit or processing network, at least one memory, for example memory unit or memory network, and at least one non-transitory data storage). It will be understood that further computer hardware components may be provided and used for carrying out steps of the computer implemented method in the computer system. The non-transitory data storage and / or the memory unit may comprise a computer program for instructing the computer to perform several or all steps or aspects of the computer implemented method described herein, for example using the processing unit and the at least one memory unit.

[0042] In another aspect, the present disclosure is directed at a vehicle comprising the computer system as described herein and a sensor configured to provide information for determining the at least one map.

[0043] In another aspect, the present disclosure is directed at a non-transitory computer readable medium comprising instructions for carrying out several or all steps or aspects of the computer implemented method described herein. The computer readable medium may be configured as: an optical medium, such as a compact disc (CD) or a digital versatile disk (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid state drive (SSD); a read only memory (ROM), such as a flash memory; or the like. Furthermore, the computer readable medium may be configured as a data storage that is accessible via a data connection, such as an internet connection. The computer readable medium may, for example, be an online data repository or a cloud storage.

[0044] The present disclosure is also directed at a computer program for instructing a computer to perform several or all steps or aspects of the computer implemented method described herein.

[0045] As described herein, a general motion prediction framework is provided. The system jointly predicts all the road users future motion in the scene, with its complexity invariant to number of road users. Furthermore, a network architecture is provided to enable efficient learning of the motions from various type and number of road users, who could have totally different maneuverability. For example, a pedestrian may change its direction rapidly but moves slower comparing to a vehicle.DRAWINGS

[0046] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following drawings, showing schematically: Fig. 1an illustration of a minimalistic static map consisting only of drivable areas and center lines; Fig. 2an illustration of the static map with a cell highlighted corresponding to possible future positions; Fig. 3an illustration of the static map, wherein the offset of a future position with respect to the cell center is illustrated; Fig. 4an illustration of the static context and the dynamic context according to various embodiments which may be used for predicting future positions of the ego-vehicle; Fig. 5an illustration of a whole scene prediction system according to various embodiments; Fig. 6an illustration of the output structure according to various embodiments for motion prediction of all objects from multiple classes; Fig. 7an illustration of the classification output to jointly predict future positions of multiple objects considering their multimodality according to various embodiments; Fig. 8an illustration of predictions and a target and an example of a gaussian kernel around the positive class label Fig. 9an illustration of a predicted position on the 2D grid in a cell and refinement of the predicted position by using an offset; Fig. 10an output portion of a trajectory prediction; Fig. 11a flow diagram illustrating a method for predicting properties (for example positions) of a plurality of objects in a vicinity of a vehicle according to various embodiments; Fig. 12a flow diagram illustrating a method for predicting properties (for example positions) of a plurality of objects in a vicinity of a vehicle according to various embodiments; Fig. 13an illustration of the system according to various embodiments; Fig. 14an illustration of a network architecture for a fixed static map; Fig. 15an illustration of a network architecture for a moveable static map; Fig. 16an illustration of the overall system architecture; Fig. 17an illustration of the un-rolled diagram; and Fig. 18a computer system with a plurality of computer hardware components configured to carry out steps of a computer implemented method as described herein. DETAILED DESCRIPTION

[0047] According to various embodiments, predicting the future positions (for example in 1 second, 2 seconds etc.) of the ego-vehicle or other objects (for example pedestrians or vehicles), given the past trajectory and surroundings may be provided.

[0048] As input data, the static context (for example including road boundaries, traffic signs, traffic lights) and / or the dynamic context (for example information about other traffic participants, including for example pedestrians, cars, trucks, and bikes) and / or ego dynamic (in other words: the dynamics of the ego-vehicle) may be available.

[0049] A problem, for which solutions are provided according to various embodiments, is that there does not exist only one possible future position, but a multitude of different possible future positions, depending e.g. on multiple possible paths, different acceleration patterns, interactions with other traffic participants etc.

[0050] According to various embodiments, the problem of motion prediction for various numbers and types of road user in complex driving scenarios is solved by treating this problem as an image generation problem. The methods and systems according to various embodiments may be applied in various driving scenarios, including, e.g., highway, urban, etc. The system complexity is invariant to the number and types of the road users in the application scene.

[0051] In contrast to methods and system which cover only one road user, the methods and system according to various embodiments cover all the road user. Thus, even for a plurality of road users, the method does not have to be repeatedly applied, which results in an improved system complexity. An empty highway with only one vehicle compared to a crowded urban intersection with lots of pedestrians and various type of vehicles will result in the same system requirements.

[0052] According to various embodiments, the method and system jointly predict all the road users motion in a scene and at the same time have a manageable complexity.

[0053] Fig. 1 shows an illustration 100 of a static map, including a road 104 with non-drivable areas (which may also be referred to as obstacles 102), and various possible positions 106, 108, 110, which may be assembled to a future trajectory 112.

[0054] A huge challenge to achieve the above task with Deep Learning tools is the nature of the training data. Various Deep Learning approaches are trained on trajectory data to achieve this task. However, trajectory data captured by sensor systems is highly imbalanced, since by far most of the trajectories follow straight lines with an approximately constant velocity. Without preprocessing of the data, the neural networks may learn only to predict straight lines. However, it is the abnormal behaviors (for example "unexpected stops", "accelerations", "turnings", "deviation from standard routes") which may be of interest. Such deviations from standard routes may be taken into account according to various embodiments.

[0055] According to various embodiments, many possible positions for multiple agents (multimodality) may be predicted, and a respective probability may be assigned to each of them.

[0056] The ability to predict multiple possible future positions of the ego-vehicle and other vehicles given the surrounding context while also estimating their probabilities may, for example, be used to provide safe autonomous driving.

[0057] According to various embodiments, devices and methods may be provided for prediction of different multimodalities, including determining probabilities which are assigned to the predictions.

[0058] According to various embodiments, the static (input) map (for example as illustrated in Fig. 1) may be (sub)divided into a grid with a plurality of cells, for example 10x10 cells. The output (in other words: output map) may first be an empty grid map, which does not necessarily corresponds to the exact same area as input static map. Only the cell number of the output map, where the ego-vehicle will be in the future, may be predicted. This is a classification task, where each class is a different cell of the grid. This could be called "coarse" prediction. It will be understood that the static information is not necessarily provided in the output grid map.

[0059] The classification task may be carried out for each pixel in the output grid map (wherein each pixel corresponds to one cell). The pixels may be indicate spaces occupied by any road users.

[0060] Fig. 2 shows an illustration 200 of the static map of Fig. 1, with a cell highlighted corresponding to the possible position 202.

[0061] The problem to assign a probability to each future prediction is solved according to various embodiments. The problem of imbalanced datasets is also solved according to various embodiments, since classification problems can cope with that problem much better (for example as described in more details when explaining the focal loss of Lin et. al.).

[0062] According to various embodiments, to refine the prediction to be more accurate, offsets to the actual positions may be provided (for example by the network employed according to various embodiments).

[0063] Fig. 3 shows an illustration 300 of the static map of Fig. 1 and Fig. 2, wherein a center 302 of the cell corresponding to the possible position 202 is illustrated. The offset may be provided relative to the center of each cell.

[0064] Providing an offset for each cell may make the regression problem much easier, since only small offsets to the actual positions are predicted.

[0065] The approach according to various embodiments was evaluated on real datasets and yielded results better than currently available.

[0066] According to various embodiments, a multi-class output structure may be provided. The grid output representation may allow for multi-modality prediction. Prediction of more than a single-target trajectory prediction may be provided, so as to provide a whole scene prediction system. By the use of the output structure according to various embodiments, the trajectory of a variable number of road users of different types at the same time for multiple time steps may be possible.

[0067] As described above, the surroundings of an ego vehicle may be divided into a 2D (two-dimensional) grid (for example a grid of size 80 x 80 with a resolution of 1 m) and the trajectories may be rasterized into an image-like structure. For each grid cell, the probability of a road user occupying it in the future may be predicted (which may be a classification task) for multiple time steps. By predicting an offset for each grid cell (which may be a regression task), an accuracy that is below the resolution of the 2D grid may be achieved.

[0068] The input data of the method according to various embodiments may be subdivided into the following major categories: static context, dynamic context, and dynamics of the ego-vehicle.

[0069] Fig. 4 shows an illustration 400 of the static context 402 and the dynamic context 404 according to various embodiments.

[0070] The static context may include static information or static objects (for example information or objects, which do not change from one time frame to another). For example, the static context may include lane and road boundaries, traffic signs, and / or traffic lights. Data of the static context may be available in today's vehicles; it may be provided either by state-of-the-art sensors or by offline maps combined with a localization of the vehicle, e.g. via GPS (global positioning system) or IMU (inertial measurement unit).

[0071] The dynamic context may include dynamic information or dynamic objects (for example information or objects, which may change or may be different from one time frame to another). For example, the dynamic context may include traffic participants like pedestrians, car, and / or bikes. The dynamic context may include some or all the measurements describing the dynamics of an object (for example coordinates, velocity, acceleration and / or yaw-rate). Data of the dynamic context may not only consist of the current measurement, but also may include past measurements. The accumulation of the past may allow for a refined assessment of the dynamics of the single objects. Data of the dynamic context may be available in today's vehicles; it may be provided by state-of-the-art sensors.

[0072] Furthermore, the class (for example "pedestrian", "bicycle", "car", "truck") of an object may be used as an additional input. This may take the different dynamics of traffic participants of different classes into account. This may further allow for a refined assessment of the dynamics of the single objects. This category of data may be available in today's vehicles, and may be provided by state-of-the-art sensors.

[0073] According to various embodiments, devices and methods may be provided which jointly predict future positions of multiple objects (of all the objects present in the input) for different future points in time, which may be possible due to the output structure according to various embodiments as described below.

[0074] Fig. 5 shows an illustration 500 of a whole scene prediction system according to various embodiments. For example, future positions may be predicted for a vehicle 502 and two pedestrians 508, 510. For example, a past trajectory 504 of the vehicle 502 may be used to determine potential future positions or future trajectories of the vehicle 502. For example, various positions 506 in a next point of time may be predicted; for example, the positions 506 illustrated in Fig. 5 may be the positions with the highest determined probability. For example, a pre-determined number of future trajectories may be determined (for example three as illustrated in Fig. 5 for the vehicle 502). Alternatively, a future position or future trajectory may be taken into account if the probability is higher than a pre-determined threshold, so that the number of possible future positions is flexible (depending on the probability of the future positions).

[0075] According to various embodiments, for the task of trajectory prediction, a grid based output structure may be defined which allows to jointly predict the trajectory of a variable number of road users of different types (pedestrian, vehicles, bicycles etc.) in a scene for multiple time steps.

[0076] The output structure may be defined as I T ×W×L×F< , where: ∘ T may describe the number of predicted future time steps ▪ T ∈ ℕ + , e.g. T = 2: Prediction for two future time steps (for example for 1 s and 2 s ahead). ∘ W x L may set the size of the 2D grid, defining the region of interest (ROI, which may be the region where it may be desired to get a trajectory prediction for all road users) ▪ Each grid cell (defined by W x L, with width W and length L) may correspond to a position where an object can be located at a future time step t. ▪ For example, a ROI may be 80 x 80 m around the ego vehicle. To calculate W and L the chosen resolution has to be considered. For example, if the resolution is 0.1 m, the dimension of (W × L) would be (800 x 800). ∘ F may define the output features for every grid cell: ▪ F = C o x o y cov where: C may be a set of classes, e.g.: C = c Pedestrian c Vehicle c Truck c Bicycle o x , o y may describe the offset in x and y direction from the centre of a grid cell to refine the actual position within a grid cell (or even outside the present cell). cov may be the covariance of x and y.

[0077] I(t, w, l, c) = p may describe the probability p ∈ [0,1] that the grid cell at (w, l) is occupied at the future time step t from an object of the class c.

[0078] Fig. 6 shows an illustration 600 of the output structure according to various embodiments for motion prediction of all objects from multiple classes (for example pedestrian and vehicle) for two time steps ahead. For demonstration purposes, a small ROI of 10 x 10 m with a resolution of 1m was chosen. It will be understood that the size of the ROI may be variable. The output structure according to various embodiments may also allow a variable number of classes and predicted time steps.

[0079] In Fig. 6, the time is shown along axis 602 and various features are illustrated as images along direction 604. Over the two time steps, a plurality of output images 606 may be provided.

[0080] Due to the randomness of many traffic situations, there may exist many possible future positions (which may be called modalities), which the system may account for. Each of the possible future positions may usually have different probabilities of occurrence. The method according to various embodiments may be accounting for the multi-modality by outputting a probability density defined on a grid, wherein each cell represents a specific location, as illustrated in Fig. 7. As described above, each cell may alternatively or additionally include the covariance of x and y. and may thus describe a probability density.

[0081] Fig. 7 shows an illustration 700 of the classification output to jointly predict future positions of multiple objects considering their multimodality according to various embodiments. In the left image 702, probabilities for positions 704 a time of 1 s in the future are illustrated. In the right image 706, probabilities for positions 708 a time of 2 s in the future are illustrated. The future positions may be assembled to trajectories. As illustrated in Fig. 7, several possible future trajectories may be identified.

[0082] According to various embodiments, the input includes trajectories of a plurality of road users, and the output is a grid map for occupancy. Each cell pixel in the prediction (i.e. the grid map for occupancy) may not have the corresponding relationship with the original track, so the assignment to the original track is lost. According to various embodiments, post-processing may be provided to assign the predicted grid cell to the original tracks to recover a predicted trajectory.

[0083] According to various embodiments, a loss function for (artificial) neural network (NN) based training may be provided. The loss function may be minimized during training; for example, parameters of the NN may be determined so that the loss function takes a minimum. The ground truth for each prediction time horizon may be considered as an image, wherein only at the image positions occupied by a road user, it has values (different from 0). The remaining pixels may be empty (in other words: have a value of 0).

[0084] According to various embodiments, the classification may be applied on a grid-cell-level where each grid cell (in the following also called pixel) is assigned a probability p of an object being there at time step t.

[0085] As a (classification) loss function, a pixel-wise distance aware cross entropy may be used, for example from the work of Law et al. (CornerNet: Detecting Objects as Paired Keypoints, ECCV 2018) which was originally used for the task of object detection.

[0086] In comparison to Law et al., according to various embodiments, the factor α may be added in the loss for the case of a positive label respectively (1 - α) for a label from the negative class based on the focal loss from Lin et al. work to cope with class imbalance (since most of the pixels may have a label from the negative class, meaning they will not be occupied in the future).

[0087] According to various embodiments, the classification loss DCE may be defined as follows: DCE p = − α ⋅ 1 − p γ ⋅ log p if y = 1 − 1 − α ⋅ 1 − y β ⋅ p γ ⋅ log 1 − p else wherein o y may be a label (in other words: ground truth; i.e. including information of the actual class): ▪ y ∈ 0 1 ▪ y = 1: Label of the positive class (e.g. pedestrian) ▪ y < 0: Label of the negative class (e.g. no pedestrian) ∘ p may be a probability of the specific class (e. g. pedestrian) at a pixel ∘ α may be a factor to handle class imbalance ∘ β, γ may be hyperparameters to weigh factors α, β, γ are hyper-parameters and may be tuned (for example may be defined empirically, for example by testing and tuning).

[0088] Around each positive label (y = 1), a gaussian kernel may be defined as illustrated in Fig. 8. When a prediction for the positive class is in the proximity of a positive label, 0 < y < 1, the loss is reduced by the factor (1 - y). The kernel's output value may be the value y.

[0089] Fig. 8 shows an illustration 800 of predictions 808 and a target 806 (in the left image 802) and an example of a gaussian kernel (in the right image 804) around the positive class label 806, which weighs false positive predictions.

[0090] The size of the gaussian kernel may be determined by its standard deviation σ. According to various embodiments, a σ which is based on the current velocity of an object and the time step of the prediction may be used: σ v x = σ max v max ⋅ v x + σ t wherein v may be the velocity of the object x, and σ t may be the initial σ based on the time step t.

[0091] The kernel may be a kernel other than a Gaussian kernel, and may not necessarily be related to a normal distribution. Thus, the kernel may be described using a covariance matrix. Thus, the axis may have different lengths, and it could be rotated. σ x may be different from σ y , which may also be rotated. The rotation may be determined by the heading / yaw of an object at a specific time. σ x , σ y and σ xy may be calculated from v x and v y .

[0092] The classification loss for a class may be summed over all pixels of the 2D grid and divided through the number of objects in the scene.

[0093] The distance aware cross entropy in combination with gaussian kernels around positive labels (for example based on velocity and how far in time a prediction is) may allow the artificial neural network to predict an uncertainty and encourages multimodality in the prediction e.g. at intersections where a vehicle could turn or drive straight.

[0094] According to various embodiments, further to the classification loss, a regression loss may be defined.

[0095] By predicting an offset in x and y from the centre of a pixel, the position may be refined and an accuracy below the resolution rate of the 2D grid may be obtained.

[0096] The regression loss may be defined as follows: R x x = x ^ − x 2 ⋅ η R y y = y ^ − y 2 ⋅ η R = R x + R y wherein ∘ R x , R y may be the regression loss for the offset in x and y direction, ∘ x̂, ŷ may be the ground truth offset in x and y direction, ∘ x, y may be the predicted offset in x and y direction, ∘ η may be an object indicator; for example, η = 1 if there is an object at that pixel, and η = 0 otherwise.

[0097] Fig. 9 shows an illustration 900 of a predicted position on the 2D grid in a cell 904 (in the left image 902) and refinement of the predicted position by using an offset 908 in x and y (in the right image 906).

[0098] According to various embodiments, the classification loss and the regression loss may be combined in a combined weighted loss.

[0099] Fig. 4 shows an input for trajectory prediction, and Fig. 5 shows an illustration of a trajectory prediction output from the prediction system for a scene for 2 time steps, showing static context-awareness especially for the highlighted vehicle approaching the curve. Fig. 5 shows multimodal trajectories. The past trajectory may be following a straight line. To make an accurate prediction, not just the past trajectory is considered, but also the static context of the object like the course of a road. The outputs are shown in illustration 1000 of Fig. 10.

[0100] Fig. 11 shows a flow diagram 1100 illustrating a method for predicting properties of a plurality of objects in a vicinity of a vehicle. At 1102, a grid map representation of road users perception data may be determined. The road users perception data may include or may be tracked perception results and / or untracked sensor intermediate detections. At 1104, a grid map representation of static environment data may be determined based on data obtained from a perception system and / or a pre-determined map. At 1106, the properties of the plurality of objects may be determined based on the grid map representation of road users perception data and the grid map representation of static environment data.

[0101] According to various embodiments, the properties of the plurality of objects may be determined as a function of at least one variable point in time for which the properties are determined.

[0102] According to various embodiments, the properties of the plurality of objects may be determined in the form of Y(t) = At 2< + Bt 1< + C, wherein Y(t) denotes the properties (as a function of t), t denotes at least one point in time for which the properties are determined, and A, B, and C are constants, which are preferably used for sampling or interpolation or extrapolation at the variable time t, and which are preferably determined as variable output of a network.

[0103] According to various embodiments, the values of t may be discrete values taken from a set T, and the properties may then be given by Y = {C t }, t ∈ T, as described above.

[0104] According to various embodiments, the properties may be determined for a past point in time and / or a present point in time and / or a future point in time.

[0105] According to various embodiments, the properties of the plurality of objects may include or may be a probability matrix, wherein each entry of the probability matrix is related to a respective pre-determined region in the vicinity of the vehicle and indicates a probability of an object being present in the respective region at a time different from the pre-determined time; wherein the method may further include determining an offset to the center of the region for each of the entries of the probability matrix; wherein preferably the offsets are determined using a regression method.

[0106] According to various embodiments, the grid map representation of road users perception data may be determined further based on data from a perception system; and / or the grid map representation of static environment data may be determined further based on a pre-determined map.

[0107] According to various embodiments, the grid map representation of road users perception data may be encoded using a dynamic context encoder; wherein the grid map representation of static environment data may be encoded using a static context encoder; and wherein the properties of the plurality of objects may be decoded using a decoder.

[0108] According to various embodiments, the classification method may include or may be a machine learning method, preferably an artificial neural network; wherein preferable the machine learning method is trained based on training data, the training data including or being traffic situations of a plurality of objects moving over a plurality of time frames.

[0109] According to various embodiments, the properties of the plurality of objects may be determined for a plurality of classes, wherein the plurality of object classes may include or may be at least two of an ego vehicle class, a pedestrian class, a bicyclist class, a vehicle class, a truck class, a lorry class, a fast moving object class, or a slow moving object class.

[0110] Fig. 12 shows a flow diagram 1200 illustrating a method for predicting positions of a plurality of objects in a vicinity of a vehicle according to various embodiments. At 1202, at least one map including a representation of the vicinity of the vehicle at a pre-determined time may be determined. At 1204, a classification method may be applied to the at least one map, the classification method providing as an output a probability matrix, wherein each entry of the probability matrix is related to a respective pre-determined region in the vicinity of the vehicle and indicates a probability of an object being present in the respective region at a time different from the pre-determined time.

[0111] According to various embodiments, the at least one (input) map may include or may be a map indicating a static context of the vicinity.

[0112] According to various embodiments, the at least one (input) map may include or may be a map indicating a dynamic context of the vicinity.

[0113] According to various embodiments, the classification method may include or may be a machine learning method, preferably an artificial neural network.

[0114] According to various embodiments, the machine learning method may be trained based on training data, the training data comprising traffic situations of a plurality of objects moving over a plurality of time frames.

[0115] According to various embodiments, the machine learning method may be trained based on a loss function, wherein the loss function preferably includes or is a pixel-wise distance aware cross entropy.

[0116] According to various embodiments, the classification method may determine a plurality of probability matrices, each probability matrix related to one of a plurality of object classes.

[0117] According to various embodiments, the plurality of object classes may include or may be at least two of a pedestrian class, a bicyclist class, a vehicle class, a truck class, a lorry class, a fast moving object class, or a slow moving object class.

[0118] According to various embodiments, the probability matrix may be related to a center of the respective pre-determined region, and the method may further include determining an offset to the center of the region for each of the entries of the probability matrix.

[0119] According to various embodiments, the offset may be determined based on a regression loss.

[0120] According to various embodiments, a plurality of probability matrices may be determined related to a plurality of times.

[0121] According to various embodiments, the method may further include determining an occupation matrix, wherein each entry of the occupation matrix may be related to a respective pre-determined region in the vicinity of the vehicle and indicates whether or not an object is present at the region, and wherein a corresponding entry of the probability matrix may only be determined if the entry of the occupation matrix indicates that an object is present at the region.

[0122] Each of the steps 1102, 1104, 1106, 1202, 1204, and the further steps described above may be performed by computer hardware components.

[0123] According to various embodiments, a method for predicting future positions of all registered (by a sensor) objects may be provided.

[0124] According to various embodiments, not only coordinates are used, but all measurements (e.g. velocity, acceleration, etc.). Furthermore, the past measurements, e.g. velocity, acceleration and the classes of single objects, e.g. car, bike, pedestrian may be used.

[0125] According to various embodiments, the occupancy grid or cost map is not only based on registered coordinates, but on dynamic context and static context, including the class of the objects. The occupancy grid or cost map according to various embodiments is made for future points in time, e.g. in 1 second, 2 seconds etc. Hence, not only the current presence and / or vicinity of objects is described, but also the future ones. Furthermore, the class of each predicted object is determined.

[0126] According to various embodiments, the output layer is subdivided into cells and a cost value is assigned to each cell, and in addition, the exact positions of the objects may be refined by using additional offsets.

[0127] Fig. 13 shows an illustration 1300 of the system according to various embodiments. The inputs may include the dynamic context 1302, the static context 1304, and the ego dynamics 1306. A deep neural network 1308 may be used to generate the output 1312. When training the deep neural network 1308, the output 1312 and the ground truth (GT) 1310 may be provided to a loss function 1314 for optimization of the deep neural network 1308.

[0128] The dynamic context 1302 may include a series of images, which may describe the road users' movements in the past time frames. Possible sources for the dynamic context image are rasterized tracks of road users detected by sensors, or a dynamic occupancy grid which may be detected by sensors such as camera, radar or lidar to represent the areas to represent the occupancy and velocities of an area (grid cell).

[0129] Possible representations and sources of the dynamic context image may include raw detections from the sensors, for example detected bounding boxes from camera (projected on the ground plane) or lidar sensors (with a similar approach for rasterization to images; multiple close or overlapped bounding boxes can belong to the same road user), or for example radar reflection points from moving objects (with a similar approach for rasterization to images). By doing so, "tracking" in the perception system may not be needed anymore. The network may work with raw sensor detections directly.

[0130] The static context 1304 may be represented by one or more of the following: Rasterized image from the HD (high definition) map, wherein the high definition map may cover the accurate lane markings position, so a vehicle's position may be accurately localized in the lanes; Drivable area from sensors (for example a grid map or image data structure, wherein each pixel in this image may represent the drivability of the specific area in the field. This term may refer to the detected drivable area from sensors, such as camera, lidar or radar); Lane / road detection from sensors (for example using the detected lane markings, road boundaries, guard rails, etc from the sensor, one can also build a grip map or image like data (similar to rasterized Map) to describe the static context); Static occupancy grid map (for example similar to "drivable area", an occupancy grid map (detected from sensor) could be used as static context input).

[0131] For the ego dynamic 1306, the ego vehicle may be represented as one road user and thus included in the dynamic context input. Depending on the output, when focused on prediction of Ego vehicle, the ego vehicle's data may be formulated also as: vector, or separated image input, or separated channel of the dynamic context input.

[0132] The output 1312 may provide future positions of all road users. The output 1312 may be any function in time.

[0133] According to various embodiments, the output 1312 may have the following form Y t = A t 2 + B t 1 + C .

[0134] Y may denote the final output, and may be formulated in relation to t, so that prediction may be carried out at any given time. It will be understood that the 2nd-order function may serve as an example here.

[0135] The term "predict" may generally be used in artificial NN (neural networks) and it may mean that a NN takes an input and gives an output. As such, prediction does not necessarily indicate a temporal relation between input and output, while for the trajectory prediction according to various embodiments, prediction may mean anticipate something in the future. It will be understood that prediction as used herein may have the meaning of giving an estimation.

[0136] t may be the prediction time horizon. t may be negative, 0, or positive.

[0137] Negative t may mean estimation for the past. For example, we may classify the past maneuvers when more current data are given. Similar to the concept of "Kalman smoothing".

[0138] Zero t may mean estimation for the current time frame. if we are using raw sensor detections to create inputs, then prediction the trajectory at t = 0 actually works as a "tracking" method.

[0139] Positive t may mean prediction for the future.

[0140] A ,B ,C may be different (from each other) and may be predicted 3D matrices. Each of A, B, C may represent the output grid map with size WxL and the 3 rd< dimension describes the various features it can predict (for example occupancy probability, in grid offsets, covariances of the predicted positions, possible maneuvers, or other features such as future velocity, for example).

[0141] With the output formulated as described above, a continuous prediction formulation regarding to time horizon t may be provided. It will be understood that the output as described above is merely an example, and that higher order or lower order (or any other non-linear function) may be used as an output.

[0142] According to various embodiments, the values of t may be discrete values taken from a set T, and the properties may then be given by Y = {C t }, t ∈ T, as described herein.

[0143] According to various embodiments, the second order output Y(t) = A*t^2 + B*t + C may be provided as a first order with variable prediction horizon: Y(t) = B*t + C. Herein, the two outputs may be B and C. B may describe the part which is depending on t (thus moving), and C describes the part which is independent from t (thus static). In other words: B describes the agents which are moving, and C describes agents which are standing (i.e. not moving).

[0144] The output 1312 may describe future ego positions. When the ego vehicle's past trajectory is also given in the input dynamic context image (as described above), then the output can also cover the predictions for ego vehicle.

[0145] The ground truth 1310 may define the task of the artificial neural network. It may cover positions as occupancy probability and in-grid offsets or other regression and classification tasks, for example future positions, velocity, maneuvers etc. of the road users in the scenes as image.

[0146] Fig. 14 shows an illustration 1400 of a network architecture for a fixed static map. The dynamic context input 1402 may be provided to a recurrent neural network (RNN) 1404, the output of which, together with the static context input 1406, may be provided to a dilated neural network (DNN) 1408, the output of which may be provided to a decoder 1410 to provide the overall output.

[0147] Fig. 15 shows an illustration 1500 of a network architecture for a moveable static map, the network architecture not being part of the invention. Compared to the network architecture of Fig. 14, both the dynamic context input 1402 and the static context input 1404 are provided to the RNN 1404, the output of which is provided to the DNN 1408.

[0148] An exemplary urban traffic scenario may have various types of vehicles with different motions, variant number of pedestrians with different behaviors and motions. All these road users may have direct or indirect interactions among them, and at the same restricted by the static context: road / lane structures, etc. There may be an infinite number of different scenarios in urban traffic, which may be a challenging problem.

[0149] According to various embodiments, images may be used as system inputs and outputs. The output images may contain all the road users' future trajectories. Thus, according to various embodiments, a trajectory prediction problem may be converted to an image generation problem. This input and output data representation may allow invariance to the number and types of the road user, and invariance to specific static-context.

[0150] According to various embodiments, an HD map rasterization may be provided. The information of an HD map may be grouped into different categories (e.g. walking areas, driving areas, lane dividers and crossing areas) and assigned each category an own RGB (red-green-blue) color. The rasterization may be done category-wise. One category may be rasterized on top of each other, starting with larger areas like driving areas and ending with smaller ones e.g. lane dividers.

[0151] According to various embodiments, a trajectory rasterization may be provided. A road user's trajectory may be defined as a series of points at continuous time frames. Each point includes features, such as: target position (x,y), velocities, accelerations, types, size, orientation.

[0152] A fixed region of interest (ROI) around the ego vehicle may first be rasterized as an empty multi-channel image, with each pixel covers a fixed area. For example, an 80x80 meters of ROI in front of the ego vehicle can be rasterized into an 80x80 pixel image, with each pixel represents a 1x1 meter square area. It will be understood that the x and y resolutions can be different.

[0153] At each time frame, all road users' positions may be rasterized as pixel positions in the image, and their trajectory features at this time frames may be stored at their corresponding pixels.

[0154] The above process can be repeated for all continuous time frames, thus a series of images may be generated representing all road users' trajectories.

[0155] During above process, each road user may be considered as a point-object, thus only located in one pixel. Alternatively, the road user's shape, size and orientation can all be considered, thus rasterized into multiple connected pixels.

[0156] Because of the input and output formulation, the motion prediction of a whole scene may be converted to an image generation problem.

[0157] Fig. 16 shows an illustration 1600 of the overall system architecture. Various portions are similar or identical to the architecture shown in Fig. 14, so that the same reference signs may be used and duplicate description may be omitted. It will be understood that the specific input size and number of network layers shown in Fig. 16 are for illustrative purpose only.

[0158] At each past time frame, the positions of all road users may be represented in a multi-channel image. A series of such images may represent the motion of the whole scene in the temporal domain.

[0159] CNNs are good at learning the correlation among the data under its kernels. In the input, such data correlation may be intuitively understood as the possible interactions among road users and the subsequent effects on their behaviors and trajectories. Thus, CNN may be provided to build the trajectory encoder. A multilayer of CNN may be applied on the image data, and this network may be referred to as trajectory encoder.

[0160] Map integration into the prediction system may be provided using CNN to extract features from the map and jointly learned with the trajectory prediction. This network may be referred to as map encoder.

[0161] According to various embodiments, skip connections with ConvLSTM may be provided.

[0162] As the trajectory data may be a series of images processed by the trajectory encoder, whose output is also a series of feature maps, a convolutional recurrent neural network(ConvRNN), for example a ConvLSTM, may be applied to learn the motion in the temporal domain.

[0163] Because various types of road users have different maneuverability, their range of motion at a fixed time horizon may be very different. For example, a pedestrian may move 1 meter at 1 second time horizon, but a vehicle may have moved 10 meters.

[0164] Also the interaction among the road users may happen at different scales. Pedestrian may already avoid bump into each other a few meters in advance, but vehicles may require longer distance. Thus, the final CNN layer with a wide receptive field may not be optimal to capture all these different motions and interactions.

[0165] Considering the different receptive fields of different CNN layers, "skip connections" may be provided by giving each trajectory encoder layer's output to trajectory decoder. RNN (convLSTM) may be integrated at the skip connections for time series data processing. This makes this architecture from a single image process to a time series data process framework. Static context features may be integrated (for example by concatenation) with the output features after the convLSTM also through the skip connections. This may allow using different rasterization resolution, or different layers of CNN for map encoder. If the rasterized map and trajectory images have the same resolution and size, they may be concatenated and given as input to the trajectory decoder.

[0166] According to various embodiments, a trajectory decoder may be provided. To generate the output image, the transposed convolution network may be applied. Through the skip connections, the time series trajectory (images) data may be passed to the decoder at different levels and processed to the final output.

[0167] The output at each prediction time horizon may include: An image, which may be denoted as I t , which may represent the predicted position at future time horizon t. I t may have N different channels, denoted as I t n< , wherein each channel may present the prediction for one type of road user, such as pedestrian or vehicle. The pixel value, between [0,1], of the image may represent the possibility (or probability) of that pixel being occupied. A two-channel image O t , the pixel values of which may represent the in-pixel x and y offsets when this pixel is predicted as the future position in I t . This may be because the input and output are all rasterized images, each pixel in such images may have a fixed and limited resolution. For example, one pixel may represent a 1m x 1m area in real world. To achieve better accuracy, the in-pixel offsets may be predicted also as a two-channel image. This in-pixel offsets may be regardless of the specific type of road users. According to various embodiments, for each I t n a specific offset O t may be provided. The offsets may be for all the types of road users.

[0168] According to various embodiments, iterative outputs may be provided. Instead of predicting at fix time horizons, the system according to various embodiments may use a customized ConvLSTM. One difference of this ConvLSTM to a conventional one is that it only needs input at first iteration. At each iteration, it outputs a prediction at +Δt future time horizon. Δt is the time interval between two consecutive prediction time horizons. Thus, when iteratively running this last output layer, a series of predictions at Δt, 2Δt, 3Δt, ... may be obtained. This flexibility may allow the users to decide how far in the future he wants to predict. This customized ConvLSTM may require only input at the beginning.

[0169] Fig. 17 shows an illustration 1700 of the un-rolled diagram. A conv LSTM layer 1702 may be the last output layer of the decoder, and, in an unrolled view, a plurality of conv LSTM layers 1704, 1706, 1708 may be provided to provide a plurality of outputs I_1, I_2, I_3. The input C shown on the left side of Fig. 17 may be the feature output of preview NN (neural network) layers. Thus, C may be provided to the convLSTM and may iteratively be decoded to multiple output grid maps. The layers 1702, 1704, 1706, 1708 are actually the same convLSTM 1702, which are illustrated in this way for demonstration of "iteratively calling". Thus, the layers 1704, 1706, 1708 are actually all 1702, but at different (i.e. subsequent) iterations.

[0170] Fig. 18 shows a computer system 1800 with a plurality of computer hardware components configured to carry out steps of a computer implemented method as described herein. The computer system 1800 may include a processor 1802, a memory 1804, and a non-transitory data storage 1806. Various sensors (not shown in Fig. 18) may be provided as part of the computer system 1800, or may be provided external to the computer system 1800.

[0171] The processor 1802 may carry out instructions provided in the memory 1804. The non-transitory data storage 1806 may store a computer program, including the instructions that may be transferred to the memory 1804 and then executed by the processor 1802.

[0172] The processor 1802, the memory 1804, and the non-transitory data storage 1806 may be coupled with each other, e.g. via an electrical connection 1808, such as e.g. a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals.

[0173] The terms "coupling" or "connection" are intended to include a direct "coupling" (for example via a physical link) or direct "connection" as well as an indirect "coupling" or indirect "connection" (for example via a logical link), respectively.

[0174] It will be understood that what has been described for one of the methods above may analogously hold true for the computer system 1800.

[0175] As described herein, a general motion prediction framework, for example including a machine learning system using dynamic and static context inputs may be provided for estimation of various features of the past, current and future.

[0176] As described herein, according to various embodiments, the trajectory prediction may be formulated as image generation problem, thus making it possible to predict the whole scene motion in a general applicable framework.

[0177] CNN may be chosen for trajectory and map encoding, which may efficiently learn the interaction among the road users (dynamic context) and the interaction with the static context (static context).

[0178] The architecture may consider that road users have different maneuverability. RNN may be integrated into the system (in addition to CNN based encoders and decoder) to capture the road users' dynamic in temporal domain.

[0179] An iterative output layer provided in the decoder may provide the flexibility of variable prediction horizons.

[0180] The methods and systems according to various embodiments provide good scalability (for example, simple to complex scenarios may be handled with the same system). The system complexity may be invariant to road user number and types, with a relatively fixed system computational requirement, which may be important for functional safety.

[0181] According to various embodiments, a general system architecture and general input representation may be provided, which may take various types of input data, for example including tracked objects and / or raw detections, for various types of applications, including tracking, fusion and prediction.

[0182] Various embodiments may provide (dynamic and static) context awareness per ML (machine learning) for trajectory prediction covering various complex driving scenarios.

[0183] Various embodiments may provide efficient prediction, for example invariant to the number of road users.Reference numeral list

[0184] 100illustration of a static map 102obstacle 104road 106possible position 108possible position 110possible position 112future trajectory 200illustration static map 202possible position 300illustration of static map 302center of cell 400illustration of static context and dynamic context 402static context 404dynamic context 500illustration of a whole scene prediction system according to various embodiments 502vehicle 504past trajectory 506various positions 508pedestrian 510pedestrian 600illustration of the output structure according to various embodiments 602axis 604direction 606output images 700illustration of classification output according to various embodiments 702left image 704positions 706right image 708positions 800illustration of predictions, a target and an example of a gaussian kernel 802left image 804right image 806target 808predictions 900illustration of a predicted position and refinement of the predicted position 902left image 904cell 906right image 908offset 1000illustration of outputs 1100flow diagram illustrating a method for predicting properties of a plurality of objects in a vicinity of a vehicle 1102step of determining a grid map representation of road users perception data 1104step of determining a grid map representation of static environment data 1106step of determining the properties of the plurality of objects 1200flow diagram illustrating a method for predicting positions of a plurality of objects in a vicinity of a vehicle according to various embodiments 1202step of determining at least one map including a representation of the vicinity of the vehicle at a pre-determined time 1204step of applying a classification method to the at least one map 1300illustration of the system according to various embodiments 1302dynamic context 1304static context 1306ego dynamics 1308deep neural network 1310ground truth 1312output 1314loss function 1400illustration of a network architecture for a fixed static map 1402dynamic context input 1404recurrent neural network 1406static context input 1408dilated neural network 1410decoder 1500illustration of a network architecture for a moveable static map 1600illustration of overall system architecture 1700illustration of un-rolled diagram 1702conv LSTM layer 1704conv LSTM layer 1706conv LSTM layer 1708conv LSTM layer 1800computer system according to various embodiments 1802processor 1804memory 1806non-transitory data storage 1808connection

Claims

1. Computer implemented method for predicting properties of a plurality of objects in a vicinity of a vehicle (502), wherein the properties comprise positions, velocities, heading angles, or collision risk values, the method comprising the following steps carried out by computer hardware components: - determining (1102) a grid map representation of road users perception data, the road users perception data comprising tracked detections and / or untracked detections based on radar data or camera data; - determining (1104) a grid map representation of static environment data based on sensor data provided by a sensor and / or a pre-determined map; and - determining (1106) the properties of the plurality of objects based on the grid map representation of road users perception data and the grid map representation of static environment data; wherein the grid map representation of road users perception data is encoded using a dynamic context encoder; wherein the grid map representation of static environment data is encoded using a static context encoder; wherein the properties of the plurality of objects are decoded using a decoder; characterized in that an output of the dynamic context encoder is an input of a recurrent neural network (1404) provided between the dynamic context encoder and the static context encoder, and an input of the static context encoder is an output of the recurrent neural network (1404), and an output of the static context encoder is an input of a dilated neural network (1408) provided between the static context encoder and the decoder (1410), and an input of the decoder (1410) is an output of the dilated neural network (1408).

2. The computer implemented method of claim 1, wherein the properties of the plurality of objects are determined as a function of at least one variable point in time for which the properties are determined.

3. The computer implemented method of claim 2, wherein the properties of the plurality of objects are determined based on the formula of Y(t) = At2 + Bt1 + C, wherein Y denotes the properties, t denotes at least one point in time for which the properties are determined, and A, B, and C are constants, which are preferably used for sampling or interpolation or extrapolation at the variable time t, and which are preferably determined as variable output (1312) of a machine learning method, preferably an artificial neural network (1308).

4. The computer implemented method of claim 2, wherein the properties of the plurality of objects can also be determined at discrete points in time.

5. The computer implemented method of at least one of claims 1 to 4, wherein the properties are determined for a past point in time and / or a present point in time and / or a future point in time.

6. The computer implemented method of at least one of claims 1 to 5, wherein the properties of the plurality of objects comprises a probability matrix, wherein each entry of the probability matrix is related to a respective pre-determined region in the vicinity of the vehicle (502) and indicates a probability of an object being present in the respective region at time t; wherein the computer implemented method further comprises determining an offset (908) to the center of the region for each of the entries of the probability matrix; wherein the offset (908) is provided in x and y direction from the center (302) of a grid cell (904) of the probability matrix to refine the actual position within a grid cell (904) or outside the grid cell (904); wherein preferably the offsets (908) are determined using a regression method.

7. The computer implemented method of at least one of claims 1 to 6, wherein the properties of the plurality of objects are determined for a plurality of classes, wherein the plurality of object classes comprise at least two of an ego vehicle class, a pedestrian class, a bicyclist class, a vehicle class, a truck class, a lorry class, a fast moving object class, or a slow moving object class.

8. The computer implemented method of at least one of claims 1 to 7, wherein the properties of the plurality of objects comprise at least one of positions, speeds, heading angles, intentions, or collision risk values.

9. The computer implemented method of at least one of claims 1 to 8, wherein the computer implemented method is used for multimodal future position prediction.

10. Computer system (1800), the computer system (1800) comprising a plurality of computer hardware components configured to carry out steps of the computer implemented method of at least one of claims 1 to 9.

11. Vehicle (502) comprising: the computer system (1800) of claim 10; and a sensor configured to provide sensor data, wherein the grid map representation of road users perception data and / or the grid map representation of static environment data is determined based on the sensor data.

12. Non-transitory computer readable medium comprising instructions for carrying out the computer implemented method of at least one of claims 1 to 9.