Methods and systems for predicting characteristics of a plurality of objects in a vicinity of a vehicle

Abstract

The present disclosure provides methods and systems for predicting characteristics of a plurality of objects in a vicinity of a vehicle. A computer-implemented method of predicting characteristics of a plurality of objects in a vicinity of a vehicle, the method comprising the following steps performed by computer hardware components: determining a grid map representation of road user perception data, the road user perception data comprising tracked perception results and / or untracked sensor intermediate detections; determining a grid map representation of static environment data based on data obtained from a perception system and / or based on a predetermined map; and determining the characteristics of the plurality of objects based on the grid map representation of road user perception data and the grid map representation of static environment data.

Description

Technical Field

[0001] This disclosure relates to methods and systems for predicting the characteristics of multiple objects in the vicinity of a vehicle. Background Technology

[0002] Predicting the future location of an object is an important task in various applications, such as (at least partially) autonomous driving.

[0003] Therefore, there is a need to provide an effective and reliable method for predicting the future positions of one or more objects in a scene. Summary of the Invention

[0004] This disclosure provides a method implemented by a computer, a computer system, and a non-transitory computer-readable medium. Embodiments are shown in the specification and accompanying drawings.

[0005] In one aspect, this disclosure relates to a computer-implemented method for predicting the characteristics of multiple objects in a neighborhood of a vehicle, the method comprising the steps performed by computer hardware components: determining a grid map representation of road user perception data, the road user perception data including tracked perception results and / or untracked sensor intermediate detections; determining a grid map representation of static environment data based on data obtained from a perception system and / or based on a predetermined map; and determining the characteristics of the multiple objects based on the grid map representation of the road user perception data and the grid map representation of the static environment data.

[0006] Untracked intermediate sensor detections can include, for example, radar data or camera data. For radar data, radar detections can include points, each a reflection of radio waves emitted by the radar. Metallic objects have strong reflectivity, resulting in a larger number of detection points. These points can be located at any random position on the vehicle, and the number of detected points is variable and random. For camera data as intermediate detections, the method can act on the image and typically detect multiple bounding boxes around the target (e.g., the vehicle), where these boxes may differ only slightly from each other but all originate from the same object. In both radar and camera data, these detections can be tracked, for example, through a Kalman filter, and only after tracking can the trajectory corresponding to the vehicle be obtained, where some physical characteristics, such as position or velocity, are estimated. Untracked intermediate sensor data can include data that has not undergone such tracking.

[0007] According to another aspect, the characteristics of the plurality of objects are determined as a function of at least one variable time point that determines the characteristics.

[0008] According to another aspect, the characteristics of the plurality of objects are as follows: The form is used to determine the characteristic, where Y(t) represents the characteristic (as a function of t), t represents at least one time point in time that determines the characteristic, and A, B, and C are constants that are preferably used for sampling, interpolation, or extrapolation at variable time t, and are preferably determined as the variable output of the network.

[0009] According to another aspect, characteristics can also be determined for discrete time (thus eliminating A and B), where t can be the time obtained from the point set T. In the discrete form, the output can be a set of characteristics predicted by the network (for each of these discrete and fixed times), and can be represented as... .

[0010] According to another aspect, the characteristics are determined with respect to past time points and / or current time points and / or future time points.

[0011] According to another aspect, the characteristics of the plurality of objects include a probability matrix, wherein each entry of the probability matrix relates to a corresponding predetermined region of the vehicle’s neighborhood and indicates the probability of an object being present in the corresponding region at a time other than a predetermined time; wherein the computer-implemented method further includes determining an offset relative to the center of the region for each entry of the probability matrix; wherein, preferably, a regression method is used to determine the offset.

[0012] According to another aspect, a grid map representation of the road user perception data is further determined based on data from the perception system, and / or a grid map representation of the static environment data is further determined based on a predetermined map.

[0013] According to another aspect, a dynamic context encoder is used to encode the grid map representation of the road user-perceived data; wherein a static context encoder is used to encode the grid map representation of the static environment data; wherein a decoder is used to decode the characteristics of the plurality of objects. For example, a recurrent neural network is provided between the dynamic context encoder and the static context encoder, and an extended neural network is provided between the static context encoder and the decoder; or a recurrent neural network and an extended neural network are provided between the static context encoder and the decoder.

[0014] According to another aspect, the computer-implemented method is a classification method including a machine learning method, preferably an artificial neural network, preferably the artificial neural network including multiple skip connections with time filtering in a pyramid neural network architecture; wherein, preferably, the machine learning method is trained based on training data, the training data including traffic conditions of multiple objects moving across multiple time frames.

[0015] According to another aspect, the characteristics of the plurality of objects are determined for a plurality of object classes, wherein the plurality of object classes include at least two of the following: self-vehicle class, pedestrian class, cyclist class, vehicle class, truck class, van class, fast-moving object class, and slow-moving object class.

[0016] According to another aspect, the characteristics of the plurality of objects include at least one of position, velocity, heading angle, intent, and collision risk value.

[0017] According to another aspect, the computer-implemented method is used for fusion or for prediction. For example, the input can come from various sensors, and then the method can be used as a fusion method (e.g., in a fusion system).

[0018] In another aspect, this disclosure relates to a computer-implemented method for predicting characteristics (e.g., position, speed, heading angle, driving intention, or collision risk value) of multiple objects (which may be, for example, multiple classes) in the vicinity of a vehicle. The method includes the following steps performed by computer hardware components: determining at least one map including a representation of the vehicle's vicinity at a predetermined time; and determining a time function based on the at least one input map, the time function indicating the probability of an object being present in the corresponding region at a time t different from the predetermined time, wherein the time function includes a constant portion determined by applying a classification method to the at least one input map, the classification method providing a probability matrix as output, wherein each entry of the probability matrix relates to a corresponding predetermined region of the vehicle's vicinity and indicates the probability of an object being present in the corresponding region at a time t different from the predetermined time, wherein the classification method determines multiple time functions and multiple probability matrices, each time function and each probability matrix relating to one of multiple object classes, each class including multiple objects.

[0019] In other words, multimodal future location predictions based on given contextual information can be provided (e.g., multimodal future location predictions for multiple agents). It should be understood that "multimodal" future location prediction can mean that multiple future locations are feasible for a single agent. A technique can be provided for predicting the future locations of multiple objects for different future points in time.

[0020] According to another aspect, the at least one map includes a map indicating a static context of the adjacent area. For example, a static context can represent immobile objects (e.g., roads, buildings, or trees). A static context can represent an environment that restricts the movement of road users and is unaffected by them. For example, a static context can include immobile objects (e.g., roads, buildings, or trees). A static context can also include traffic rules. A static context can include dynamic objects, such as standard traffic lights or weather conditions. All these static contexts affect road users, and the behavior of road users does not affect the static context.

[0021] According to another aspect, the at least one map includes a map indicating the dynamic context of the adjacent area. The dynamic context can represent moving objects, such as other traffic participants.

[0022] A dynamic context can include the state of multiple road users as a whole, and it can also include ego vehicles. If you want to predict the future trajectory of a particular road user, you can consider all other road users as the dynamic context of that particular road user.

[0023] According to various implementation methods, all road users can be predicted jointly, thus treating their overall trajectory as a dynamic context of the scene.

[0024] The movement of a self-vehicle can be part of a dynamic context and can be equally considered as a dynamic context input to other road users in the grid map.

[0025] Furthermore, the movement of the egovehicle can also be indirectly used for data preprocessing. According to the egovehicle perception system, the trajectories of other road users can be provided in the egovehicle coordinate system (VCS). Since the egovehicle can be in motion, perceptions at various frames can be represented in the coordinate system in which it is moving. According to various implementations, ego-dynamics (which may be or may include the movement of the egovehicle) can be used to compensate for and transform the trajectories of other users into the world coordinate system (WCS); in this case, trajectory data from multiple past time frames can have the same fixed coordinate system. Thus, ego-dynamics can be indirectly used as input (e.g., for preprocessing perception results of other road users).

[0026] According to another aspect, the classification method includes a machine learning method, preferably an artificial neural network. The machine learning method can be trained using the recorded motion scene.

[0027] According to another aspect, the machine learning method is trained based on training data, which includes traffic conditions of multiple objects moving across multiple time frames.

[0028] According to another aspect, the machine learning method is trained based on a loss function, wherein the loss function preferably includes pixel-level distance-aware cross-entropy.

[0029] According to another aspect, the classification method determines multiple probability matrices, each of which is associated with one of multiple object classes.

[0030] According to another aspect, the plurality of object classes includes at least two of the following: self-vehicle class, pedestrian class, cyclist class, vehicle class, truck class, van class, fast-moving object class, and slow-moving object class.

[0031] According to another aspect, the probability matrix is ​​related to the center of the corresponding predetermined region, and the computer-implemented method also includes the following steps performed by computer hardware components: determining the offset relative to the center of the region for each entry of the probability matrix.

[0032] From another perspective, the offset is determined based on regression loss. The determination of the offset (based on regression loss) reduces the difference between the predicted and actual locations by allowing for a higher spatial resolution than that provided by the cells (in other words, pixels) of the image or matrix used in the classification method.

[0033] According to another aspect, multiple probability matrices are determined for multiple times that are different from the predetermined time.

[0034] According to another aspect, the computer-implemented method further includes the following steps performed by the computer hardware components: determining an occupancy matrix, wherein each entry of the occupancy matrix relates to a corresponding predetermined region of the vehicle's neighborhood and indicates whether an object is present in that corresponding predetermined region, and wherein the corresponding entry of the probability matrix is ​​determined only if an entry of the occupancy matrix indicates the presence of an object in that corresponding predetermined region. This has been found to improve efficiency.

[0035] In a general approach, this method can support trajectory prediction for any number of road users. The computational workload is independent of the number of road users, which is a significant advantage compared to existing techniques. This method can support predictions for various types of traffic participants, such as cars, pedestrians, and bicycles, and can consider multimodal factors (predicting many possible future locations).

[0036] In another aspect, this disclosure relates to a computer system comprising a plurality of computer hardware components configured to perform some or all of the steps of the computer-implemented methods described herein. The computer system may be part of a vehicle.

[0037] A computer system may include multiple computer hardware components (e.g., a processor, such as a processing unit or processing network, at least one memory, such as a memory cell or memory network, and at least one non-transitory data storage device). It should be understood that additional computer hardware components may be provided and used to perform the steps of the computer-implemented methods within the computer system. The non-transitory data storage and / or memory cell may include computer programs that instruct the computer, for example, to use the processing unit and at least one memory cell to perform some or all of the steps or aspects of the computer-implemented methods described herein.

[0038] In another aspect, the present invention relates to a vehicle comprising a computer system as described herein and sensors configured to provide information for determining the at least one map.

[0039] In another aspect, this disclosure relates to a non-transitory computer-readable medium comprising instructions for performing several or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium may be configured as: an optical medium, such as an optical disc (CD) or digital versatile disc (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM), such as flash memory; and so on. Furthermore, the computer-readable medium may be configured as a data storage service accessible via a data connection such as an Internet connection. The computer-readable medium may, for example, be an online database or cloud storage.

[0040] This disclosure also relates to a computer program for instructing a computer to perform some or all of the steps or aspects of the computer-implemented methods described herein.

[0041] As described in this paper, a general motion prediction framework is provided. This system jointly predicts the future motion of all road users in a scenario, with complexity remaining constant regardless of the number of road users. Furthermore, a network architecture is provided to enable efficient learning of motion from a wide variety of types and numbers of road users who may have vastly different maneuverability. For example, pedestrians can change direction quickly but move more slowly than vehicles. Attached Figure Description

[0042] This document describes exemplary embodiments and functions of the present disclosure in conjunction with the following schematically illustrated figures:

[0043] Figure 1 A minimal static map consisting only of the drivable area and the centerline is shown;

[0044] Figure 2 A static map is shown with highlighted cells corresponding to possible future locations;

[0045] Figure 3 A static map is shown, which displays the offset of future locations relative to the center of the cells;

[0046] Figure 4 Static and dynamic contexts, according to various implementations, are shown that can be used to predict the future location of a self-vehicle;

[0047] Figure 5 An overall scene prediction system according to various implementation methods is shown;

[0048] Figure 6 The output structure for motion prediction of all objects from multiple classes is shown according to various implementations;

[0049] Figure 7 The diagram illustrates the classification output for jointly predicting the future positions of multiple objects by considering the multimodalities of multiple objects according to various implementations. The left figure shows the cell probability for 1 second, and the right figure shows the cell probability for 2 seconds.

[0050] Figure 8 An example is shown with the prediction and target, as well as the Gaussian kernel around the positive class label.

[0051] Figure 9 The predicted position on the 2D grid in the cell is shown, as well as the refinement of the predicted position using offset;

[0052] Figure 10 It is the output part of trajectory prediction;

[0053] Figure 11 This is a flowchart illustrating a method for predicting the characteristics (e.g., location) of multiple objects in a neighborhood of a vehicle, according to various embodiments;

[0054] Figure 12 This is a flowchart illustrating a method for predicting the characteristics (e.g., location) of multiple objects in a neighborhood of a vehicle, according to various embodiments;

[0055] Figure 13 Systems according to various embodiments are shown;

[0056] Figure 14 The network architecture used for fixed static mapping is shown;

[0057] Figure 15 The network architecture for movable static maps is shown;

[0058] Figure 16 The entire system structure is shown;

[0059] Figure 17 The unfolded diagram is shown; and

[0060] Figure 18 A computer system having multiple computer hardware components is shown, the multiple computer hardware components being configured to perform the steps of the computer-implemented methods as described herein. Detailed Implementation

[0061] According to various implementations, given a past trajectory and the surrounding environment, it is possible to provide a prediction of the future position of a self-vehicle or other object (e.g., within 1 second, 2 seconds, etc.).

[0062] As input data, static context (e.g., including road boundaries, traffic signs, traffic lights) and / or dynamic context (e.g., information about other traffic participants, including pedestrians, cars, trucks, and bicycles) and / or self-dynamics (in other words: the dynamics of the self-vehicle) may be available.

[0063] The problem with providing solutions according to various implementation methods is that there is not only one possible future location, but multiple different possible future locations, depending on factors such as multiple possible routes, different acceleration modes, and interactions with other traffic participants.

[0064] According to various implementations, the above-mentioned problem is solved by treating the motion prediction problem of various numbers and types of road users in complex driving scenarios as an image generation problem. The methods and systems according to various implementations can be applied to various driving scenarios, including, for example, highways and urban areas. The system complexity remains constant regardless of the number and type of road users in the application scenario.

[0065] Compared to methods and systems that cover only one road user, methods and systems implemented in various ways cover all road users. Therefore, even for multiple road users, the method does not need to be repeatedly applied, which reduces system complexity. An empty highway with only one vehicle will have the same system requirements as a congested urban intersection with a large number of pedestrians and various types of vehicles.

[0066] According to various implementations, the methods and systems jointly predict the movement of all road users in a scenario, while simultaneously possessing manageable complexity.

[0067] Figure 1 The illustration 100 shows a static map including roads 104 along with non-drivable areas (which may also be referred to as obstacles 102), and various possible locations 106, 108, 110 that can be assembled into a future trajectory 112.

[0068] A major challenge in achieving the above task using deep learning tools is the nature of the training data. Various deep learning schemes are trained on trajectory data to accomplish this task. However, trajectory data captured by sensor systems is highly imbalanced, as most trajectories so far follow a straight line with an approximately constant velocity. Without data preprocessing, a neural network might only learn to predict straight lines. However, anomalous behaviors (e.g., "unexpected stops," "acceleration," "turns," "deviations from the standard route") may be of interest. Depending on the implementation, this deviation from the standard route can be considered.

[0069] According to various implementation methods, many possible locations of multiple agents (multimodal) can be predicted, and each agent in the multiple agents can be assigned its own probability.

[0070] The ability to predict multiple possible future positions of a self-contained vehicle and other vehicles, given their surrounding environment, and to estimate their probabilities, can be used, for example, to provide safe autonomous driving.

[0071] According to various embodiments, apparatus and methods for predicting different multimodalities can be provided, including determining probabilities assigned to the prediction.

[0072] According to various implementation methods, static (input) maps (e.g., ...) Figure 1 The image (as shown) can be (finely) divided into a grid with multiple cells (e.g., 10×10 cells). The output (in other words: the output map) can initially be an empty grid map, which does not necessarily correspond to the exact same area as the input static map. It is possible to predict only the number of cells in the output map where the ego vehicle will be located in the future. This is a classification task, where the classes are different cells in the grid. This can be called "coarse" prediction. It should be understood that static information is not necessarily provided in the output grid map.

[0073] A classification task can be performed on each pixel in the output grid map (where each pixel corresponds to a cell). Pixels can indicate the space occupied by any road user.

[0074] Figure 2 It shows Figure 1 Illustration 200 of the static map, in which the cells corresponding to possible locations 202 are highlighted.

[0075] Various implementations address the problem of assigning probabilities to each future prediction. These implementations also address the problem of imbalanced datasets, as classification problems handle this issue well (e.g., as described in more detail in explaining the focus loss of Lin et al.).

[0076] According to various implementations, in order to refine the prediction and make it more accurate, an offset relative to the actual location can be provided (e.g., through the network used according to various implementations).

[0077] Figure 3 It shows Figure 1 and Figure 2 The illustration 300 of the static map shows the center 302 of the cell corresponding to the possible location 202. Offsets relative to the center of each cell can be provided.

[0078] Providing offsets to each cell makes regression problems very easy because only small offsets relative to the actual location need to be predicted.

[0079] The schemes, based on various implementation methods, were evaluated on real-world datasets, yielding results superior to those currently available.

[0080] Various output structures can be provided according to different implementations. Grid output representation allows for multimodal prediction. Predictions beyond single-object trajectory prediction can be provided, thus offering a holistic scene prediction system. By using output structures according to various implementations, it is possible to simultaneously predict the trajectories of a variable number of road users of different types over multiple time steps.

[0081] As described above, the surrounding environment of the self-contained vehicle can be divided into a 2D (two-dimensional) grid (e.g., an 80×80 grid with a resolution of 1 m), and the trajectory can be rasterized into an image-like structure. For each grid cell, the probability of a road user occupying that grid cell in the future can be predicted over multiple time steps (this could be a classification task). By predicting the offset for each grid cell (this could be a regression task), accuracy below 2D grid resolution can be achieved.

[0082] The input data of the methods according to various implementations can be subdivided into the following main categories: static context, dynamic context, and dynamics of the self-vehicle.

[0083] Figure 4 Illustration 400 shows static context 402 and dynamic context 404 according to various embodiments.

[0084] Static context can include static information or static objects (e.g., information or objects that do not change from one time frame to another). For example, static context can include lane and road boundaries, traffic signs and / or traffic lights. Static context data is available in today's vehicles; it can be provided by existing sensors or via offline maps combined with vehicle positioning, such as via GPS (Global Positioning System) or IMU (Inertial Measurement Unit).

[0085] Dynamic context can include dynamic information or dynamic objects (e.g., information or objects that may change or differ from one time frame to another). For example, dynamic context can include traffic participants such as pedestrians, cars, and / or bicycles. Dynamic context can include some or all of the measurements describing the dynamics of an object (e.g., coordinates, velocity, acceleration, and / or yaw rate). Dynamic context data can include not only current measurements but also past measurements. Accumulation of past data can allow for a fine-grained assessment of the dynamics of individual objects. Dynamic context data is readily available in today's vehicles; it can be provided by sensors from existing technologies.

[0086] Furthermore, the object's class (e.g., "pedestrian," "bicycle," "car," "truck") can be used as additional input. This allows for consideration of the different dynamics of traffic participants from different classes. This can further allow for fine-grained evaluation of the dynamics of individual objects. This category data is readily available in today's vehicles and can be provided by sensors using existing technologies.

[0087] According to various embodiments, there are apparatuses and methods that can jointly predict the future positions of multiple objects (in all objects in the input) for different future points in time, which may be possible due to the output structure according to the various embodiments described below.

[0088] Figure 5 A diagram 500 illustrating an overall scene prediction system according to various embodiments is shown. For example, the future positions of vehicle 502 and two pedestrians 508, 510 can be predicted. For example, the potential future position or future trajectory of vehicle 502 can be determined using the past trajectory 504 of vehicle 502. For example, individual positions 506 at the next time point can be predicted; for example, Figure 5 The location 506 shown can be the location with the highest probability of certainty. For example, a predetermined number of future trajectories can be determined (e.g., such as...). Figure 5 (As shown in the three examples for vehicle 502). Alternatively, if the probability is higher than a predetermined threshold, future locations or future trajectories can be considered, making the number of possible future locations flexible (depending on the probability of the future location).

[0089] According to various implementation methods, for trajectory prediction tasks, a grid-based output structure can be defined, which allows for the joint prediction of trajectories of a variable number of road users of different types (pedestrians, vehicles, bicycles, etc.) in a scene at multiple time steps.

[0090] The output structure can be defined as follows: I T×W×L×F :in:

[0091] T can describe the number of future time steps being predicted.

[0092] For example, T=2: the prediction is made for two future time steps (e.g., after 1 second and 2 seconds).

[0093] W×L The size of the 2D grid can be set to define the region of interest (ROI, which may be the area where trajectory predictions for all road users may be desired).

[0094] Each grid cell (by) W×L A definition (with width W and length L) can correspond to the position that an object may be located at in the future time step t.

[0095] For example, the ROI could be an 80 m × 80 m area around the self-propelled vehicle. To calculate W and L, the chosen resolution needs to be considered. For example, if the resolution is 0.1 m, then ( W×L The dimensions will be 800×800 。

[0096] F Output characteristics can be defined for each grid cell:

[0097] ,in:

[0098] C can be a set of classes, for example:

[0099]

[0100] o x , o y It can be described from the center of the grid cell at x direction and y Offset in direction to refine the actual position within the grid cell (or even outside the current cell).

[0101] cov It can be x and y The covariance.

[0102] Can describe ( w The probability that the grid cell at position l) will be occupied by an object of class c at future time step t. .

[0103] Figure 6A diagram 600 illustrates an output structure according to various embodiments for motion prediction of all objects from multiple classes (e.g., pedestrians and vehicles) after two time steps. For demonstration purposes, a small ROI of 10 m × 10 m with a resolution of 1 m was chosen. It should be understood that the size of the ROI can be variable. The output structure according to various embodiments can also allow for a variable number of classes and prediction time steps.

[0104] exist Figure 6 In the diagram, time is shown along axis 602, and various features are shown as images along direction 604. Multiple output images 606 can be provided at two time steps.

[0105] Due to the randomness of many traffic scenarios, there may be many possible future locations (which can be called modalities) that the system can consider. Each possible future location can typically have a different probability of occurrence. Methods according to various implementations can consider multimodalities by outputting a probability density defined on a grid, where each cell represents a specific location, such as... Figure 7 As shown. As mentioned above, individual cells may optionally or additionally include... x and y The covariance of . And therefore can be used to describe the probability density.

[0106] Figure 7 Illustration 700 shows a classification output that jointly predicts the future positions of multiple objects by considering multimodal factors of multiple objects according to various embodiments. In the left image 702, the probability for position 704 in the next second is shown. In the right image 706, the probability for position 708 in the next two seconds is shown. The future positions can be assembled into a trajectory. Figure 7 As shown, multiple possible future trajectories can be identified.

[0107] According to various implementations, the input includes trajectories of multiple road users, and the output is a grid map for occupancy. Individual cell pixels in the prediction (i.e., the grid map for occupancy) may not have a correspondence with the original tracks, thus losing the allocation of the original tracks. According to various implementations, post-processing can be provided to allocate the predicted grid cells to the original tracks to recover the predicted trajectories.

[0108] According to various implementations, a loss function can be provided for training based on an (artificial) neural network (NN). During training, the loss function can be minimized; for example, the parameters of the NN can be determined such that the loss function takes its minimum value. The ground reality for each prediction time range can be considered as an image, where it has a value (other than 0) only at image locations occupied by road users. The remaining pixels can be empty (in other words: have a value of 0).

[0109] According to various implementation methods, classification can be applied at the grid cell level, wherein each Grid cells (Hereinafter referred to as pixels) are assigned to time steps t The probability that the object is there. p .

[0110] As a (classification) loss function, pixel-wise distance-aware cross-entropy, originally used for object detection tasks, can be used, for example, from the work of Law et al. (CornerNet: Detecting Objects as Paired Keypoints, ECCV 2018).

[0111] Compared to Law et al., based on the focus loss from the work of Lin et al., according to various implementation methods, for the case of positive labels, the factor... α It can be added to the loss; for labels from the negative class, a factor (1-) can be added. α This is to address class imbalance (because most pixels may have labels from negative classes, meaning they will not be occupied in the future).

[0112] According to various implementation methods, the classification loss DCE can be defined as follows:

[0113]

[0114] in,

[0115] y can be a label (in other words: ground reality; i.e., information including actual data):

[0116] ;

[0117] y=1: Positive class label (e.g., pedestrian);

[0118] y<0: Negative class label (e.g., no pedestrians);

[0119] p It could be the probability of a specific class (e.g., pedestrian) at a pixel;

[0120] α It can be a factor that addresses class imbalances;

[0121] β , γ Hyperparameters can be used to measure factors.

[0122] α , β , γ It is a hyperparameter and can be tuned (e.g., it can be defined empirically, for example, through testing and tuning).

[0123] Around each positive label (y = 1), a Gaussian kernel can be defined as shown in Figure 8 . When the prediction of the positive class is close to the positive label, 0 < y < 1, the loss is reduced by the factor (1 - y). The output value of the kernel can be the value y.

[0124] Figure 8 Illustration 800 shows an example of a prediction 808 and a target 806 (in the left image 802) and a Gaussian kernel around the positive class label 806 (in the right image 804), which measures false positive predictions.

[0125] The size of the Gaussian kernel can be determined by its standard deviation σ . According to various embodiments, can be used, which is based on the current speed of the object and the predicted time step: α where can be the speed of the object, and can be an initial based on the time step

[0126]

[0127] where can possibly be the speed of the object , and can be based on the time step t initial σ .

[0128] The kernel can be a kernel different from the Gaussian kernel and may not necessarily be related to the normal distribution. Therefore, a covariance matrix can be used to describe the kernel. Thus, the axes can have different lengths and can be rotated. σ x can be different from σ y , which can also be rotated. The rotation can be determined by the heading / yaw of the object at a specific time. And σ x , σ y , σ xy can be calculated according to v x and v y .

[0129] The classification loss for the class can be summed over all pixels of the 2D grid and divided by the number of objects in the scene.

[0130] Distance-aware cross-entropy combined with a Gaussian kernel around the positive label (e.g., based on speed and how far in time the prediction is) can allow an artificial neural network to predict uncertainty and promote multimodality in predictions, such as at intersections where a vehicle may turn or go straight.

[0131] According to various embodiments, in addition to the classification loss, a regression loss can also be defined.

[0132] By predicting relative to the pixel center x and y Offsets can refine the position and achieve accuracy below 2D mesh resolution.

[0133] The regression loss can be defined as follows:

[0134]

[0135] R x R y It might be a regression loss for the offsets in the x and y directions.

[0136] It can be the actual ground offset in the x and y directions.

[0137] It can be the prediction offset in the x and y directions.

[0138] It can be an object indicator; for example, if an object exists at that pixel. =1 otherwise =0.

[0139] Figure 9 The illustration 900 shows the predicted position on the 2D grid in cell 904 (in the left image 902) and a refinement of the predicted position using offsets 908 in the x and y directions (in the right image 906).

[0140] According to various implementation methods, classification loss and regression loss can be combined into a combined weighted loss.

[0141] Figure 4 The inputs used for trajectory prediction are shown, and Figure 5 An illustration is shown of the trajectory prediction output from a prediction system for a scene at two time steps, which demonstrates static context awareness, particularly for vehicles highlighting approach curves. Figure 5 Multimodal trajectories are shown. Past trajectories can follow straight lines. For accurate predictions, not only past trajectories are considered, but also the static context of objects resembling road routes. The output is in Figure 10 As shown in Figure 1000.

[0142] Figure 11A flowchart 1100 illustrates a method for predicting the characteristics of multiple objects in the vicinity of a vehicle. At 1102, a grid map representation of road user perception data can be determined. Road user perception data may include or may be tracked perception results and / or untracked sensor intermediate detections. At 1104, a grid map representation of static environment data can be determined based on data obtained from the perception system and / or based on a predetermined map. At 1106, the characteristics of multiple objects can be determined based on the grid map representation of the road user perception data and the grid map representation of the static environment data.

[0143] According to various implementation methods, the characteristics of multiple objects can be determined as a function of at least one variable time point that determines the characteristics.

[0144] According to various implementation methods, the characteristics of multiple objects can be The form is determined, where Y(t) represents the characteristic (as a function of t), t represents at least one time point in time that determines the characteristic, and A, B, and C are constants, which are preferably used for sampling, interpolation, or extrapolation at variable time t, and are preferably determined as the variable output of the network.

[0145] According to various implementations, the value of t can be a discrete value taken from set T, and as mentioned above, the characteristics can be derived from... Provided.

[0146] According to various implementation methods, characteristics can be determined for past time points and / or current time points and / or future time points.

[0147] According to various embodiments, the characteristics of the multiple objects may include or may be a probability matrix, wherein each entry of the probability matrix is ​​related to a corresponding predetermined region of the vehicle's neighborhood and indicates the probability that an object appears in that corresponding region at a time different from a predetermined time; wherein the method may further include determining an offset relative to the center of the region for each entry of the probability matrix; wherein, preferably, a regression method is used to determine the offset.

[0148] According to various implementation methods, a grid map representation of road user perception data can be further determined based on data from the perception system; and / or a grid map representation of static environment data can be further determined based on a predetermined map.

[0149] According to various implementation methods, a dynamic context encoder can be used to encode a grid map representation of road user-perceived data; a static context encoder can be used to encode a grid map representation of static environment data; and a decoder can be used to decode the characteristics of the plurality of objects.

[0150] According to various embodiments, the classification method may include or may be a machine learning method, preferably an artificial neural network; wherein preferably, the machine learning method is trained based on training data, the training data including or representing traffic conditions of multiple objects moving across multiple time frames.

[0151] According to various implementations, the characteristics of multiple objects can be defined for multiple classes, wherein the multiple object classes may include or may be at least two items from the following classes: self-vehicle class, pedestrian class, cyclist class, vehicle class, truck class, van class, fast-moving object class, and slow-moving object class.

[0152] Figure 12 A flowchart 1200 illustrates a method for predicting the location of multiple objects in the vicinity of a vehicle, according to various embodiments. At 1202, at least one map representing the vicinity of the vehicle at a predetermined time can be determined. At 1204, a classification method can be applied to the at least one map, which provides a probability matrix as output, wherein each entry of the probability matrix relates to a corresponding predetermined area in the vicinity of the vehicle and indicates the probability of the presence of an object in the corresponding area at a time different from the predetermined time.

[0153] According to various implementations, at least one (input) map may include or may be a map indicating a static context of the neighboring area.

[0154] According to various implementations, at least one (input) map may include or may be a map indicating the dynamic context of the neighboring area.

[0155] According to various implementation methods, the classification method may include or may be a machine learning method, preferably an artificial neural network.

[0156] According to various implementations, the machine learning method can be trained based on training data, which includes traffic conditions of multiple objects moving across multiple time frames.

[0157] According to various implementation methods, machine learning methods can be trained based on loss functions, wherein, preferably, the loss function includes or is pixel-level distance-aware cross-entropy.

[0158] According to various implementation methods, the classification method can determine multiple probability matrices, each of which is associated with one of multiple object classes.

[0159] According to various implementations, the multiple object classes may include or may be at least two of the following: pedestrian, cyclist, vehicle, truck, van, fast-moving object, and slow-moving object.

[0160] According to various implementations, the probability matrix may be related to the center of a corresponding predetermined region, and the method may further include determining an offset relative to the center of the region for each entry of the probability matrix.

[0161] According to various implementation methods, the offset can be determined based on regression loss.

[0162] According to various implementation methods, multiple probability matrices related to multiple times can be determined.

[0163] According to various embodiments, the method may further include determining an occupancy matrix, wherein each entry of the occupancy matrix may be associated with a corresponding predetermined area of ​​the vehicle’s neighboring region and indicate whether an object is present in the corresponding predetermined area, and wherein a corresponding entry of the probability matrix may be determined only if an entry of the occupancy matrix indicates that an object is present in the corresponding predetermined area.

[0164] Each of steps 1102, 1104, 1106, 1202, and 1204, as well as the further steps described above, can be performed by computer hardware components.

[0165] According to various implementation methods, a method for predicting the future location of all registered (sensor-based) objects can be provided.

[0166] According to various implementations, not only coordinates are used, but also all measurement results (e.g., velocity, acceleration, etc.). Furthermore, past measurement results, such as velocity, acceleration, and the class of a single object, such as a car, bicycle, or pedestrian, can be used.

[0167] According to various implementations, the occupancy grid or cost map is based not only on registered coordinates but also on dynamic and static contexts, including the class of the object. The occupancy grid or cost map, according to various implementations, is created for future points in time (e.g., at 1 second, 2 seconds, etc.). Therefore, it describes not only the current presence and / or proximity of objects but also their future presence and / or proximity. Furthermore, the class of each predicted object is determined.

[0168] According to various implementations, the output layer is subdivided into cells, and cost values ​​are assigned to each cell. Furthermore, the precise location of objects can be refined by using additional offsets.

[0169] Figure 13A diagram 1300 illustrating a system according to various embodiments is shown. Inputs may include a dynamic context 1302, a static context 1304, and a self-dynamic context 1306. An output 1312 may be generated using a deep neural network 1308. When training the deep neural network 1308, the output 1312 and ground truth (GT) 1310 may be provided to a loss function 1314 for optimizing the deep neural network 1308.

[0170] Dynamic context 1302 may include a series of images that describe the movement of road users in past time frames. Possible sources of dynamic context images are rasterized trajectories of road users detected by sensors, or dynamic occupancy grids that can be detected by sensors such as cameras, radar, or lidar to represent areas, indicating occupancy and velocity of areas (grid cells).

[0171] Possible representations and sources of dynamic contextual images can include raw detections from sensors, such as cameras (projected onto the ground plane) or lidar sensors (using a scheme similar to rasterization to the image; multiple close or overlapping bounding boxes may belong to the same road user), or, for example, radar reflections from moving objects (using a scheme similar to rasterization to the image). By doing so, "tracking" in the perception system can be eliminated. The network can directly utilize raw sensor detections.

[0172] Static context 1304 can be represented by one or more of the following:

[0173] - Rasterized images from HD (high-resolution) maps, where high-resolution maps can cover the precise lane marking locations, so the vehicle's position can be accurately located within the lane;

[0174] - Driving area from sensors (e.g., a grid map or image data structure, where individual pixels in the image can represent the drivability of a specific area in the field of view). This term can refer to the drivable area detected from sensors (e.g., cameras, LiDAR, or radar);

[0175] - Lane / road detection from sensors (e.g., using lane markings, road boundaries, guardrails, etc. detected from sensors, and also constructing grid maps or image-like data (similar to rasterized maps) to describe static context);

[0176] - Static occupancy grid map (e.g., similar to "drivable area", occupancy grid map (detected from sensors) can be used as static context input).

[0177] For ego-dynamic 1306, the ego vehicle can be represented as a road user and is therefore included in the dynamic context input. Depending on the output, when focusing on the prediction of the ego vehicle, the ego vehicle data can also be represented as: a vector, or separate image inputs, or separate channels of the dynamic context input.

[0178] Output 1312 can provide the future locations of all road users. Output 1312 can be any time function.

[0179] According to various implementation methods, output 1312 can have the form .

[0180] It can represent the final output, and can be relative to t This is expressed so that predictions can be performed at any given time. It should be understood that a second-order function can be used as an example here.

[0181] The term "prediction" is commonly used in artificial neural networks (NNs), and it can mean that the NN takes inputs and outputs. Thus, prediction does not necessarily indicate a temporal relationship between the inputs and outputs, but for trajectory prediction according to various implementations, prediction can mean anticipating future events. It should be understood that the prediction used herein can have the meaning of providing an estimate.

[0182] It can be the forecast time range. t can be negative, 0, or positive.

[0183] Negative t can represent an estimate based on the past. For example, given more current data, we can classify past manipulations. This is similar to the concept of "Kalman smoothing".

[0184] Zero t can represent an estimate for the current time frame. If we use raw sensor detections to create the input, then predicting the trajectory at t=0 is actually used as a "tracking" method.

[0185] A positive t can represent a prediction about the future.

[0186] They can be different from each other and can be predicted as 3D matrices. A, B, and C can each represent an output grid map with a size of W×L, and the third dimension describes various features that can be predicted (e.g., occupancy probability, grid offset, covariance of predicted location, possible manipulation, or other features such as future velocity).

[0187] Using the output presented above, a continuous prediction over a time range t can be provided. It should be understood that the output described above is merely an example, and higher or lower order (or any other nonlinear function) functions can be used as the output.

[0188] According to various implementation methods, the value of t can be a discrete value taken from the set T, and as described herein, the characteristics can be derived from... Provided.

[0189] According to various implementation methods, the second-order output Y(t) = A t 2 + B t + C can be provided as a first-order Y(t) with a variable prediction range: Y(t) = B t + C. Here, the two outputs can be B and C. B can describe the part that depends on t (and is therefore moving), while C describes the part that is independent of t (and is therefore static). In other words: B describes the agent that is moving, and C describes the agent that is standing still (i.e., not moving).

[0190] Output 1312 can describe the future self-position. The output can also cover the prediction of the self-vehicle when the past trajectory of the self-vehicle is also given in the input dynamic context image (as described above).

[0191] Ground Reality 1310 can define tasks for artificial neural networks. It can cover tasks such as occupancy probability and in-grid offset location or other regression and classification tasks, such as the future location, speed, and manipulation of road users in a scene as an image.

[0192] Figure 14 A diagram 1400 illustrates a network architecture for a fixed static mapping. A dynamic context input 1402 can be provided to a recurrent neural network (RNN) 1404, whose output, along with the static context input 1406, can be provided to an extended neural network (DNN) 1408, whose output can be provided to a decoder 1410 to provide the total output.

[0193] Figure 15 Diagram 1500 illustrates the network architecture used for movable static maps. (Compared to...) Figure 14 Compared to the previous network architecture, both the dynamic context input 1402 and the static context input 1406 are provided to RNN 1404, and the output of RNN 1404 is provided to DNN 1408.

[0194] Exemplary urban traffic scenarios can include various types of vehicles with different motions and varying numbers of pedestrians with different behaviors and movements. All these road users can interact directly or indirectly and are simultaneously constrained by a static context: road / lane structure, etc. The potentially infinite number of different scenarios in urban traffic can be a challenging problem.

[0195] According to various implementations, images can be used as both system input and output. The output image can contain the future trajectories of all road users. Therefore, according to various implementations, the trajectory prediction problem can be transformed into an image generation problem. This input and output data representation can allow invariance to the number and type of road users, as well as invariance to a specific static context.

[0196] According to various implementation methods, HD map rasterization can be provided. Information from the HD map can be grouped into different categories (e.g., pedestrian areas, driving areas, lane dividers, and intersections) and each category can be assigned its own RGB (red-green-blue) color. Rasterization can be performed by category. One category can be rasterized over another, starting with a larger area (e.g., driving areas) and ending with a smaller area (e.g., lane dividers).

[0197] Trajectory rasterization can be provided according to various implementation methods. The trajectory of a road user can be defined as a series of points over consecutive time frames. Each point includes features such as: target position (x, y), velocity, acceleration, type, size, and orientation.

[0198] A fixed region of interest (ROI) surrounding the ego vehicle can first be rasterized into an empty multichannel image, where each pixel covers the fixed area. For example, an 80 m × 80 m ROI in front of the ego vehicle can be rasterized into an 80 × 80 pixel image, where each pixel represents an area of ​​1 × 1 square meters. It should be understood that the x and y resolutions can be different.

[0199] At each time frame, the location of all road users can be rasterized into pixel locations in the image, and their trajectory features at this time frame can be stored at their corresponding pixels.

[0200] The above process can be repeated for all consecutive time frames, thereby generating a series of images representing the trajectories of all road users.

[0201] In the above process, each road user can be considered as a point object, and therefore resides in only one pixel. Alternatively, the shape, size, and orientation of the road user can be considered, thus rasterizing it into multiple connected pixels.

[0202] Because of the input and output tables, motion prediction for the entire scene can be transformed into an image generation problem.

[0203] Figure 16 The diagram 1600 shows the overall system architecture. The various parts are related to... Figure 14 The structures shown are similar or identical, therefore the same reference numerals can be used, and repeated descriptions can be omitted. It should be understood that... Figure 16The specific input size and number of network layers shown are for illustrative purposes only. Figure 16 The operations described are cascaded. The actual number of CNN, ConvLSTM, and transposed Conv layers in the architecture is for illustrative purposes only. Furthermore, the custom ConvLSTM is a slightly modified version of the ConvLSTM.

[0204] In each past time frame, the positions of all road users can be represented in a multi-channel image. A series of such images can represent the motion of the entire scene in the time domain.

[0205] CNNs excel at learning the correlations between data points within their kernels. In the input, these correlations can be intuitively understood as potential interactions between road users and their subsequent impact on their behavior and trajectories. Therefore, CNNs can be used to construct trajectory encoders. Multiple layers of CNNs can be applied to image data, and this network can be termed a trajectory encoder.

[0206] CNNs can be used to integrate maps into prediction systems to extract features from the maps and learn them along with trajectory predictions. This network can be called a map encoder.

[0207] According to various implementation methods, skip connections to ConvLSTM can be provided.

[0208] Since trajectory data can be a series of images processed by a trajectory encoder, and its output is also a series of feature maps, convolutional recurrent neural networks (ConvRNNs) (such as ConvLSTM) can be applied to learn motion in the time domain.

[0209] Because different types of road users have different maneuverability, their range of movement within a fixed time frame can vary greatly. For example, a pedestrian may move 1 meter in 1 second, but a vehicle may have moved 10 meters.

[0210] Moreover, interactions between road users can occur at different scales. Pedestrians may have taken several meters to avoid collisions, but vehicles may need to travel much further. Therefore, the final CNN layer with a wide receptive field may not be the optimal layer to capture all these different movements and interactions.

[0211] Considering the different receptive fields of different CNN layers, "skip connections" can be provided by feeding the outputs of individual trajectory encoder layers to the trajectory decoder. RNNs (convLSTMs) can be integrated at these skip connections for time-series data processing. This expands the architecture from single-image processing to a time-series data processing framework. Static contextual features can also be integrated with the output features after the ConvLSTM via skip connections (e.g., through cascading). This allows for the use of different rasterization resolutions or different CNN layers for the map encoder. If the rasterized map and trajectory images have the same resolution and size, they can be cascaded and fed as input to the trajectory decoder.

[0212] According to various implementation methods, a trajectory decoder can be provided. To generate the output image, a transposed convolutional network can be applied. Through skip connections, time-series trajectory (image) data can be passed to the decoder at different levels and processed into the final output.

[0213] The outputs at each prediction time range can include:

[0214] - An image, which can be represented as It can represent the predicted location within a future time frame t. It can be represented as There are N different channels, each of which can present a prediction for a type of road user (e.g., pedestrian or vehicle). The pixel value of the image between [0, 1] can represent the probability (or likelihood) that the pixel is occupied.

[0215] - Dual-channel image Its pixel value can represent when the pixel is predicted as The intra-pixel x and y offsets at the future position in the image. This is likely because both the input and output are rasterized images, where individual pixels can have fixed and finite resolution. For example, a pixel can represent a 1 m × 1 m area in the real world. To achieve better accuracy, the intra-pixel offsets can also be predicted as a dual-channel image. These intra-pixel offsets can be independent of the specific type of road user. Depending on the implementation, they can be predicted for various... Provides specific offset Offsets can be applied to all types of road users.

[0216] Depending on the implementation, iterative outputs can be provided. Systems according to various implementations can use a custom ConvLSTM instead of predicting over a fixed time range. One difference between this ConvLSTM and the traditional one is that it only requires input during the first iteration. In each subsequent iteration, it... Future time range output prediction. It is the time interval between two consecutive prediction time ranges. Therefore, when running this last output layer iteratively, it is possible to... It obtains a series of predictions. This flexibility allows users to decide how far into the future they want to predict. This custom ConvLSTM can be built with only input at the beginning.

[0217] Figure 17 The diagram 1700 shows the unfolded graph. The convLSTM layer 1702 can be the final output layer of the decoder, and multiple convLSTM layers 1704, 1706, and 1708 can be provided in the unfolded graph to provide multiple outputs I_1, I_2, and I_3. Figure 17 The input C shown on the left can be the feature output of a preview NN (neural network) layer. Therefore, C can be fed into a convLSTM and iteratively decoded into multiple output grid maps. Layers 1702, 1704, 1706, and 1708 are actually the same convLSTM 1702, shown in this way to illustrate the "iterative call". Therefore, layers 1704, 1706, and 1708 are actually all 1702, but at different (i.e., subsequent) iterations.

[0218] Figure 18 A computer system 1800 with multiple computer hardware components is shown, said multiple computer hardware components being configured to perform the steps of the computer-implemented methods as described herein. The computer system 1800 may include a processor 1802, a memory 1804, and a non-transitory data storage unit 1806. Various sensors ( Figure 18 (Not shown in the image) may be provided as part of computer system 1800 or may be provided outside of computer system 1800.

[0219] Processor 1802 can execute instructions provided in memory 1804. Non-transitory data storage unit 1806 can store computer programs, including instructions that can be transferred to memory 1804 and then executed by processor 1802.

[0220] The processor 1802, memory 1804, and non-transitory data storage unit 1806 can be connected to each other, for example, via an electrical connection 1808 such as a cable or computer bus, or via any other suitable electrical connection, to exchange electrical signals.

[0221] The terms “connection” or “link” are intended to include direct “connection” (e.g., via a physical link) or direct “link” as well as indirect “connection” or indirect “link” (e.g., via a logical link).

[0222] It should be understood that the above description of one of the methods can be similarly applied to computer system 1800.

[0223] As described in this paper, a general motion prediction framework can be provided, for example, including machine learning systems that use dynamic and static contextual inputs, to estimate various features of the past, present, and future.

[0224] As described in this paper, trajectory prediction can be expressed as an image generation problem according to various implementation methods, thereby enabling the prediction of motion across the entire scene within a general applicable framework.

[0225] CNNs can be chosen for trajectory and map encoding, as they can effectively learn interactions between road users (dynamic context) as well as interactions with static context (static context).

[0226] This architecture can take into account the varying maneuverability of road users. RNNs can be integrated into the system (attached to CNN-based encoders and decoders) to capture the dynamics of road users in the time domain.

[0227] The iterative output layer provided in the decoder can offer flexibility for variable prediction range.

[0228] The methods and systems implemented in various ways offer good scalability (e.g., the same system can be used to handle scenarios ranging from simple to complex). System complexity can be constant for the number and type of road users, with relatively fixed system computational requirements, which may be important for functional safety.

[0229] According to various implementation methods, a general system architecture and a general input representation can be provided, which can take various types of input data (e.g., including tracked objects and / or raw detections) for various types of applications (including tracking, fusion, and prediction).

[0230] Various implementations can provide per-ML (machine learning) (dynamic and static) context awareness for trajectory prediction covering a wide range of complex driving scenarios.

[0231] Various implementation methods can provide effective predictions that, for example, do not change with the number of road users.

[0232] List of reference numerals

[0233] 100 Static Map Illustrations

[0234] 102 Obstacles

[0235] 104 Road

[0236] 106 Possible Locations

[0237] 108 Possible Locations

[0238] Possible location of 110

[0239] 112 Future Trajectory

[0240] 200 illustrations of static maps

[0241] 202 Possible Location

[0242] 300 Static Map Illustrations

[0243] The center of cell 302

[0244] 400 Examples of Static and Dynamic Contexts

[0245] 402 Static Context

[0246] 404 Dynamic Context

[0247] 500 Illustrations of overall scene prediction systems according to various implementation methods

[0248] Vehicle 502

[0249] 504 Past Trajectory

[0250] 506 Various positions

[0251] 508 pedestrians

[0252] 510 pedestrians

[0253] 600 Illustrations of the output structure according to various implementation methods

[0254] 602 Axis

[0255] 604 direction

[0256] 606 Output Image

[0257] 700 Illustrations output according to the classification of various implementation methods

[0258] 702 Left Image

[0259] 704 location

[0260] 706 Right Image

[0261] Location 708

[0262] Illustrations of examples of 800 prediction, target, and Gaussian kernel.

[0263] 802 Left Image

[0264] 804 Right Image

[0265] 806 Target

[0266] 808 Prediction

[0267] 900 Predicted location and detailed illustration of the predicted location.

[0268] 902 Left Image

[0269] Cell 904

[0270] 906 Right Image

[0271] 908 offset

[0272] The output diagram of 1000

[0273] 1100 shows a flowchart of a method for predicting the characteristics of multiple objects in the vicinity of a vehicle.

[0274] 1102 Steps for determining the grid map representation of road user perception data

[0275] 1104 Steps for determining the grid map representation of static environmental data

[0276] 1106 Steps to determine the properties of multiple objects

[0277] 1200 shows a flowchart illustrating a method for predicting the positions of multiple objects in the vicinity of a vehicle according to various embodiments.

[0278] 1202 The step of determining at least one map including a representation of the vicinity of the vehicle at a predetermined time.

[0279] 1204 The step of applying the classification method to the at least one map

[0280] 1300 Illustrations of systems according to various embodiments

[0281] 1302 Dynamic Context

[0282] 1304 Static Context

[0283] 1306 Self-Dynamic

[0284] 1308 Deep Neural Network

[0285] 1310 Ground Status

[0286] 1312 Output

[0287] 1314 Loss Function

[0288] 1400 Diagram of a network architecture for fixed static mapping

[0289] 1402 Dynamic Context Input

[0290] 1404 Recurrent Neural Network

[0291] 1406 Static Context Input

[0292] 1408 Extended Neural Network

[0293] 1410 Decoder

[0294] 1500 Diagram of network architecture for movable static maps

[0295] 1600 Overall System Architecture Diagram

[0296] 1700 Unfolded Diagram

[0297] 1702 ConvLSTM layers

[0298] 1704 ConvLSTM layers

[0299] 1706 ConvLSTM layers

[0300] 1708 ConvLSTM layers

[0301] 1800 Computer systems according to various implementation methods

[0302] 1802 processor

[0303] 1804 memory

[0304] 1806 Non-transitory Data Storage Department

[0305] 1808 connection

Claims

1. A computer-implemented method for predicting the characteristics of multiple objects in a neighborhood of a vehicle. in, The characteristics of the plurality of objects include at least one of position, velocity, heading angle, and collision risk value. The method includes the following steps performed by computer hardware components: Determine (1102) a grid map representation of road user perception data, wherein the road user perception data includes detection of tracking and / or detection of non-tracking based on radar data or camera data; A grid map representation of static environmental data (1104) determined based on sensor data provided by sensors and / or based on a predetermined map; as well as The characteristics of the plurality of objects are determined (1106) based on the grid map representation of the road user perception data and the grid map representation of the static environment data. Specifically, a dynamic context encoder is used to encode the grid map representation of the road user-perceived data; Specifically, a static context encoder is used to encode the grid map representation of the static environment data; Specifically, a decoder is used to decode the characteristics of the plurality of objects; Wherein, the output of the dynamic context encoder is the input of a recurrent neural network located between the dynamic context encoder and the static context encoder, and the input of the static context encoder is the output of the recurrent neural network; The output of the static context encoder is the input of the extended neural network located between the static context encoder and the decoder, and the input of the decoder is the output of the extended neural network.

2. The computer-implemented method according to claim 1, wherein, The characteristics of the plurality of objects are determined as a function of at least one variable time point that determines the characteristics.

3. The computer-implemented method according to claim 2, in, The characteristics of the plurality of objects are The form is used to determine the characteristic, where Y represents the characteristic, t represents at least one time point in time that determines the characteristic, and A, B, and C are constants.

4. The computer-implemented method according to claim 3, in, The constants A, B, and C are used for sampling, interpolation, or extrapolation at a variable time t.

5. The computer-implemented method according to claim 3, in, The constants A, B, and C are determined as the variable outputs of the machine learning method.

6. The computer-implemented method according to claim 2, in, It is also possible to determine the characteristics of the plurality of objects at discrete points in time.

7. The computer-implemented method according to claim 1, in, The characteristics are determined for past time points and / or current time points and / or future time points.

8. The computer-implemented method according to claim 1, in, The characteristics of the plurality of objects include a probability matrix, wherein each entry of the probability matrix is ​​related to a corresponding predetermined region in the vicinity of the vehicle, and indicates the probability that an object is present in the corresponding region at time t; The computer-implemented method further includes determining the offset relative to the center of the region for each entry of the probability matrix.

9. The computer-implemented method according to claim 8, in, The offset is determined using a regression method.

10. The computer-implemented method according to claim 1, in, The computer-implemented method is a classification method that includes machine learning methods.

11. The computer-implemented method according to claim 10, in, The machine learning method mentioned is an artificial neural network.

12. The computer-implemented method according to claim 11, in, The artificial neural network includes multiple skip connections with time filtering in a pyramid neural network architecture.

13. The computer-implemented method according to claim 10, in, The machine learning method is trained based on training data, which includes traffic conditions of multiple objects moving across multiple time frames.

14. The computer-implemented method according to claim 1, in, The characteristics of the plurality of objects are determined for a plurality of object classes, wherein the plurality of object classes include at least two of the following: self-vehicle class, pedestrian class, cyclist class, vehicle class, truck class, van class, fast-moving object class, and slow-moving object class.

15. The computer-implemented method according to claim 1, in, The computer-implemented method is used for fusion or for prediction.

16. A computer system comprising a plurality of computer hardware components configured to perform the steps of the computer-implemented method according to any one of claims 1 to 15.

17. A vehicle comprising: The computer system according to claim 16; as well as A sensor configured to provide sensor data, wherein a grid map representation of the road user perception data and / or a grid map representation of the static environment data is determined based on the sensor data.

18. A non-transitory computer-readable medium comprising instructions for performing the computer-implemented method of any one of claims 1 to 15.

Images