Method for determining spatio-temporal patterns related to vehicle environment

Abstract

A method for determining a pattern related to an environment of a host vehicle is provided. Characteristics are detected by a perceiving system of a host vehicle within an environment of the host vehicle. At least two processing levels having different scales are defined. For each processing level, respective current input data associated with a characteristic of a current point in time is combined with respective memory data associated with a characteristic of a previous point in time to generate joint spatio-temporal data for the respective processing level. An attention algorithm is applied to the joint spatio-temporal data at all processing levels to generate an aggregated data set, and at least one pattern related to the environment is determined from the aggregated data set.

Description

Technical Field

[0001] The present disclosure relates to a method for determining multi-scale spatiotemporal patterns associated with an environment of a host vehicle from sequentially recorded data. Background Art

[0002] Understanding the traffic scene in the external environment around the host vehicle is an important and challenging task for autonomous driving and various advanced driver assistance systems (ADAS). For example, based on sensor data acquired at the host vehicle, spatial and temporal correlations can be exploited to understand the traffic scene, such as for tracking and / or predicting the dynamics of any items present in the external environment.

[0003] To provide information about the host vehicle's environment, a machine learning algorithm including a recursive unit may be applied to data provided by sequential sensor scans performed by sensors of the host vehicle's perception system. Such a machine learning algorithm may be able to derive spatial and temporal patterns via the data provided by the sequential sensor scans (e.g., related to moving objects in the host vehicle's environment).

[0004] In detail, such a sensor scan is performed for a predefined number of time points, and sensor data may be aggregated over previous time points (i.e., before the current time point) in order to generate a so-called memory state. The memory state is combined with data from the current sensor scan (i.e., for the current time point) in order to derive spatial and temporal patterns (i.e., based on correlations within sequential sensor scans).

[0005] A machine learning algorithm comprising a recursive unit may also use multiple scales, i.e., multiple spatial resolutions, for the input data provided by the current sensor scan and for the memory state. Input data and memory states having different scales or resolutions are processed separately by the corresponding recursive unit for each scale or resolution level, and the features provided by the corresponding recursive unit are thereafter recombined at all scale levels. In this way, spatio-temporal correlations within the sensor scan may be exploited, for example, in order to increase the maximum resolvable velocity that can be derived for items or objects within the receptive field of the sensor.

[0006] However, known machine learning algorithms typically use static reorganizations of spatiotemporal patterns or correlations that can be extracted for the corresponding scales or resolutions of the input data and memory states. This can be a significant weakness of existing methods, as these methods can have low flexibility when adapting to changing conditions in the host vehicle environment may be required.

[0007] Therefore, there is a need for a method that can provide reliable spatiotemporal models of a vehicle's environment in response to changing environmental conditions. Summary of the invention

[0008] The present disclosure provides a computer implemented method, a computer system and a non-transitory computer readable medium according to the independent claims. Embodiments are given in the sub-claims, the detailed description and the drawings.

[0009] In one aspect, the present disclosure relates to a computer-implemented method for determining a pattern associated with an environment of a host vehicle based on sequentially recorded data. According to the method, a set of characteristics is determined within the environment of the host vehicle, wherein the characteristics are detected by a perception system of the host vehicle for a current time point and for a predefined number of previous time points. Via a processing unit of the host vehicle, at least two processing levels with different scales are defined for data associated with the respective levels. For each processing level, a corresponding current input data set associated with the set of characteristics at the current time point is combined with a corresponding memory data set associated with the set of characteristics at the previous time points to generate a joint spatiotemporal data set for the respective processing level. An attention algorithm is applied to the joint spatiotemporal data set of all processing levels to generate an aggregated data set, and at least one pattern associated with the environment of the host vehicle is determined based on the aggregated data set.

[0010] The perception system may include, for example, a radar system, a lidar system, and / or one or more cameras mounted at the host vehicle to monitor its external environment, which may be mapped, for example, into a bird's-eye view. Thus, the perception system may be able to monitor the dynamic background of the host vehicle including a plurality of objects that are able to move in the external environment of the host vehicle. The objects may include, for example, other vehicles 140 and / or pedestrians 150. The perception system may also be configured to monitor static objects, i.e., the static background of the host vehicle. For example, if the perception system of the vehicle includes a radar system and one or more cameras, such static objects may include traffic signs or lane markings.

[0011] The corresponding set of characteristics detected by the perception system can be obtained from sequentially recorded data, for example, by using a constant sampling rate. Therefore, the current time point and the previous time point can be defined in such a way that there is a constant time interval between them. However, a constant sampling rate is not a prerequisite for performing the method. The predefined number of previous time points may include the earliest time point and the latest time point that may be immediately before the current time point.

[0012] The output of the method, i.e., at least one pattern related to the environment of the host vehicle, can be provided as an abstract feature map that can be stored in a grid map. The grid map can be represented by a two-dimensional grid in a bird's eye view relative to the vehicle. However, other representations of the grid map can be implemented alternatively. The grid map can include a predefined number of cells. For each cell, a predefined number of features can be associated in order to generate a feature map.

[0013] Further tasks (e.g. comprising decoding procedures), which are implemented, for example, as further corresponding machine learning algorithms, can be applied to the feature map comprising at least one pattern related to the environment of the host vehicle. These tasks can include different kinds of information about the dynamics of the corresponding objects, for example about their position, their speed and / or about a bounding box around the corresponding objects. That is, the objects themselves (i.e. their position) and their dynamic properties can be detected and / or tracked by applying the corresponding tasks to the feature map comprising the patterns. Furthermore, grid segmentation can be performed as a task applied to the feature map, for example in order to detect free space in the environment of the host vehicle.

[0014] The method may be implemented as one or more machine learning algorithms, for example, as a neural network for which a suitable training procedure is defined. When training a machine learning algorithm or a neural network, the output of the method and the valid truth value may be provided to a loss function for optimizing the corresponding machine learning algorithm or neural network. The valid truth value is generated for a known environment of the host vehicle, for which the sensor data provided by the perception system may be pre-processed in order to generate data associated with, for example, a grid map in a bird's eye view.

[0015] Data can be processed by the method and corresponding results, for example, on further tasks for determining object detection and / or segmentation of free space can be related to the known environment of the host vehicle. A loss function can capture the error of the model (i.e., the model on which the machine learning algorithm or neural network relies) relative to the effective truth. The weights of the machine learning algorithm or neural network can be updated accordingly to minimize the loss function, i.e., the error of the model.

[0016] The method according to the present disclosure differs from the previous method in that an attention algorithm is applied to aggregate at least two processing levels, i.e., a set of spatiotemporal data generated for each processing level. For each processing level, the attention algorithm can determine a specific weight for aggregation, which can be based on the correlation determined by the combination of current input data and memory data at each level.

[0017] For example, if the method is implemented as a machine learning algorithm, valid queries for all processing levels can be matched with keys determined individually for each processing level to provide specific weights for each processing level. These weights can be combined with values ​​provided for each processing level, which can also be based on a combination of the current input data set and the corresponding memory data at each processing level.

[0018] Thus, the method is able to dynamically determine how to combine joint spatiotemporal data extracted at respective processing levels with a certain scale or resolution, for example to satisfy a target object function, since the weights and thus the aggregation performed by the attention algorithm are based on the characteristics obtained by the perception system. Furthermore, the weights can define the extent to which the spatiotemporal data of each processing level is considered in the resulting output or feature map according to the method of the present disclosure. Thus, the method is able to dynamically adapt to changing environmental conditions of the host vehicle.

[0019] According to an embodiment, the respective current input data set and the respective memory data set may be associated with respective grid maps having different respective spatial resolutions at each processing level. The association with the grid map may facilitate the representation of respective patterns associated with the environment of the host vehicle. For example, the grid map may include a plurality of grid cells, each grid cell including a different channel for each of the respective characteristics detected in the environment of the host vehicle.

[0020] The highest grid resolution may be provided for the first processing level, and a grid resolution lower than the highest grid resolution may be provided for subsequent processing levels. In other words, a hierarchy may be defined for scales or resolutions defined for different processing levels. That is, a refinement with respect to scale or resolution may be provided from the last processing level until the first processing level. This may improve the adaptability of the method to changing environmental conditions.

[0021] According to a further embodiment, the attention algorithm may include a query vector that is independent of the processing level. In addition, the attention algorithm may include a corresponding key vector and a value vector defined at each corresponding processing level after combining the corresponding set of current input data with the corresponding set of memory data. The key vector for each processing level may be combined with the query vector to provide weights for the elements of the value vector.

[0022] Therefore, the key vector and therefore the weight on each processing level can depend on the corresponding joint spatiotemporal data describing the space-time correlation (i.e., for different scales or resolutions) between the current input data and the memory data set on each processing level. The key vector of each processing level can be combined with a query vector independent of the processing level via a dot product so that the weight of each processing level describes the matching of the key vector and the query vector on each processing level. Therefore, the weight of each processing level, that is, the weight of the corresponding scale or resolution of the level can be determined in parallel with the weights of other processing levels. For example, during the training of a machine learning algorithm, specific weights for obtaining a value vector according to a corresponding set of corresponding input data can be learned or adjusted separately for each level. Using these specific weights, a value vector can be calculated according to the corresponding input data set of each level. Finally, the value vector of the corresponding level can ultimately be weighted by the corresponding matching of the key vector and the query vector.

[0023] If the resolution of the corresponding processing level is lower than the resolution of the query vector, the corresponding key vector and the corresponding value vector defined at the corresponding processing level can be upsampled to the resolution of the query vector. For example, the query vector may also be associated with a grid map having a predefined resolution or scale and associated with the current set of input data on which the query vector depends. In contrast, the corresponding key vectors defined at corresponding processing levels with different scales may be associated with different grid resolutions. Therefore, upsampling of the corresponding key vector to the resolution of the query vector may ensure the compatibility of the key vector with the query vector, so that these vectors may be combined, for example, by dot products, for determining the weights of the elements of the value vector when aggregating different processing levels within the attention algorithm. For example, upsampling of the key vector may be performed by applying interpolation to the elements of the key vector. Interpolation may be implemented as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation. Alternatively, deconvolution may be applied to perform upsampling. However, upsampling may not be mandatory in order to perform aggregation of processing levels via the attention algorithm.

[0024] The combination of the corresponding current input data set and the corresponding memory data set can be provided by applying a recursive algorithm at each processing level. For example, a convolutional gated recursive unit (ConvGRU) can be applied. Although recursive algorithms are generally known, they are used in a special way at each processing level in the context of the method according to the present disclosure in order to provide a basis for calculating the key vector and value vector of the attention algorithm.

[0025] At least one pattern associated with the environment of the host vehicle may be associated with at least one object detected in the environment of the host vehicle. However, any environmental pattern that is not necessarily associated with an object may be derived by the method according to the present disclosure. If at least one pattern is associated with at least one detected object, the corresponding pattern may be employed to track the object. For example, the velocity of an object may be derived with increased reliability due to the adaptation of the scale provided for the respective processing level to the changing environmental conditions.

[0026] In another aspect, the present disclosure relates to a computer system configured to perform some or all of the steps of the computer-implemented method described herein. The computer system may be further configured to receive a corresponding set of characteristics within the environment of the host vehicle, the characteristics being detected by a perception system of the host vehicle for a current point in time and for a predefined number of points in time prior to the current point in time.

[0027] The computer system may include a processing unit, at least one storage unit and at least one non-transient data storage. The non-transient data storage and / or memory unit may include a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein.

[0028] As used herein, terms like processing unit and module may refer to, be part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or group) that executes code, other suitable components that provide the functionality, or a combination of some or all of the above, such as in a system on a chip. A processing unit may include a memory (shared, dedicated, or group) that stores code executed by the processor.

[0029] In another aspect, the present disclosure relates to a vehicle comprising a sensing system and a computer system as described herein. For example, the vehicle may be an automotive vehicle.

[0030] According to an embodiment, the vehicle may further comprise a control system configured for receiving information derived from at least one mode provided by the computer system and applying the information to control the vehicle.

[0031] In another aspect, the present disclosure relates to a non-transitory computer-readable medium comprising instructions for performing some or all steps or aspects of the computer-implemented methods described herein. The computer-readable medium may be configured as: an optical medium, such as a compact disk (CD) or a digital versatile disk (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; or the like. In addition, the computer-readable medium may be configured as a data storage accessible via a data connection, such as an Internet connection. The computer-readable medium may be, for example, an online data repository or a cloud storage.

[0032] The present disclosure also relates to a computer program for instructing a computer to execute several or all steps or aspects of the computer-implemented method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Exemplary embodiments and functionality of the present disclosure are described herein in conjunction with the following drawings, which schematically illustrate:

[0034] Figure 1 a vehicle and a diagram of the vehicle's surroundings including a computer system according to the present disclosure,

[0035] Figure 2 According to the overview of the method of the present disclosure,

[0036] Figure 3 About Figure 2 Details of the steps of the method are shown,

[0037] Figure 4 like Figure 3 The details of the aggregation module are shown,

[0038] Figure 5 An upsampling procedure for data associated with different scales at different processing levels,

[0039] Figure 6 FIG. 1 is a flow chart illustrating a method for determining a pattern associated with an environment of a host vehicle from sequentially recorded data according to various embodiments,

[0040] Figure 7 An illustration of a system according to various embodiments, and

[0041] Figure 8 A computer system having a plurality of computer hardware components configured to perform the steps of a computer-implemented method as described herein. DETAILED DESCRIPTION

[0042] Figure 1A schematic diagram of a vehicle 100 and objects 130 that may surround the vehicle 100 in a traffic scene is depicted. The vehicle 100 includes a perception system 110 having an instrument field of view indicated by line 115. The vehicle 100 further includes a computer system 120, which includes a processing unit 121 and a data storage system 122, which includes, for example, a memory and a database. The processing unit 121 is configured to receive data from the perception system 110 and store the data in the data storage system 122. The vehicle 100 further includes a control system 124 configured to control the vehicle 100.

[0043] The perception system 110 may include a radar system, a lidar system, and / or one or more cameras to monitor the external environment or surroundings of the vehicle 100. Thus, the perception system 110 is configured to monitor a dynamic background 125 of the vehicle 100, which includes a plurality of objects 130 that can move in the external environment of the vehicle 100. The objects 130 may include, for example, other vehicles 140 and / or pedestrians 150.

[0044] The perception system 110 is also configured to monitor a static background 160 of the vehicle 100. The static background 160 may include static objects 130 such as traffic signs 170 and lane markings 180.

[0045] The perception system 110 is configured to determine characteristics of the objects 130. The characteristics include a current position, a current speed, and an object category of each road user 130 at a plurality of time points. The current position and the current speed are determined by the perception system 110 relative to the vehicle 100 (i.e., relative to a coordinate system having its origin, e.g., at the center of mass of the vehicle 100, its x-axis along the longitudinal direction of the vehicle 100, and its y-axis along the transverse direction of the vehicle 100). In addition, the perception system 100 determines the characteristics of the road user 130 at a predetermined number of previous time points and at the current time point (e.g., every 0.5 s).

[0046] The computer system 120 will generate the results or outputs 250 (see Figure 2 ) derived from the at least one mode (ie, from at least one mode related to the environment of the host vehicle 100) is transmitted to the control system 124 so that the control system 124 can use the information derived from the at least one mode to control the vehicle 100.

[0047] Figure 2 Depicted is a method for determining an environment associated with a host vehicle 100 based on sequentially recorded data (see, e.g., Figure 1) related modes. First, for multiple objects 130 in the environment of the host vehicle 100 (see Figure 1 ), a corresponding set of characteristics is detected by the perception system 110 of the host vehicle 100 for the current time point and for a predefined number of previous time points.

[0048] The characteristics of the current time point and the previous time point are transmitted to the processing unit 121, and the processing unit 121 generates a current input data set 210 and a memory data main set H associated with the characteristic set of the current time point by aggregating the characteristic sets at a predefined number of previous time points. t-1 .

[0049] The current input data set 210 and the main memory data set H t-1 is associated with a corresponding grid map defined for the receptive field within the environment of the host vehicle 100. That is, the corresponding dynamic background 125 and static background 160 (see Figure 1 ) is provided in the form of an image at the current time point and as an aggregation of previous time points. In other words, the features of the object 130 are rasterized or associated with corresponding elements of the grid map within a predefined region of interest around the vehicle 100. The predefined region of interest of the vehicle 100 is first rasterized into an empty multi-channel image, where each pixel covers a fixed area.

[0050] Therefore, the current input data set 210 is associated with a grid image comprising L×T pixels, where L represents the number of pixels in the first or longitudinal direction and T represents the number of pixels in the second or transverse direction. For example, the region of interest may cover a 280m×160m area in front of the vehicle 100 and may be rasterized into an image of 280×160 pixels, where each pixel represents a 1m×1m square area.

[0051] For each pixel or cell of the grid map or image, a corresponding channel is associated with one of the properties or features of the object 130. Thus, the empty multi-channel image mentioned above and representing the rasterized area of ​​interest near the vehicle 100 is populated with the properties of the object 130 that are associated with the corresponding channel of the pixel or grid cell.

[0052] In order to process the current input data 210, processing levels PL0, PL1, PL2 are defined, each processing level having a different scale or resolution for the data associated with the corresponding level. Figure 2 and Figure 3In the example shown, three processing levels PL0 , PL1 , PL2 are provided, to which modified versions of the original current input data 210 are transmitted, respectively.

[0053] The first processing level PL0 has a scale or resolution of L×T, i.e., the same resolution as the resolution of the grid map associated with the original current input data 210. That is, the first processing level PL0 receives a current input data set 212 associated with a set of features at the current point in time and associated with a grid map having a scale or spatial resolution of L×T like the original current input data 210.

[0054] The second processing level PL1 has a scale or spatial resolution of L / 2×T / 2, i.e. half the resolution of the first processing level PL0. At the second processing level PL1, a current input data set 214 is generated, which is associated with the grid map of this processing level PL1 with a scale or resolution of L / 2×T / 2. Similarly, the third processing level PL2 has a further reduced scale or resolution of L / 4×T / 4, i.e. L x T relative to the resolution of the grid map associated with the original current input data 210.

[0055] Each processing level PL0, PL1, PL2 comprises a respective recursive unit 232, 234, 236 which combines the respective current input data set 212, 214, 216 of each processing level PL0, PL1, PL2 with the respective memory data set 222, 224, 226, which are respectively related to the feature set of the previous time point. The respective memory data set 222, 224, 226 is also combined with the respective grid map having the same resolution as the resolution provided for the respective grid map associated with the respective current input data set 212, 214, 216 at each processing level PL0, PL1, PL2.

[0056] The corresponding recursive units 232, 234, 236 generate a joint spatiotemporal data set as a corresponding output on each processing level PL0, PL1, PL2 based on the combination of the corresponding current input data set 212, 214, 216 and the corresponding memory data set 222, 224, 226. The corresponding joint spatiotemporal data of each processing level PL0, PL1, PL2 is used to provide a corresponding input for the attention algorithm 240.

[0057] In detail, the corresponding key vectors K0, K1, K2 and the corresponding value vectors V0, V1, V2 are calculated at each processing level PL0, PL1, PL2 based on the outputs of the corresponding recursive units 232, 234, 236, and the corresponding key vectors K0, K1, K2 and the corresponding value vectors V0, V1, V2 are provided as inputs to the attention algorithm 240. In addition, a query vector Q is generated based on the original set 210 of the current input data, and the query vector Q is provided as an input to the attention algorithm 240. The query vector Q is independent of the processing levels PL0, PL1, PL2.

[0058] The attention algorithm 240 performs a matching of the query vector Q with the key vectors K0, K1, K2 of the corresponding processing levels PL0, PL1, PL2 in order to provide weights for the corresponding value vectors V0, V1, V2 when aggregating the data provided at the different processing levels PL0, PL1, PL2, as will be described in detail below. Based on this aggregation, the attention algorithm 240 provides an output 250 according to the method of the present disclosure, i.e., a data set from the aggregated data set at least related to the environment of the host vehicle 100 (see Figure 1 ) related patterns.

[0059] The output 250 , namely at least one pattern associated with the environment of the host vehicle 100 , is provided as an abstract feature map stored in a grid map generated in a manner similar to that described above for the grid map generated for association with the current input data set 210 .

[0060] The output 250 and the corresponding grid map of the current input data set 210 are represented by a two-dimensional grid in a bird's-eye view relative to the host vehicle 100. However, other representations of the grid map may be implemented alternatively. The grid map includes a predefined number of cells. For the output 250, a predefined number of features are assigned to each cell to generate a feature map.

[0061] The output 250 (i.e., the feature map described above) is transmitted to the task module 260, which applies further tasks to the feature map, which includes at least one pattern related to the environment of the host vehicle 100. These tasks include tasks related to object detection and / or to segmentation of the environment of the host vehicle 100, i.e., tasks related to segmentation of the grid map associated with the output 250.

[0062] Object detection provides different kinds of information about the dynamics of the respective object 130, such as about its position, its speed and / or about the bounding box around the respective object 130. That is, the objects 130 themselves (i.e., their positions) and their dynamic properties are detected and / or tracked by applying the respective tasks of the task module 260 to the feature map comprising the pattern. Furthermore, mesh segmentation is applied, for example, in order to detect free space in the environment of the host vehicle 100.

[0063] The results of the task module 260 as described above are provided to the control system 124 to use the results, such as the properties of the object 130 and / or free space, as information for controlling the host vehicle 100 .

[0064] The recursive units 232, 234, 236 and the attention algorithm 240 are implemented as corresponding machine learning algorithms, for example, as corresponding neural networks for which suitable training procedures are defined. The task module 260 including other tasks is also implemented as one or more machine learning algorithms, for example, including corresponding decoding procedures associated with corresponding tasks. The task module 260 includes a corresponding header for each required task.

[0065] When training a machine learning algorithm or a neural network, the output 250 and the valid truth value are provided to a loss function used to optimize the neural network. The valid truth value is generated for the known environment of the host vehicle 100, for which the sensor data provided by the perception system is preprocessed in order to generate data associated with, for example, a grid map in a bird's eye view. The data is processed by the method and the corresponding results of the different heads of the task module 260 (i.e., with respect to object detection and / or segmentation for, for example, determining free space) are related to the known environment of the host vehicle 100. The loss function obtains the error of the model (i.e., the model on which the machine learning algorithm or neural network relies) relative to the valid truth value. The specific weights of the machine learning algorithm or neural network are updated accordingly to minimize the loss function, i.e., the error of the model.

[0066] Figure 3 Describes Figure 2 The method is implemented as a machine learning algorithm, such as a neural network, which includes a method for Figure 2 and Figure 3 The corresponding collection of layers for the item or element shown.

[0067] Before generating a query vector 310 based on the original set 210 of current input data (i.e., when executing the training procedure of the entire neural network), a layer norm 312 is applied to these original input data 210 to obtain a uniform distribution of values ​​across training samples. Via the layer norm 312, the current input data or feature map 210 is scaled by the variance across entries calculated within the layer.

[0068] If the layer norm 312 is applied to a sparse feature map or the current input data set 210, for example, provided by a radar sensor, a large peak may be obtained due to a significant imbalance in the sparse feature map or the input data between entries containing relevant values ​​and the empty background, i.e., having values ​​in a range much larger than, for example, 1. For example, a few entries in the sparse feature map may have values ​​> 0, while the vast majority of entries may have values ​​equal to 0. For example, for a sparse feature map provided by a radar sensor, the feature map or the input data set 210 subjected to the layer norm 312 may be scaled to obtain a large peak. t To reduce the imbalance between entries in feature maps or input data.

[0069] In such Figure 3 In the example shown, the original current input data set 210 is associated with a grid map having a size of 280×160 cells to provide a current input data set 212 for the first processing level PL0. A max pooling layer 320 is applied to the original current input data set 210 to generate current input data sets 214, 216 for the second processing level PL1 and the third processing level PL2, which have respective lower scales or resolutions of 140×80 and 70×40, respectively, for the grid maps to which the respective current input data sets 214, 216 are associated.

[0070] The respective recursive unit 232 , 234 , 236 for each processing level PL0 , PL1 , PL2 is implemented as a respective convolutional gated recursive unit (ConvGRU) which combines the respective current input data set 212 , 214 , 216 with the respective memory data set 222 , 224 , 226 .

[0071] The spatiotemporal data or features extracted by the respective ConvGRU 232, 234, 236 are employed to generate respective key vectors 332, 334, 336 and respective value vectors 342, 344, 346 for each of the processing levels PL0, PL1, PL2. The respective key vectors 332, 334, 336 are matched with the query vector 310 within the aggregation module 360 ​​of the attention algorithm 214 to provide weights for data aggregation across all processing levels PL0, Pl1, PL2, as will be described below. Figure 4Prior to aggregating the second processing level PL1 and the third processing level PL2, the corresponding key vectors 334, 336 and value vectors 344, 346 of these processing levels are upsampled, as indicated at 350, to be compatible with the size or dimension of the query vector 310. Figure 5 Details of upsampling 350 are provided in the context of .

[0072] Figure 4 Describes the attention algorithm 240 (see Figure 3 ) of the aggregation module 360. After applying the layer norm 312 to the original current input data set 210, the query vector 310 is generated independently of the processing levels PL0, PL1, PL2. In detail, the input data or feature I subjected to the layer norm 312 t,N Linear projection is performed by the dense layer of the neural network to calculate the query vector 310 as follows:

[0073]

[0074] Where Q defines a set of trainable parameters. t Each of the L×T grid cells associated with the set is assigned a size d k , and thus the dimensions of the final query vector or query matrix 310 are defined as

[0075]

[0076] The query vector or matrix 310 is matched with the corresponding key vectors 332, 334, 336 generated for each processing level PL0, PL1, PL2 and, in detail, is calculated by a linear dense layer as described below:

[0077]

[0078] Among them, the key vector K i From the feature maps or joint spatiotemporal data H generated by the corresponding ConvGRU 332, 334, 336 t,i Get (see Figure 3 ). p represents the number of processing levels PL0, PL1, PL2, which corresponds to the number of different scales or resolutions of the grid associated with the current input data at the corresponding processing level. Figure 2 and Figure 3 In the example shown, N p Equal to 3. K defines a set of trainable parameters used to calculate the key vector K i In a similar manner to the query vector or matrix 310, the key K i The dimension is defined by

[0079]

[0080] Where L i and T i represents the number of grid cells in the longitudinal and transverse dimensions for the respective processing level PL0, PL1, PL2.

[0081] At 410, the corresponding key vector K i 332, 334, 336 are matched with the query vector or matrix 310 of each processing level PL0, PL1, PL2, respectively, wherein the dot product of the corresponding key vector 332, 334, 336 and the query vector or matrix 310 is calculated. Before calculating the dot product, the key vectors 334, 336 of the second processing level PL1 and the third processing level PL2 are upsampled, as mentioned above and in the following Figure 5 described in detail in the context of .

[0082] At 412, the dot product calculated at 410 is divided by In order to provide a value range that allows a larger gradient to pass through the subsequent softmax (flexible maximum transfer function) 420, so as to generate weights W for the value vectors 342, 344, 346 at each processing level PL0, PL1, PL2. Since the weights W are generated by the attention algorithm using the dot product of the key vector and the query vector, the weights can be expressed as attention weights and the dimensions of the attention weights are defined by the following formula:

[0083]

[0084] in

[0085]

[0086] Therein, X and Y denote respective sets of cells along the longitudinal and transverse dimensions of the grid cells provided for the respective processing levels PL0 , PL1 , PL2 .

[0087] At 430, the weight W Att is applied to the corresponding value vectors 342, 344, 346 in order to define the output 250 of each processing level PL0, PL1, PL2 to the method implemented by the machine learning algorithm (see Figure 2 and Figure 3) as a result of the algorithm's training procedure, the underlying model is able to dynamically determine which spatiotemporal data or features extracted at the respective processing levels PL0, PL1, PL2 are best suited to satisfy the target object function, for example, provided by the effective truth values ​​used for training and based on the characteristics obtained via the perception system 110 of the vehicle 100. Thus, the method is able to flexibly adapt to changing environmental conditions, which can be obtained by the characteristics detected in the environment of the host vehicle 100.

[0088] The value vectors 342, 344, 346 are obtained from the feature map H provided by the corresponding ConvGRU 332, 334, 336 in a similar manner as the key vectors 332, 334, 336. t,i In detail, the corresponding value vector is defined as follows:

[0089]

[0090] where f V Define a with trainable parameters θ V The dense layer with trainable parameters θ V The value vector includes the feature encoding of each unit of the associated grid map at each processing level PL0, PL1, PL2. Then at 440, the weight W associated with each processing level PL0, PL1, PL2 at 430 is calculated. Att The combined value vectors 342, 344, 346 are summed or aggregated to provide a result 250 according to the method of the present disclosure. The dimension of the value vector is defined by

[0091]

[0092] where d V represents the length of the value vector, which can be compared to the dimension d of the key vector described above. K different.

[0093] Figure 5 The upsampling 350 required for the second processing level PL1 and the third processing level PL2 before performing the aggregation of the processing levels PL0, PL1, PL2 is depicted (see Figure 3 ) details. Figure 5As seen on the left side of FIG. 3 , the respective grids associated with the memory data 222, 224, 226 and therefore the output features of the respective Conv Gro 332, 334, 336 have different specific resolutions at each processing level PL0, PL1, PL2. To be compatible with the dimensions of the query vector or matrix 310, nearest neighbor interpolation or bilinear interpolation is applied to the data associated with the respective grid maps on the second processing level PL1 and the third processing level PL2 to provide respective grid maps having the same resolution as the grid map associated with the data on the first processing level PL0. This is indicated by Figure 5 The middle grid is shown. After upsampling by 350, as previously described in Figure 4 The steps described in the context of are applied to each grid cell, in particular the application of dot product attention 410 and aggregation 440, to provide the grid cell with an output 250 of the method.

[0094] Figure 6 A flow chart 600 is shown illustrating a method for determining a pattern associated with an environment of a host vehicle from sequentially recorded data.

[0095] At 602, a set of characteristics can be determined within an environment of a host vehicle, wherein the characteristics can be detected by a perception system of the host vehicle. At 604, at least two processing levels with different scales can be defined for data associated with the respective levels via a processing unit of the host vehicle. At 606, for each processing level, a corresponding current input data set associated with a set of characteristics at a current point in time can be combined with a corresponding memory data set associated with a set of characteristics at a previous point in time to generate a joint spatiotemporal data set for the respective processing level. At 608, an attention algorithm can be applied to the joint spatiotemporal data set of all processing levels to generate an aggregated data set. At 610, at least one pattern related to the environment of the host vehicle can be determined based on the aggregated data set.

[0096] According to various embodiments, the respective current input data set and the respective memory data set may be associated with respective grid maps having different respective spatial resolutions at each processing level.

[0097] According to various embodiments, a first processing level may be provided with a highest grid resolution, and subsequent processing levels may be provided with grid resolutions lower than the highest grid resolution.

[0098] According to various embodiments, the attention algorithm may include a query vector that is independent of the processing level.

[0099] According to various embodiments, the attention algorithm may include a respective key vector and a value vector defined at each respective processing level after combining a respective set of current input data with a respective set of memory data.

[0100] According to various embodiments, the key vector for each processing level may be combined with the query vector to provide weights for the elements of the value vector.

[0101] According to various embodiments, if the resolution of the corresponding processing level is lower than the resolution of the query vector, the corresponding key vector and the corresponding value vector defined on the corresponding processing level may be upsampled to the resolution of the query vector.

[0102] According to various embodiments, upsampling of the key vector may be performed by applying interpolation to the elements of the key vector.

[0103] According to various embodiments, a combination of a corresponding current input data set and a corresponding memory data set may be provided by applying a recursive algorithm at each processing level.

[0104] According to various embodiments, at least one pattern related to the environment of the host vehicle may be associated with at least one object detected in the environment of the host vehicle.

[0105] According to various embodiments, at least one pattern associated with at least one object may be used to track the object.

[0106] Each of steps 602, 604, 606, 608, 610, and 612, and further steps described above, may be performed by computer hardware components.

[0107] Figure 7 A pattern determination system 700 is shown according to various embodiments. The pattern determination system 700 may include a characteristic determination circuit 702, a processing level definition circuit 704, a data combination circuit 706, an attention algorithm circuit 708, and a pattern determination circuit 710. The pattern determination system 700 may include a corresponding data combination circuit 706 for each processing level defined by the processing level definition circuit 704.

[0108] The characteristics determination circuit 702 may be configured to determine a set of characteristics detected by a perception system of the host vehicle within the host vehicle's environment.

[0109] The processing level definition circuit 704 may be configured to define at least two processing levels having different scales for data associated with the respective levels.

[0110] The data combination circuit 706 can be configured to combine, for each processing level, a corresponding current input data set associated with a feature set at a current time point with a corresponding memory data set associated with a feature set at a previous time point so as to generate a joint spatiotemporal data set for the corresponding processing level.

[0111] The attention algorithm circuit 708 may be configured to apply an attention algorithm to the joint spatiotemporal data set of all processing levels to generate an aggregated data set.

[0112] The mode determination circuit 710 may be configured to determine at least one mode associated with an environment of the host vehicle based on the aggregated data set.

[0113] The characteristic determination circuit 702, the processing level definition circuit 704, the data combination circuit 706, the attention algorithm circuit 708 and the pattern determination circuit 710 can be coupled to each other, for example, via an electrical connection 711 (such as a cable or a computer bus) or via any other suitable electrical connection to exchange electrical signals.

[0114] A “circuit” may be understood as any type of logic-implemented entity, which may be a dedicated circuit or a processor that executes a program stored in a memory, firmware, or any combination thereof.

[0115] Figure 8 A computer system 800 having multiple computer hardware components configured to perform the steps of a computer-implemented method for predicting respective trajectories of multiple road users according to various embodiments is shown. The computer system 800 may include a processor 802, a memory 804, and a non-transitory data store 806.

[0116] The processor 802 may execute the instructions provided in the memory 804. The non-transitory data storage 806 may store a computer program including instructions that may be transferred to the memory 804 and then executed by the processor 802.

[0117] The processor 802 , the memory 804 , and the non-transitory data store 806 may be coupled to each other, for example, via an electrical connection 808 , such as, for example, a cable or a computer bus, or via any other suitable electrical connection, to exchange electrical signals.

[0118] In this way, the processor 802, memory 804 and non-volatile data memory 806 can represent the characteristic determination circuit 702, processing level determination circuit 704, data combination circuit 706, attention algorithm circuit 708 and pattern determination circuit 710 as described above.

[0119] The terms “coupling” or “connection” are intended to include a direct “coupling” (eg, via a physical link) or a direct “connection” as well as an indirect “coupling” or an indirect “connection” (eg, via a logical link), respectively.

[0120] It should be understood that what is described with respect to one of the above methods may be similarly applicable to the pattern determination system 700 and / or the computer system 800 .

[0121] Reference numerals list

[0122] 100 Transportation

[0123] 110 Perception System

[0124] 115 Field of view

[0125] 120 Computer Systems

[0126] 121 processing units

[0127] 122 Storage, Database

[0128] 124 Control System

[0129] 125 Dynamic Background

[0130] 130 objects

[0131] 140 Transportation

[0132] 150 Pedestrians

[0133] 160 static background

[0134] 170 Traffic signs

[0135] 180 Lane markings

[0136] 210 The original set of current input data

[0137] 212,214,216 Input data at the first processing level, the second processing level, and the third processing level

[0138] 222, 224, 226 Memory data at the first processing level, the second processing level, and the third processing level

[0139] 232,234,236 Recursive units at the first processing level, the second processing level, and the third processing level

[0140] 240 Attention Algorithm

[0141] 250 Output

[0142] 260 Mission Module

[0143] 310 Query vector or matrix

[0144] 312 Layer Norm

[0145] 320 Max Pooling Layer

[0146] 332,334,336 Key vectors at the first processing level, the second processing level, and the third processing level

[0147] 342,344,346 Value vectors at the first processing level, the second processing level, and the third processing level

[0148] 350 Upsampling

[0149] 360 Aggregation Module

[0150] 410 Application of dot product

[0151] 412 Scale

[0152] 420 softmax function

[0153] 430 Weight of value vector

[0154] 440 Processing Level Aggregation

[0155] 600 is a flow chart illustrating a method for determining a pattern associated with an environment of a host vehicle from sequentially recorded data.

[0156] 602 Determine a set of characteristics detected by a perception system of the host vehicle within the environment of the host vehicle

[0157] 604 Defining at least two processing levels with different scales for data associated with the corresponding levels via the processing unit of the host vehicle

[0158] 606 For each processing level, combine the corresponding current input data set associated with the feature set at the current time point with the corresponding memory data set associated with the feature set at the previous time point to generate a joint spatiotemporal data set for the corresponding processing level

[0159] 608 applying an attention algorithm to the joint spatiotemporal data set at all processing levels to generate an aggregated data set

[0160] 610 Determine at least one mode related to the environment of the host vehicle based on the aggregated data set

[0161] 700 Mode Determination System

[0162] 702 Characteristic Determination Circuit

[0163] 704 Processing Level Definition Circuit

[0164] 706 Data combination circuit

[0165] 708 Attention Algorithm Circuit

[0166] 710 Mode determination circuit

[0167] 711 connections

[0168] 800 Computer system according to various embodiments

[0169] 802 processor

[0170] 804 Memory

[0171] 806 Non-transient data storage

[0172] 808 Connection

[0173] K0, K1, K2 key vectors

[0174] L The number of pixels or cells in the longitudinal direction

[0175] Q query vector

[0176] PL0, PL1, PL2 processing levels

[0177] T is the number of pixels or cells in the horizontal direction

[0178] V0, V1, V2 value vector

[0179] W weight.

Claims

1. A computer-implemented method for determining a pattern associated with an environment of a host vehicle (100) based on sequentially recorded data, The method comprises: determining a set of characteristics within the environment of the host vehicle (100) detected by a perception system (110) of the host vehicle (100), and Via the processing unit (121) of the host vehicle (100): defining at least two processing levels (PL0, PL1, PL2) having different scales for data associated with the respective levels (PL0, PL1, PL2), for each processing level (PL0, PL1, PL2), combining a corresponding current input data set (212, 214, 216) associated with the set of characteristics at a current point in time with a corresponding memory data set (222, 224, 226) associated with the set of characteristics at a previous point in time, so as to generate a joint spatiotemporal data set for the corresponding processing level (PL0, PL1, PL2), applying an attention algorithm (240) to the joint spatiotemporal data set of all processing levels (PL0, PL1, PL2) to generate an aggregated data set, and At least one pattern related to the environment of the host vehicle is determined from the aggregated data set.

2. The method according to claim 1, characterized in that The respective current input data sets (212, 214, 216) and the respective memory data sets (222, 224, 226) are associated with respective grid maps having different respective spatial resolutions at each processing level (PL0, PL1, PL2).

3. The method according to claim 2, characterized in that The first processing level (PL0) is provided with the highest grid resolution and subsequent processing levels (PL1, PL2) are provided with grid resolutions lower than said highest grid resolution.

4. The method according to claim 3, characterized in that The attention algorithm (240) includes a query vector (310) that is independent of the processing level (PL0, PL1, PL2).

5. The method according to claim 4, characterized in that The attention algorithm includes a corresponding key vector (332, 334, 336) and a value vector (342, 344, 346) defined at each corresponding processing level (PL0, PL1, PL2) after combining the corresponding current input data set (212, 214, 216) with the corresponding memory data set (222, 224, 226).

6. The method according to claim 5, characterized in that The key vector (332, 334, 336) for each processing level (PL0, PL1, PL2) is combined with the query vector (310) to provide weights for elements of the value vector (342, 344, 346).

7. The method according to claim 6, characterized in that If the resolution of the corresponding processing level (PL0, PL1, PL2) is lower than the resolution of the query vector (310), the corresponding key vector (332, 334, 336) and the corresponding value vector (342, 344, 346) defined at the corresponding processing level (PL0, PL1, PL2) are upsampled to the resolution of the query vector (310).

8. The method according to claim 7, characterized in that The upsampling of the key vector (332, 334, 336) is performed by applying interpolation to the elements of the key vector (332, 334, 336).

9. The method according to any one of claims 1 to 8, characterized in that A combination of the respective current input data set (212, 214, 216) and the respective memory data set (222, 224, 226) is provided by applying a recursive algorithm (232, 234, 236) at each processing level (PL0, PL1, PL2).

10. The method according to any one of claims 1 to 8, characterized in that At least one pattern related to the environment of the host vehicle (100) is associated with at least one object (130) detected in the environment of the host vehicle (100).

11. The method according to claim 10, characterized in that The at least one pattern associated with the at least one object (130) is used to track the object (130).

12. A computer system (900), the computer system (900) being configured to: receiving a corresponding set of characteristics within the environment of the host vehicle (100), the characteristics being detected by a perception system (110) of the host vehicle (100) for a current point in time and for a predefined number of points in time prior to the current point in time, and Execute the computer-implemented method of at least one of claims 1 to 11.

13. A vehicle (100) comprising a perception system (110) and the computer system of claim 12.

14. The vehicle (100) according to claim 13, Further included is a control system (124) configured to receive information derived from the at least one mode provided by the computer system and apply the information to control the vehicle (100).

15. A non-transitory computer-readable medium comprising instructions for executing the computer-implemented method of at least one of claims 1 to 11.

Images