Method for determining spatio-temporal patterns related to vehicle environment

Abstract

A method for determining a pattern related to an environment of a host vehicle from sequentially recorded data is provided. A respective set of characteristics detected by a perceptual system of the host vehicle in an environment of the host vehicle is determined for a current point in time and for a predefined number of previous points in time. A current set of input data associated with the set of characteristics is generated for a current point in time, and a set of memory data is generated by aggregating the set of characteristics at a previous point in time. An attention algorithm is applied to the current input data set and the memory data set to generate a joint spatio-temporal data set, and at least one pattern of the environment of the host vehicle is determined from the joint spatio-temporal data set.

Description

Technical Field

[0001] The present disclosure relates to a method for determining spatiotemporal patterns associated with an environment of a host vehicle from sequentially recorded data. Background Art

[0002] Understanding the traffic scene in the external environment around the host vehicle is an important and challenging task for autonomous driving and various advanced driver assistance systems (ADAS). For example, based on sensor data acquired at the host vehicle, spatial and temporal correlations can be exploited to understand the traffic scene, such as for tracking and / or predicting the dynamics of any items present in the external environment.

[0003] To provide information about the host vehicle's environment, a machine learning algorithm including a recursive unit may be applied to data provided by sequential sensor scans performed by sensors of the host vehicle's perception system. Such a machine learning algorithm may be able to derive spatial and temporal patterns via the data provided by the sequential sensor scans (e.g., related to moving objects in the host vehicle's environment).

[0004] Specifically, such a sensor scan is performed for a predefined number of time points, and sensor data may be aggregated over previous time points (i.e., before the current time point) in order to generate a so-called memory state. The memory state is combined with data from the current sensor scan (i.e., for the current time point) in order to derive spatial and temporal patterns (i.e., based on correlations within sequential sensor scans).

[0005] However, a machine learning algorithm including a recursive unit may only be able to derive the dynamic properties of an object implicitly, for example, based on the displacement of the corresponding pattern. For example, a recursive unit may associate a first feature cluster indicating that an object existed in a memory state at a previous point in time with a second feature cluster indicating that the object exists in novel or current data at a current point in time.

[0006] As a result, the model upon which the machine learning algorithm relies may be unable to distinguish between different objects within the receptive field associated with the host vehicle's environment. Furthermore, the model may also be unable to incorporate legacy information about the dynamics of objects at previous points in time in order to improve estimates of dynamic properties (e.g., the speed of an object) at the current point in time. As a result, such a model may also be unable to use legacy information to improve matching of objects by utilizing the dynamic properties to improve identification of the corresponding position of the object for the next point in time or frame.

[0007] Therefore, there is a need for a method that can explicitly correlate sequentially recorded data detected in the external environment of a vehicle in order to provide a reliable model for the environment. Summary of the invention

[0008] The present disclosure provides a computer implemented method, a computer system and a non-transitory computer readable medium according to the independent claims. Embodiments are given in the dependent claims, the detailed description and the drawings.

[0009] In one aspect, the present disclosure relates to a computer-implemented method for determining a pattern associated with an environment of a host vehicle based on sequentially recorded data. According to the method, a corresponding set of characteristics detected by a perception system of the host vehicle in the environment of the host vehicle is determined for a current time point and for a predefined number of previous time points. A current input data set associated with the set of characteristics is generated for the current time point, and a memory data set is generated by aggregating the feature sets of previous time points. An attention algorithm is applied to the current input data set and the memory data set to generate a joint spatiotemporal data set, and at least one pattern of the environment of the host vehicle is determined based on the joint spatiotemporal data set.

[0010] The perception system may include, for example, a radar system, a lidar system, and / or one or more cameras mounted at the host vehicle to monitor its external environment. Thus, the perception system may be able to monitor the dynamic background of the host vehicle including a plurality of objects that can move in the external environment of the host vehicle. The objects may include, for example, other vehicles 140 and / or pedestrians 150. The perception system may also be configured to monitor static objects, i.e., the static background of the host vehicle. For example, such static objects may include traffic signs or lane markings.

[0011] The corresponding set of characteristics detected by the perception system can be obtained from sequentially recorded data, for example, by using a constant sampling rate. Therefore, the current time point and the previous time point can be defined in such a way that there is a constant time interval between them. Therefore, the predefined number of previous time points can include the earliest time point and the latest time point that can immediately precede the current time point.

[0012] The output of the method, i.e., at least one pattern related to the environment of the host vehicle, can be provided as an abstract feature map that can be stored in a grid map. The grid map can be represented by a two-dimensional grid in a bird's eye view relative to the vehicle. However, other representations of the grid map can be implemented alternatively. The grid map can include a predefined number of cells. For each cell, a predefined number of features can be associated in order to generate a feature map.

[0013] Further tasks (e.g. comprising a decoding procedure), for example implemented as further corresponding machine learning algorithms, can be applied to the feature map comprising at least one pattern related to the environment of the host vehicle. These tasks can include different kinds of information about the dynamics of the corresponding objects, for example about their position, their speed and / or about a bounding box around the corresponding objects. That is, the objects themselves (i.e. their position) and their dynamic properties can be detected and / or tracked by applying the corresponding tasks to the feature map comprising the patterns. Furthermore, grid segmentation can be performed as a task applied to the feature map, for example in order to detect free space in the environment of the host vehicle.

[0014] The method may be implemented as one or more machine learning algorithms, for example, as a neural network for which a suitable training procedure is defined. When training a machine learning algorithm or a neural network, the output of the method and the valid truth value may be provided to a loss function for optimizing the corresponding machine learning algorithm or neural network. The valid truth value is generated for a known environment of the host vehicle, for which the sensor data provided by the perception system may be pre-processed in order to generate data associated with, for example, a grid map in a bird's eye view.

[0015] Data can be processed by the method and corresponding results, for example, on further tasks for determining object detection and / or segmentation of free space can be related to the known environment of the host vehicle. A loss function can capture the error of the model (i.e., the model on which the machine learning algorithm or neural network relies) relative to the effective truth. The weights of the machine learning algorithm or neural network can be updated accordingly to minimize the loss function, i.e., the error of the model.

[0016] The method according to the present disclosure differs from previous methods that apply, for example, recursive units in that an attention algorithm is applied in order to explicitly exploit the correlations between the memory data set aggregated to the latest previous time point and the current input data set. These correlations are represented by the joint spatiotemporal data set. By using an attention algorithm, the method is able to explicitly associate patterns, for example, for the dynamics of an object, for consecutive frames or consecutive time points.

[0017] Thus, the method is able to distinguish between different objects within the external environment of the host vehicle (i.e., within the receptive field of the method) and uniquely provide or track their dynamic properties for the considered time point. In addition, legacy information from the memory data (e.g., about the dynamics of different objects) is taken into account via an attention algorithm within the joint spatiotemporal data set in order to derive corresponding patterns therefrom. For example, explicit spatial displacements can be derived from the feature maps and the spatial displacements can be converted into an improved estimate of the corresponding velocity of the object and an improved estimate of the position of the object at the next frame or next time point in order to improve the matching of objects between frames or time steps.

[0018] According to various embodiments, the current input data set and the memory data set may be associated with a corresponding grid map defined for the environment of the host vehicle. The association with the grid map may facilitate the representation of corresponding patterns associated with multiple objects. For example, the grid map may include multiple grid cells, each grid cell including a different channel for each of the corresponding characteristics in the environment of the host vehicle.

[0019] When applying the attention algorithm, a match between a cell of a grid map associated with a current input data set and multiple cells of a grid map associated with a memory data set may be determined. Alternatively, a match may be determined between one of the cells of the grid map associated with the current input data set and multiple cells of a grid map associated with the memory data set and additionally associated with the current input data set. In other words, the current input data may also be considered for multiple cells of the grid map that match one of the cells of the grid map associated with the current input data set. Thus, the attention algorithm may provide a relationship between the cells of the grid map considered at the current point in time and the "dynamic history" of the state that led to the current input data (i.e., for the current point in time).

[0020] According to a further embodiment, the attention algorithm may include weights defined by associating elements of the current input data set with allocated elements of the memory data set. The weights of the attention algorithm are applied to a value generated by taking the union of the elements of the current input data set and the allocated elements of the memory data set to provide a joint spatiotemporal data set as an output of the attention algorithm.

[0021] For example, an element of the current input data set may refer to one or more grid cells associated with a certain object under consideration, and an assigned element of the memory data set associated with an element of the current input data set may be associated with a previous dynamic of the object represented by more than one grid cell. By defining or generating weights of the attention algorithm according to the relationship of the current input data and the memory data, and by applying these weights to values ​​generated based on the current input data and a portion of the memory data, when generating the joint spatiotemporal data set, legacy information about the dynamics is transferred from the memory data. Thus, the reliability of the determined pattern is improved via the attention algorithm.

[0022] Associating elements of the current input data set with allocated elements of the memory data set may include: generating a query vector by taking elements of the current input data set, generating a key vector by taking allocated elements of the memory data set and elements of the current input data set, and estimating a dot product of the query vector and the key vector, estimating a weight of the attention algorithm based on the dot product. That is, via the dot product, estimating a consistent match between the query vector and the key vector, which represents a correlation between them and defines a weight of the attention algorithm.

[0023] The key vector may be generated based on a concatenation of the allocated elements of the memory data set and the elements of the current input data set.

[0024] To generate a joint spatiotemporal data set, the output of the attention algorithm can be further combined with the memory data set. The gating program can be configured to define weights for combining corresponding channels within the memory data set with the output of the attention algorithm. Thus, the gating program can adjust how to combine the memory state and the output of the attention algorithm for each channel within the memory data set.

[0025] According to a further embodiment, information about the corresponding distances relative to a spatial reference point may be associated with the current input data set and with the memory data set in order to track the movement of the object between one of the previous time points and the current time point. The spatial reference point may be defined at the host vehicle. The information about the corresponding distances may be defined on a mesh grid of distances and / or an existing grid map may be extended by the information about the corresponding distances. By tracking the movement of the object, a speed estimation may be performed.

[0026] Information about the respective distances associated with the memory data set may be determined via a motion model including a velocity estimate of the object. Information about the respective distances associated with the memory data set may be determined via a motion model including a velocity estimate of the object.

[0027] In addition, position information may be associated with elements of the current input data set and allocated elements of the memory data set in order to estimate the velocity of objects in the environment of the host vehicle. Velocity estimation may be performed via displacement estimation between the current time point and one of the previous time points. Position information may be provided as a position encoding that may be concatenated to the input data that may then be used to compute queries, keys, and values ​​as described above.

[0028] At least one pattern may be provided to an algorithm for object detection and / or to an algorithm for segmenting the environment of the host vehicle, the algorithm for object detection determining at least one of a position, a velocity and coordinates of a bounding box associated with at least one of a plurality of objects located in the environment of the host vehicle. The segmentation of the environment may include, for example, determining free space based on a grid segmentation. Based on this information about the respective object, for example, tracking of the object may be performed.

[0029] In another aspect, the present disclosure relates to a computer system configured to perform some or all of the steps of the computer-implemented method described herein. The computer system may be further configured to receive a corresponding set of characteristics detected by a perception system of the host vehicle in the environment of the host vehicle for a current time point and for a predefined number of time points before the current time point.

[0030] The computer system may include a processing unit, at least one storage unit and at least one non-transient data storage. The non-transient data storage and / or memory unit may include a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein.

[0031] As used herein, terms like processing unit and module may refer to, be part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or group) that executes code, other suitable components that provide the functionality, or a combination of some or all of the above, such as in a system on a chip. A processing unit may include a memory (shared, dedicated, or group) that stores code executed by the processor.

[0032] In another aspect, the present disclosure relates to a vehicle comprising a sensing system and a computer system as described herein. For example, the vehicle may be an automotive vehicle.

[0033] According to an embodiment, the vehicle may further include a control system configured to receive information derived from at least one mode provided by the computer system and apply the information to control the vehicle. The information derived from the at least one mode may include properties of objects detected in the environment of the host vehicle and / or a segmentation of the environment including free space.

[0034] In another aspect, the present disclosure relates to a non-transitory computer-readable medium comprising instructions for performing some or all steps or aspects of the computer-implemented methods described herein. The computer-readable medium may be configured as: an optical medium, such as a compact disk (CD) or a digital versatile disk (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; or the like. In addition, the computer-readable medium may be configured as a data storage accessible via a data connection, such as an Internet connection. The computer-readable medium may be, for example, an online data repository or a cloud storage.

[0035] The present disclosure also relates to a computer program for instructing a computer to execute several or all steps or aspects of the computer-implemented method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Exemplary embodiments and functionality of the present disclosure are described herein in conjunction with the following drawings, which schematically illustrate:

[0037] Figure 1 a diagram of a vehicle and an environment surrounding the vehicle including a computer system according to the present disclosure,

[0038] Figure 2 According to the overview of the method of the present disclosure,

[0039] Figure 3 About Figure 2 Details of the steps of the method are shown,

[0040] Figure 4 About Figure 2 and Figure 3 Further details of the steps of the method are shown,

[0041] Figure 5 FIG. 1 is a flow chart illustrating a method for determining a pattern associated with an environment of a host vehicle from sequentially recorded data according to various embodiments,

[0042] Figure 6 An illustration of a system according to various embodiments, and

[0043] Figure 7 A computer system having a plurality of computer hardware components configured to perform the steps of a computer-implemented method as described herein. DETAILED DESCRIPTION

[0044] Figure 1A schematic diagram of a vehicle 100 and objects 130 that may surround the vehicle 100 in a traffic scene is depicted. The vehicle 100 includes a perception system 110 having an instrument field of view indicated by line 115. The vehicle 100 further includes a computer system 120, which includes a processing unit 121 and a data storage system 122, which includes, for example, a memory and a database. The processing unit 121 is configured to receive data from the perception system 110 and store the data in the data storage system 122. The vehicle 100 further includes a control system 124 configured to control the vehicle 100.

[0045] The perception system 110 may include a radar system, a lidar system, and / or one or more cameras to monitor the external environment or surroundings of the vehicle 100. Thus, the perception system 110 is configured to monitor a dynamic background 125 of the vehicle 100, which includes a plurality of objects 130 that can move in the external environment of the vehicle 100. The objects 130 may include, for example, other vehicles 140 and / or pedestrians 150.

[0046] The perception system 110 is also configured to monitor a static background 160 of the vehicle 100. The static background 160 may include static objects 130 such as traffic signs 170 and lane markings 180.

[0047] The perception system 110 is configured to determine characteristics of the objects 130. The characteristics include a current position, a current speed, and an object category of each road user 130 at a plurality of time points. The current position and the current speed are determined by the perception system 110 relative to the vehicle 100 (i.e., relative to a coordinate system having its origin, e.g., at the center of mass of the vehicle 100, its x-axis along the longitudinal direction of the vehicle 100, and its y-axis along the transverse direction of the vehicle 100). In addition, the perception system 100 determines the characteristics of the road user 130 at a predetermined number of previous time points and at the current time point (e.g., every 0.5 s).

[0048] The computer system 120 will generate the results or outputs 260 (see Figure 2 ) derived from the at least one mode (ie, from at least one mode related to the environment of the host vehicle 100) is communicated to the control system 124 so that the control system 124 can control the vehicle 100 using the information derived from the at least one mode.

[0049] Figure 2 Depicted is a method for determining an environment associated with a host vehicle 100 based on sequentially recorded data (see, e.g., Figure 1) related modes. First, for multiple objects 130 in the environment of the host vehicle 100 (see Figure 1 ), a corresponding set of features is detected by the perception system 110 of the host vehicle 100 for the current time point and for a predetermined number of previous time points.

[0050] The features of the current time point and the previous time points are passed to the processing unit 121, which generates a current input data set 210 and a memory data set 220 associated with the feature set of the current time point by aggregating the feature sets at a predefined number of previous time points. The attention algorithm 230 is applied to the current input data set 210 and the memory data set 220 via the processing unit 121 to generate a joint spatiotemporal data set.

[0051] The output 360 of the attention algorithm 230 is subjected to the first residual block 240 and the second residual block 250 to determine an output 260 including at least one pattern from the joint spatiotemporal data set. Figure 3 and Figure 4 In the context of Figure 2 Details of the various blocks 210 to 250 are shown.

[0052] The output 260, (i.e., at least one pattern related to the environment of the host vehicle 100) is provided as an abstract feature map that can be stored in a grid map. The grid map is generated in a similar manner as described below for the corresponding grid map generated for associating the current input data set 210 and the memory data set 220 with the grid map (see also Figure 3 ).

[0053] The respective grid maps of the output 260, the current input data set 210, and the memory data set 220 are represented by a two-dimensional grid in a bird's eye view relative to the host vehicle 100. However, other representations of the grid map may be implemented alternatively. The grid map includes a predefined number of cells. For the output 260, a predefined number of features are assigned to each cell to generate a feature map.

[0054] The output 260 (i.e., the feature map described above) is passed to the task module 270, which applies further tasks to the feature map, which includes at least one pattern related to the environment of the host vehicle 100. These tasks include tasks related to object detection and / or to segmentation of the environment of the host vehicle 100, i.e., tasks related to segmentation of the grid map associated with the output 260.

[0055] Object detection provides different kinds of information about the dynamics of the respective object 130, such as about its position, its speed and / or about the bounding box around the respective object 130. That is, the objects 130 themselves (i.e., their positions) and their dynamic properties are detected and / or tracked by applying the respective tasks of the task module 270 to the feature map comprising the pattern. Furthermore, mesh segmentation is applied, for example, in order to detect free space in the environment of the host vehicle 100.

[0056] The results of the task module 270 as described above are provided to the control system 124 to use the results, such as the properties of the object 130 and / or free space, as information for controlling the host vehicle 100 .

[0057] The attention algorithm 230, the first residual block 240 and the second residual block 250 are implemented as corresponding machine learning algorithms, for example, as corresponding neural networks for which suitable training procedures are defined. The task module 270 including further tasks is also implemented as one or more machine learning algorithms, for example, including corresponding decoding procedures associated with the corresponding tasks. The task module 270 includes a corresponding header for each required task.

[0058] When training a machine learning algorithm or a neural network, the output 260 and the valid truth value are provided to a loss function used to optimize the neural network. The valid truth value is generated for the known environment of the host vehicle 100, for which the sensor data provided by the perception system is pre-processed in order to generate data associated with a grid map, for example in a bird's eye view. The data is processed by the method and the corresponding results of the different heads of the task module 270 (i.e., with respect to object detection and / or segmentation for determining free space, for example) are related to the known environment of the host vehicle 100. The loss function obtains the error of the model (i.e., the model on which the machine learning algorithm or neural network relies) relative to the valid truth value. The weights of the machine learning algorithm or neural network are updated accordingly to minimize the loss function, i.e., the error of the model.

[0059] about Figure 3 , the current input data set 210 and the memory data set 220 are associated with corresponding grid maps of the same size and defined as the receptive field of the method within the environment of the host vehicle 100. That is, the corresponding dynamic background 125 and static background 160 (see Figure 1) is provided in the form of an image at the current time point and as an aggregation of previous time points. In other words, the features of the object 130 are rasterized or associated with corresponding elements of the grid map within a predefined region of interest around the vehicle 100. The predefined region of interest of the vehicle 100 is first rasterized into an empty multi-channel image, where each pixel covers a fixed area. For example, the region of interest may cover an area of ​​80m×80m in front of the vehicle 100 and may be rasterized into an image of 80×80 pixels, where each pixel represents a square area of ​​1m×1m.

[0060] For each pixel or cell of the grid map or image, a corresponding channel is associated with one of the properties or features of the object 130. Thus, the empty multi-channel image mentioned above and representing the rasterized area of ​​interest near the vehicle 100 is populated with the properties of the object 130 that are associated with the corresponding channel of the pixel or grid cell.

[0061] For example, due to the space-time correlation, the information provided by the characteristics of the object 130 for different time points is aggregated at the unit 310 under consideration. The unit 310 is located at the hypothetical vehicle 330 at the current time point. In order to derive a pattern about the object 130 (e.g., for the hypothetical vehicle 330), a match between the information provided at the unit 310 and the information provided by the plurality of units 320 is calculated, which is associated with the memory data 220 and is related to the aggregation of the information provided at the unit 310 considered within the current input data set 210.

[0062] Generally speaking, I t A subset X of the current input data set 210 represented by I Considered to be:

[0063]

[0064] It comprises a unit 310, such as Figure 3 As shown. In the subset X I and by H t-1 A subset X of the memory data set 220 represented by H The match is calculated between, that is, the memory data aggregated at previous time points up to time step t-1, where time step t-1 is the latest time step before the current time point, as follows:

[0065]

[0066] In such Figure 3 In the example shown, the subset X HA unit 320 related to information of the unit 310 in the current input data is included. The unit 320 includes, for example, a unit closest to the unit 310 in terms of Euclidean distance.

[0067] Within the attention algorithm 230, the union X of the corresponding unit of the current input data set 210 (e.g., for unit 310) and a subset of the current input data 210 and a subset of the memory data 220 is calculated. I UX H In detail, the corresponding unit in the current input data set 210, such as the unit 310 under consideration, is considered for determining the query Q in the attention algorithm 230, and for determining the key K and value V of the attention algorithm 230, such as Figure 3 As indicated by the dashed line 340 in FIG.

[0068] In contrast, a corresponding subset of the memory data set 220 (e.g., as represented by Figure 3 ) is only considered for determining the key K and value V of the attention algorithm 230, as given by Figure 3 Indicated by the dashed line 350 shown in FIG. 4 . Since in addition to the subset of grid cells (e.g., Figure 3 In addition to the memory data set 220 associated with the cell 320 shown in FIG. 1 , the current input data set 210 associated with a subset of the grid cells (e.g., the cell 310 considered) is also considered when calculating the key K and value V of the attention algorithm 230, so the key and value of the attention algorithm 230 are based on the union X. I UX H , that is, Figure 3 As shown, the union of information provided by unit 310 and unit 320.

[0069] In detail, query Q is obtained by replacing X I The linear projection is

[0070]

[0071] where Q represents the trainable parameters of the matrix used for linear projection.

[0072] Similarly, the key K is provided by the second linear projection:

[0073]

[0074] where K represents the trainable parameters of the matrix for linear projection, and [X I , X H ] Define two subsets or feature maps X I , X H Along the so-called token dimension (i.e., along the Figure 3 and Figure 4 Cascade of columns and rows of the grid diagram shown).

[0075] To calculate X I and union X I UX H The match between them is done using the dot product of the query Q and the key K, which is subjected to the following softmax (flexible maximum transfer) function:

[0076]

[0077] Among them, the term d k represents the length of the key vector K and query vector Q for each token, and W A tt represents the so-called attention weight with the following properties:

[0078]

[0079] ΣW Att =1 (7)

[0080] Note that N F Indicates that from H T-1 The subset X obtained by the receptive field within H The number of units or tokens included, e.g. Figure 3 The information provided by unit 320 is shown. Therefore, the union or feature map X I UX H Including 1+N F feature vectors, and therefore, the attention weight W A ttDefine the union X I UX H X inside I and 1+N F Individual matches between entries.

[0081] Furthermore, these values ​​are also defined by linear projection as follows:

[0082]

[0083] where V represents the trainable parameters of the corresponding matrix for linear projection, and [X I , X H ] again represents two subsets or feature maps X along the token dimension I and X H cascade.

[0084] Finally, the output 360 of the attention algorithm 230 is provided as a weighted sum of the attention weights and values ​​as follows:

[0085]

[0086] In summary, the attention algorithm 230 aggregates the information from the union provided by the current input data set 210 and the memory data set 220 to the target location for the current time point t, for example, Figure 3 The considered cell 310 is shown. Due to the application of dot products for estimating attention weights, the algorithm is able to explicitly control the aggregation of information from the union of the current input dataset 210 and the memory dataset 220 at the target position of the cell 310 for the current time point t.

[0087] The first residual block 240 again combines the output 360 of the attention algorithm 230 with the memory state H represented by the memory data set 220 t-1 Therefore, when considered at a larger scale, the first residual block 240 and the second residual block 250 provide a recursive function similar to the known recursive unit.

[0088] In detail, the first residual block 240 includes a channel gating module 370, which defines the output 360 of the attention algorithm 230 and the memory state H t-1 220. The weights defined by the channel gating module 370 are given as follows:

[0089]

[0090] where σ represents the sigmoid activation function, and θ W_Gate Represents the trainable parameters of the channel gating module 370. Att is the output 360 of the attention algorithm 230.

[0091] Without the channel gating module 370, this method would only perform weighting on the value vector V as a whole. However, known cycle units allow for individual gating such as H t-1 Such a memory status channel.

[0092] To achieve similar degrees of freedom with respect to feature propagation, channel gating module 370 uses Figure 3 The weighted sum is depicted in detail in the enlarged sub-diagram of the channel gating module 370. The calculation of the weights as defined in equation (10) above is schematically depicted at 371. The weights are applied to the memory state H at 372 and 373, respectively. t-1 The weighted channels are combined at 374 to generate a weighted sum as the output of the channel gating module 370. Due to the channel gating module 370 within the first residual block 240, the method is able to control how the memory states H are combined on a channel-by-channel basis. t-1 and output 360 of the attention algorithm 230 .

[0093] The output of the channel gating module 370 is subjected to a layer norm 375. By the layer norm 375, the gated features provided as the output of the channel gating module 370 are given within a uniform value range of the feature map that reduces the covariation shift.

[0094] The second residual block 250 includes a feed-forward block 380 and another layer norm 385. The feed-forward block 380 includes a two-dimensional convolutional network with a kernel size of 1×1, such that:

[0095]

[0096] Ou tRes1 represents the output of the first residual block 240, and FF1 and FF2 represent the trainable parameters of the first convolutional network and the second convolutional network in the feedforward module 380, respectively. ReLU represents the rectified linear unit activation function. Out Res2 represents the output of the feedforward module 380, Out Res2 It is combined at 381 with the output of the first residual module 240. The combination of the outputs of the first residual module 240 and the feed forward module 380 generated at 381 is subjected to another layer norm 385 to provide the output 260 of the overall method according to the present disclosure.

[0097] The method described above generates a joint spatiotemporal data set including a space-time correlation between the current input data set 210 and the memory data set 220, which is exploited to determine at least one pattern associated with a plurality of objects 130 detected in the environment of the host vehicle 100. In order to achieve an "increased representation" of the dynamics of the traffic scene around the host vehicle 100, i.e., to enhance object tracking, additional information may be incorporated into the image, e.g. Figure 4 in the method described.

[0098] To this end, the corresponding grids 410, 420 associated with the current input data set 210 and the memory data set 220, respectively, further contain the distance of each grid cell relative to the spatial reference position located at the rear bumper of the host vehicle 100 (see Figure 1 ). In detail, each query Q defined above is represented by the metric coordinates pos of each grid cell relative to the spatial reference position abs To expand or extend. t The same is true for the key K of the current input data set 210 represented by . abs The expansion of the query Q and key K for the current input data set 210 is Figure 4 Indicated by 430.

[0099] At the same time, for each grid cell, for the detected object 130 (see Figure 1 ) predicts the velocity v in the longitudinal and transverse directions relative to the grid at the previous time point t-1 t-1 . Speed ​​v t-1 It can be estimated by any velocity estimation head for the previous time point t-1 and provided to the method for velocity estimation at the current time point t. t-1 , projecting the metric coordinates pos of the previous time point t-1 through the following motion model abs :

[0100]

[0101] where f R is a constant frequency, for example 20 Hz, which indicates the frequency of the signal transmitted by the sensing system 110 (see Figure 1 )The frame rate at which sensor scans are performed.

[0102] The union X described above is expanded or extended in a similar manner to the query Q and key K for the current input data set 210 by the projected target position or coordinate pos′. I UX H The key K. About H t-1 The expansion of the key K of the memory data set represented by Figure 4 Indicated by 440.

[0103] In this way, the model bottom layer according to the method of the present disclosure is able to incorporate the previous assumption about the position of the object 130 moving at the current time point t-1 contained by the specific grid cell at the current time point t. Before concatenating the corresponding tokens used to calculate the query Q and the key vector K, pos abs and pos' are scaled by a constant factor so that they include values ​​in the range [1,1] in order to maintain stable gradients during training.

[0104] To provide a prediction about the velocity of the object 130, such as Figure 4 The value V is also “position-aware” as indicated by 450 in FIG. 4 . In detail, the value V is also encoded by the position (ie, the relative distance pos defined for each grid cell relative to the center position of the moving imaginary vehicle 330) rel ) to expand or extend. pos rel Also included are longitudinal and lateral coordinates defined relative to the grids 410 , 420 .

[0105] By concatenating the input token to the value calculation and pos rel , achieving the above-mentioned perception of direction and distance. Therefore, the displacement of the object 130 can be measured, for example, from any position to Figure 4 The center position of the “sliding window” in FIG. 3 is shown by the dotted line indicating the virtual vehicle 330. That is, the corresponding speed can be predicted after determining the displacement of the object 130 in the time interval between the previous time point t-1 and the current time point t.

[0106] Figure 5 A flow chart 500 is shown illustrating a method for determining a pattern associated with an environment of a host vehicle from sequentially recorded data.

[0107] At 502, a corresponding set of characteristics detected by a perception system of the host vehicle in the environment of the host vehicle can be determined for a current time point and for a predefined number of previous time points. At 504, a current input data set associated with the set of characteristics can be generated for the current time point. At 506, a memory data set can be generated by aggregating the feature sets of previous time points. At 508, an attention algorithm can be applied to the current input data set and the memory data set to generate a joint spatiotemporal data set. At 510, at least one mode of the environment of the host vehicle can be determined based on the joint spatiotemporal data set.

[0108] According to various embodiments, the current input data set and the memory data set may be associated with respective grid maps defined for the environment of the vehicle.

[0109] According to various embodiments, when applying an attention algorithm, a match between a cell of a grid map associated with a current input dataset and a plurality of cells of a grid map associated with a memory dataset may be determined.

[0110] According to various embodiments, the attention algorithm may include weights that may be defined by associating elements of a current input data set with allocated elements of a memory data set, and the weights of the attention algorithm may be applied to a value generated by taking the union of the elements of the current input data set and the allocated elements of the memory data set so as to provide a joint spatiotemporal data set as an output of the attention algorithm.

[0111] According to various embodiments, associating elements of a current input data set with assigned elements of a memory data set may include generating a query vector by employing elements of the current input data set, generating a key vector by employing assigned elements of the memory data set and elements of the current input data set, and estimating a dot product of the query vector and the key vector, estimating a weight of an attention algorithm based on the dot product.

[0112] According to various embodiments, a key vector may be generated based on a concatenation of allocated elements of a memory data set and elements of a current input data set.

[0113] According to various embodiments, to generate a joint spatiotemporal data set, the output of the attention algorithm may be further combined with the memory data set, and the gating procedure may be configured to define weights for combining corresponding channels within the memory data set with the output of the attention algorithm.

[0114] According to various embodiments, information about respective distances relative to a spatial reference point may be associated with a current input data set and with a memory data set in order to track movement of an object between one of the previous time points and the current time point.

[0115] According to various embodiments, information about the respective distances associated with the memory data set may be determined via a motion model, which may include a velocity estimate of the object.

[0116] According to various embodiments, position information may be associated with elements of a current input dataset and allocated elements of a memory dataset in order to estimate the velocity of objects in the host vehicle's environment.

[0117] According to various embodiments, at least one pattern can be provided to an algorithm for object detection and / or to an algorithm for segmenting the environment of a host vehicle, wherein the algorithm for object detection determines at least one of the position, velocity, and coordinates of a bounding box of an object associated with at least one of a plurality of objects located in the environment of the host vehicle.

[0118] Each of steps 502, 504, 506, 508, and 510, and further steps described above, may be performed by computer hardware components.

[0119] Figure 6 A pattern determination system 600 is shown according to various embodiments. The pattern determination system 600 may include a characteristic determination circuit 602, a current input data generation circuit 604, a memory data generation circuit 606, an attention algorithm circuit 608, and a pattern determination circuit 610.

[0120] The characteristic determination circuit 602 may be configured to determine, for a current point in time and for a predefined number of previous points in time, a corresponding set of characteristics detected by a perception system of the host vehicle in the environment of the host vehicle.

[0121] The current input data generating circuit 604 may be configured to generate a current input data set associated with a characteristic set at a current time point.

[0122] The memory data generating circuit 606 may be configured to generate a memory data set by aggregating characteristic sets of previous time points.

[0123] The attention algorithm circuit 608 may be configured to apply an attention algorithm to the current input data set and the memory data set to generate a joint spatiotemporal data set.

[0124] The mode determination circuit 610 may be configured to determine at least one mode of the host vehicle's environment based on the joint spatiotemporal data set.

[0125] The characteristic determination circuit 602, the current input data generation circuit 604, the memory data generation circuit 606, the attention algorithm 608 and the pattern determination circuit 610 can be coupled to each other, for example, via an electrical connection 611 (such as a cable or a computer bus) or via any other suitable electrical connection to exchange electrical signals.

[0126] A “circuit” may be understood as any type of logic-implemented entity, which may be a dedicated circuit or a processor that executes a program stored in a memory, firmware, or any combination thereof.

[0127] Figure 7 A computer system 700 having a plurality of computer hardware components configured to carry out the steps of a computer-implemented method for predicting respective trajectories of a plurality of road users according to various embodiments is shown. The computer system 700 may include a processor 702 , a memory 704 , and a non-transitory data store 706 .

[0128] The processor 702 may execute the instructions provided in the memory 704. The non-transitory data storage 706 may store a computer program including instructions that may be transferred to the memory 704 and then executed by the processor 702.

[0129] The processor 702 , the memory 704 , and the non-transitory data store 706 may be coupled to each other, for example, via an electrical connection 708 , such as, for example, a cable or a computer bus, or via any other suitable electrical connection, to exchange electrical signals.

[0130] In this way, the processor 704, memory 706 and non-transitory data storage 706 can represent the characteristic determination circuit 602, current input data generation circuit 604, memory data generation circuit 606, attention algorithm circuit 608 and pattern determination circuit 610 as described above.

[0131] The terms “coupling” or “connection” are intended to include a direct “coupling” (eg, via a physical link) or a direct “connection” as well as an indirect “coupling” or an indirect “connection” (eg, via a logical link), respectively.

[0132] It should be understood that what is described with respect to one of the above methods may similarly be applicable to the pattern determination system 600 and / or the computer system 700 .

[0133] Reference numerals list

[0134] 100 Transportation

[0135] 110 Perception System

[0136] 115 Field of view

[0137] 120 Computer Systems

[0138] 121 processing units

[0139] 122 Storage, Database

[0140] 124 Control System

[0141] 125 Dynamic Background

[0142] 130 objects

[0143] 140 Transportation

[0144] 150 Pedestrians

[0145] 160 static background

[0146] 170 Traffic signs

[0147] 180 Lane markings

[0148] 210 Current input data set

[0149] 220 Memory Data Collection

[0150] 230 Attention Algorithm

[0151] 240 First residual block

[0152] 250 Second residual block

[0153] 260 Method Output

[0154] 270 Mission Module

[0155] 310 The grid cells associated with the current input data set

[0156] 320 A plurality of cells associated with a memory data set

[0157] 330 Hypothetical means of transport

[0158] 340 dotted line

[0159] 350 dotted line

[0160] Output of the 360 ​​attention algorithm

[0161] 370 channel gate module

[0162] 371 Calculation of Weights

[0163] 372 weights applied to memory data

[0164] 373 Weights applied to the output of the attention algorithm

[0165] 374 Calculation of weighted sum

[0166] 375 Layer Norm

[0167] 380 Feedforward Block

[0168] 381 combinations

[0169] 385 Layer Norm

[0170] 410 Grid associated with current input data extended by position information

[0171] 420 A grid associated with a memory data set extended by location information

[0172] 430 Distance relative to reference position

[0173] 440 Projection distance of memory data

[0174] 450 Relative Position

[0175] 500 is a flow chart showing a method for determining a mode associated with an environment of a host vehicle from sequentially recorded data.

[0176] 502 A step of determining, for a current time point and for a predetermined number of previous time points, a corresponding set of characteristics of a plurality of objects detected by a perception system of a host vehicle in an environment of the host vehicle

[0177] 504 A step of generating a current input data set associated with a feature set for a current time point

[0178] 506 A step of generating a memory data set by aggregating a feature set at a previous time point

[0179] 508 A step of applying an attention algorithm to the current input data set and the memory data set to generate a joint spatiotemporal data set

[0180] 510 A step of determining at least one pattern associated with a plurality of objects based on the joint spatiotemporal data set

[0181] 600 Mode Determination System

[0182] 602 Characteristic Determination Circuit

[0183] 604 Current input data generation circuit

[0184] 606 memory data generation circuit

[0185] 608 Attention Algorithm Circuit

[0186] 610 Mode determination circuit

[0187] 611 Connections

[0188] 700 Computer system according to various embodiments

[0189] 702 processor

[0190] 704 Memory

[0191] 706 Non-transient data storage

[0192] 708 connected.

Claims

1. A computer-implemented method for determining a pattern associated with an environment of a host vehicle (100) based on sequentially recorded data, The method comprises: determining a corresponding set of characteristics detected by a perception system (110) of the host vehicle (100) in the environment of the host vehicle (100) for a current point in time and for a predefined number of previous points in time, and Via the processing unit (121) of the host vehicle (100): generating a current input data set associated with the feature set for the current time point (210), generating a memory data set (220) by aggregating the set of characteristics of the previous time points, applying an attention algorithm (230) to the current input data set (210) and the memory data set (220) to generate a joint spatiotemporal data set, and At least one mode of the environment of the host vehicle (100) is determined based on the joint spatiotemporal data set.

2. The method according to claim 1, characterized in that The current input data set (210) and the memory data set (220) are associated with respective grid maps defined for the environment of the host vehicle (100).

3. The method according to claim 2, characterized in that When applying the attention algorithm (230), a match is determined between a cell (310) of the grid map associated with the current input data set (210) and a plurality of cells (320) of the grid map associated with the memory data set (220).

4. The method according to claim 3, characterized in that The attention algorithm (230) includes weights defined by associating elements of the current input data set (210) with allocated elements of the memory data set (220), The weights of the attention algorithm (230) are applied to values ​​generated by taking the union of the elements of the current input data set (210) and the allocated elements of the memory data set (220) to provide the joint spatiotemporal data set as an output (360) of the attention algorithm (230).

5. The method according to claim 4, characterized in that Associating the elements of the current input data set (210) with the allocated elements of the memory data set (220) comprises: generating a query vector by taking said elements of said current input data set (210), generating a key vector by taking the assigned elements of the memory data set (220) and the elements of the current input data set (210), and A dot product of the query vector and the key vector is estimated, and the weight of the attention algorithm (230) is estimated based on the dot product.

6. The method according to claim 5, characterized in that The key vector is generated based on a concatenation of the assigned elements of the memory data set (220) and the elements of the current input data set (210).

7. The method according to claim 6, characterized in that To generate the joint spatiotemporal dataset, the output of the attention algorithm (230) is further combined with the memory dataset (220), and The gating program (370) is configured to define weights for combining corresponding channels within the memory data set (220) with the output (360) of the attention algorithm (230).

8. The method according to any one of claims 1 to 7, characterized in that Information about respective distances (430, 440) relative to a spatial reference point is associated with the current input data set (210) and with the memory data set (220) in order to track movement of an object (130) between one of the previous time points and the current time point.

9. The method according to claim 8, characterized in that The information about the respective distances (430, 440) associated with the memory data set (220) is determined via a motion model including a velocity estimate of the object (130).

10. The method according to any one of claims 1 to 7, characterized in that Position information (450) is associated with elements of the current input data set (210) and allocated elements of the memory data set (220) in order to estimate the velocity of objects in the environment of the host vehicle (100).

11. The method according to any one of claims 1 to 7, characterized in that The at least one pattern is provided to an algorithm for object detection and / or to an algorithm for segmenting the environment of the host vehicle (100), the algorithm for object detection determining at least one of a position, a velocity and coordinates of a bounding box associated with an object of a plurality of objects (130) located in the environment of the host vehicle (100).

12. A computer system (700), the computer system (700) being configured to: receiving, for a current time point and for a predefined number of time points before the current time point, a corresponding set of characteristics detected by the perception system (110) of the host vehicle (100) in the environment of the host vehicle (100), and Execute the computer-implemented method of at least one of claims 1 to 11.

13. A vehicle (100) comprising a perception system (110) and the computer system (700) of claim 12.

14. The vehicle (100) according to claim 13, Further included is a control system (124) configured to receive information derived from the at least one pattern provided by the computer system (700) and apply the information to control the vehicle (100).

15. A non-transitory computer-readable medium comprising instructions for executing the computer-implemented method of at least one of claims 1 to 11.

Images