A trajectory prediction method and device based on dual attention mechanism

Abstract

The present invention discloses a trajectory prediction method and device based on a dual attention mechanism. The method includes: obtaining trajectory data of multiple targets in an observation area in the previous time period; constructing graph structure data, in which nodes and targets correspond one to one; the graph attention network module is configured to extract spatial interaction information between targets and update node features of the graph structure data based on the spatial interaction information; the temporal attention network module extracts temporal interaction information from the updated target features and obtains the predicted trajectory of the target in the next time period based on the extracted temporal interaction information. The fusion of spatial interaction information and temporal interaction information is realized, that is, spatiotemporal interaction information is obtained, which can more effectively model and predict the motion pattern of targets in traffic scenarios to obtain better target trajectory prediction results, and can predict multiple target trajectories in parallel to speed up trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the field of trajectory prediction, and in particular to a trajectory prediction method and device based on a dual attention mechanism. Background Art

[0002] The trajectory data of mobile targets (such as pedestrians, cars, driverless smart cars, mobile robots, etc.) not only contains rich temporal feature information, but also has complex spatial interaction information between targets in the same scene at the same time. The association information between different targets at different times is defined as spatiotemporal association information. For a certain target, the spatial interaction information with the surrounding neighboring targets and the spatiotemporal association information may affect the future trajectory of the target. However, in the prior art, trajectory prediction is often performed based only on the temporal feature information in the target trajectory data, and the prediction accuracy is not high. Summary of the invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a trajectory prediction method and device based on a dual attention mechanism.

[0004] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a trajectory prediction method based on a dual attention mechanism, including: obtaining trajectory data of multiple targets in an observation area in the previous time period; preprocessing the trajectory data of the target, and using the preprocessing result as the initial value of the target feature; constructing graph structure data, in which nodes and targets correspond one to one, the initial value of the node feature is the initial value of the target feature of the corresponding target, and connecting adjacent nodes to form the edge of the graph structure data; inputting the graph structure data into a graph attention network module, the graph attention network module is configured to extract spatial interaction information between targets and update the node features of the graph structure data based on the spatial interaction information, that is, update the target features; inputting the updated target features into the temporal attention network module, the temporal attention network module is configured to extract temporal interaction information from the updated target features and obtain the predicted trajectory of the target in the next time period based on the extracted temporal interaction information.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a trajectory prediction device, including: an acquisition module, used to acquire trajectory data of multiple targets in an observation area in the previous time period, and input the acquired trajectory data into the trajectory prediction module; a graph structure data construction module, used to construct graph structure data and input the graph structure data into a graph attention network module, wherein the nodes in the graph structure data correspond to the targets one by one, the initial value of the node feature is the initial value of the target feature of the corresponding target, and the adjacent nodes are connected to form the edge of the graph structure data; the graph attention network module is configured to extract spatial interaction information between targets and update the node features of the graph structure data based on the spatial interaction information, that is, update the target features, and input the updated target features into the temporal attention network module; the temporal attention network module is configured to extract temporal interaction information from the updated target features and obtain the predicted trajectory of the target in the next time period based on the extracted temporal interaction information.

[0006] In summary, due to the adoption of the above-mentioned technical scheme, the beneficial effects of the present invention are: the spatial interaction information in the target trajectory data is extracted through the graph attention network module, and the spatial interaction information is used to update the target features, and then the updated target features with spatial interaction information are input into the temporal attention network module for extracting temporal interaction information. The temporal interaction information obtained at this time is essentially spatiotemporal correlation information, which realizes the fusion of spatial interaction information and temporal interaction information, and extracts richer information from the trajectory data. In this way, the motion pattern of the target in the traffic scene can be more effectively modeled and predicted to obtain better target trajectory prediction results, and the trajectory prediction of multiple targets in the observation area can be realized in parallel, which speeds up the trajectory prediction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a flowchart of a trajectory prediction method based on a dual attention mechanism in a specific embodiment of the present invention;

[0008] Figure 2 It is a schematic diagram of a partial framework of a trajectory prediction device in a specific embodiment of the present invention;

[0009] Figure 3 It is a schematic diagram of a process of extracting spatial interaction information between adjacent objects by a graph attention network in a specific embodiment of the present invention;

[0010] Figure 4 is a schematic diagram of the structure of a gated activation module in a specific embodiment of the present invention;

[0011] Figure 5 is a schematic diagram of target feature update of a multi-head graph attention mechanism in a specific embodiment of the present invention;

[0012] Figure 6 is a schematic diagram of a two-layer graph attention network structure in a specific implementation of the present invention;

[0013] Figure 7 It is a schematic diagram of the structure of a temporal attention convolutional network in a specific embodiment of the present invention;

[0014] Figure 8 is a schematic diagram of a causal convolution process in a specific implementation manner of the present invention;

[0015] Fig. 9 is a schematic diagram of the operation of a temporal attention convolution module in a specific embodiment of the present invention;

[0016] Fig.10 It is a schematic diagram of the processing process of enhancing the residual module in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0017] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0018] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0019] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0020] The present invention discloses a trajectory prediction method based on a dual attention mechanism. In a preferred embodiment, Figure 1 As shown, including:

[0021] Step S100, obtaining trajectory data of multiple targets in the observation area in the previous time period. The size of the observation area can be set manually, and the observation area is preferably, but not limited to, an area with a radius of 3 meters or more. The size of the previous time period can also be set as needed, such as greater than or equal to 1 second. The target is preferably, but not limited to, a pedestrian, a mobile robot, a vehicle, or an unmanned vehicle. The trajectory data is preferably the coordinate position information of the target.

[0022] Step S200, preprocess the trajectory data of the target, and use the preprocessing result as the initial value of the target feature. Preferably, the specific process is: aggregate the absolute position coordinates and relative position coordinates of each sampling point of the target in the previous time period, and perform linear transformation on the aggregated results of all sampling points of the target to obtain the initial value of the target feature. The linear transformation is preferably, but not limited to, constructing the aggregated results of all sampling points of the target into a column vector in the order of sampling time. For example: let the aggregated results of the absolute position coordinates and relative position coordinates of the i-th target at the t-th sampling point be represents the absolute position coordinates of the i-th target at the t-th sampling point, Represents the relative position coordinates of the i-th target at the t-th sampling point. | represents the position coordinate aggregation symbol. The | operation preferably, but not limited to, places the relative position coordinates after the absolute position coordinates. obs Aggregate results of all sampling points in Composed column vector H i , H i represents the initial value of the target feature of the i-th target.

[0023] Step S300, constructing graph structure data, in which nodes and targets correspond one to one, the initial value of the node feature is the initial value of the target feature of the corresponding target, and connecting adjacent nodes to form edges of the graph structure data. Specifically, Figure 2 As shown, the graph structure data includes T in the previous time period obs The graph structure corresponding to the sampling points. It should be noted that Figure 2 For ease of understanding, the graph structure corresponding to each sampling point is connected to a multi-head gated graph attention network, but it is actually the same multi-head gated graph attention network, which processes the graph structures of multiple sampling points simultaneously and in parallel.

[0024] Step S400, inputting the graph structure data into the graph attention network module, the graph attention network module is configured to extract the spatial interaction information between the targets and update the node features of the graph structure data based on the spatial interaction information, that is, update the target features.

[0025] In this embodiment, preferably, the graph attention network module includes one or more layers of cascaded graph attention networks, and a graph output layer; the graph attention network is configured to: extract the attention correlation coefficient between adjacent targets based on the graph structure data input to the graph attention network, regularize the attention correlation coefficient to obtain the spatial interaction information between adjacent targets, and update the target features based on the spatial interaction information between adjacent targets; the graph output layer outputs the target features of each target updated by the last layer of graph attention network, and the graph output layer outputs the updated target features of each target respectively, assuming that the updated target feature of the i-th target is H′ i . Suppose the next time period contains T pred time point, the predicted absolute position coordinates of target i at the future time t′ are: The relative position coordinates of target i predicted at the future time t′ are:

[0026] In this embodiment, it is further preferred that before the first layer of the graph attention network extracts the attention correlation coefficient between adjacent targets, the initial value of the target feature of the target is cross-multiplied with the shared weight matrix of the graph attention network to obtain the first feature, and the attention correlation coefficient between the targets is extracted based on the first feature of the target. Let H i represents the initial value of the target feature of the i-th target, then WH i represents the first feature of the i-th target. Similarly, WH j represents the first feature of the jth target. Suppose there are N targets in total, i = 1, 2, ..., T obs , i is not equal to j, j = 1, 2, ..., T obs W represents the shared weight matrix of the graph attention network, which is continuously learned during the training process of the graph attention network.

[0027] In this embodiment, Figure 3 The graph attention network extracts spatial interaction information between adjacent targets. a represents the graph attention network, which is usually a one-layer feedforward neural network. Assume that nodes i and j are adjacent, H i , H j They represent the initial values ​​of the target features of the i-th target and the j-th target respectively. The initial values ​​of the target features of the i-th target and the j-th target can be regarded as a time series. Then the first feature WH of the i-th target is i and the first feature WH of the jth target j It can also be viewed as a time series, such as Figure 3 As shown. The attention network extracts the attention correlation coefficient e between the i-th target and the j-th target ij =a(WH i , W.H.j )=a T [WH i ||WH j ], || represents the aggregation of vectors, WH i ||WH j Indicates that WH i The tail and WH j The head of WH is connected. T represents the transpose operation of the matrix, that is, i ||WH j Input the graph attention network a and transpose the output of the graph attention network a. After obtaining the attention correlation coefficients between adjacent targets, the activation SoftMax layer is introduced to regularize all attention correlation coefficients to obtain the spatial interaction information between adjacent targets, as shown in the following formula:

[0028] The SoftMax layer has a LeakyRelu activation function. Node j is the neighboring node of node i, and node k is the other node in the observation area that is not equal to i. k = 1, 2, ..., N.

[0029] In this embodiment, if Figure 3 As shown in Figure 2, the updated node feature H′ is obtained by multiplying the attention correlation coefficient with the original node feature and then summing them up. i ,Right now:

[0030] Where σ(·) represents a nonlinear activation function. N i Represents the set of adjacent nodes of node i in the graph.

[0031] Step S500, inputting the updated target features into the temporal attention network module, the temporal attention network module is configured to extract temporal interaction information from the updated target features and obtain the predicted trajectory of the target in the next time period based on the extracted temporal interaction information.

[0032] In a preferred embodiment, unlike the traditional graph attention network, the graph attention network module also includes a gated activation module; the gated activation module processes the first feature of each target and uses the processing result as the second feature of the target, and updates the target feature based on the spatial interaction information between adjacent targets and the second feature of the target. The original dimension of the trajectory data is low, so extra attention needs to be paid to the information loss and subtle feature loss of the original data, and the gated activation module can dynamically adjust the information loss and better retain the detailed features.

[0033] In this embodiment, the structural diagram of the gate activation module is as follows: Figure 4Before the first feature of the node is multiplied by the graph attention correlation coefficient to output a new feature, the present invention uses a gated activation function to further process the input feature, as follows:

[0034] g i =f g (WH i +b h )⊙(WH i +b h )

[0035] where f g (·) is the tanh activation function, b h is the bias, and ⊙ represents the Hadamard product of element-by-element multiplication. The gating mechanism can ensure that the gradient does not disappear during the training of the graph attention network and stabilize the training process.

[0036] In a preferred embodiment, in order to improve efficiency and simplify model training, the position difference between adjacent targets is added as the prior knowledge of edge features into the calculation process of attention correlation coefficient. The prior knowledge of edge features at time point t in the previous period is:

[0037] Among them, φ r represents a linear transformation, which encodes the relative position information of pedestrians in the same scene into a high-dimensional space F1 represents the preset target feature dimension size, preferably but not limited to 3; W r is a learnable transformation parameter, and the optimal value is learned during training. The spatial interaction information between adjacent targets after adding the prior knowledge of edge features is expressed as:

[0038]

[0039] In a preferred embodiment, in order to extract more spatial interaction features between target trajectory data and improve the accuracy of trajectory prediction, the graph attention network is a multi-head graph attention network. The multi-head graph attention network obtains the spatial interaction information between adjacent targets respectively. Each head graph attention network updates the target features with its own spatial interaction information, and the target features updated by the multi-head graph attention network are aggregated to obtain the final updated target features of the target. The specific calculation process can be seen in Figure 5 As shown, the final updated target feature obtained according to this process can be expressed as:

[0040]

[0041] Among them, || represents the aggregation of vectors, α k，ijrepresents the graph attention correlation coefficient between nodes i and j of the kth head, σ represents the ELU activation function, R(·) represents the graph output layer (which can be a fully connected layer), and W r ′ represents the parameters of the graph output layer.

[0042] In a preferred embodiment, when the graph attention network module includes multiple layers of cascaded graph attention networks, at least one residual connection branch and / or at least one jumper connection branch is provided between the multiple layers of graph attention networks, such as Figure 6 As shown in the figure, a two-layer graph attention network is used to better model spatial relationships. In order to more accurately extract the spatial interaction features of pedestrians in the scene, residual blocks and skip connections are added to the traditional graph attention network to retain more global and detail features. Figure 6 In the figure, two residual connection branches are set in the two-layer graph attention network, namely the branch connecting the input to the output of the first-layer graph attention network and the branch connecting the input to the output of the second-layer graph attention network. A skip branch is also shown, that is, the branch connecting the input of the first-layer graph attention network to the output of the second-layer graph attention network.

[0043] In a preferred embodiment, the temporal attention network module extracts temporal interaction information of different scales from the updated target features through the temporal attention module and the first causal convolution module, and obtains the predicted trajectory of the target in the next time period based on the extracted temporal interaction information of different scales. The trajectory prediction model extracts temporal features of different scales of the trajectory data of target i through the temporal attention module and the first causal convolution module, and obtains the predicted trajectory of target i in the next time period based on the extracted temporal features of different scales. The temporal attention network module can extract the temporal interaction information of all targets in parallel and output the predicted trajectory of the next time period.

[0044] In this embodiment, the recursive neural network model has the ability to process unknown time series data, and recursively outputs time series prediction data by inputting the sequence data and hidden state of the last moment. The data processing mode of the recursive neural network model has two advantages. First, it can process sequences of any length, and the hidden state always has the same size; the second advantage is that the same network parameters can be used for all data. These two factors make the recursive neural network model a concise and reasonable time series processing model. However, the parameter sharing mechanism used by the recursive neural network model depends on the following assumption: the characteristics of the time series data are consistent in time distribution. In fact, since multiple factors and time jointly affect the time series transformation, the above assumptions are not valid in most time series processing. Therefore, the prediction effect of simply using a recursive neural network model to process time series data is not good, and the prediction accuracy is low. In addition, since the output of the next moment must depend on the hidden state of the previous moment, the recursive neural network model cannot achieve parallel processing in the time dimension, and the prediction speed is slow. Time series convolutional neural network models include causal convolution models, void convolution models, etc. The main purpose of the temporal convolutional neural network model is to solve the problems of instability and non-parallelization of recursive neural networks during training, but there are problems such as single feature extraction and low trajectory prediction accuracy.

[0045] In this embodiment, the temporal attention network module includes one or more cascaded temporal attention convolutional networks and a decoder; Figure 7 As shown, the first branch and the second branch are both connected between the input end of the time attention convolution network and the input end of the aggregation module of the time attention convolution network. The first branch is provided with a time attention module, and the second branch is provided with a first causal convolution module. Preferably, for the convenience of calculation, the aggregation module is configured to superimpose the feature data output by the first branch and the second branch. The decoder maps the time interaction information output by the aggregation module of the last layer of the time attention convolution network to the predicted trajectory of the target in the next time period. Preferably, the decoder includes a fully connected layer, and the time interaction information output by the aggregation module of the last layer of the time attention convolution network is mapped to the predicted trajectory of the target in the next time period through a fully connected layer. Specifically, the decoder maps the time interaction information of each target to the predicted trajectory of the target in the next time period.

[0046] In this embodiment, due to the convolution processing of the convolution kernel of the convolutional neural network in the time dimension, this convolution mode is called causal convolution, thereby obtaining a feature with the convolution kernel receptive field size containing time dimension information. The implementation of causal convolution is to add padding data to the left side of the time series data so that the features of a certain moment of the causal output are only extracted from the data before the current moment. In terms of order, it is ensured that the information of the future moment will not be leaked to the information of the previous moment, ensuring the correctness of the timing.

[0047] In this embodiment, if Figure 8 As shown in Figure 1, the data processing process of the first causal convolution in the l-th layer of the temporal attention convolutional network is shown in Figure 1. For the i-th target, let the vector of the first causal convolution input to the l-th layer be Then its output is the input time series of the first causal convolution of the (l+1)th layer As the first causal convolution module is stacked with multiple layers, the receptive field of the features output by the high-level layers also increases linearly. In the causal convolution processing time series, the receptive field corresponds to how many historical moments of data can be observed at the current moment. Assuming that the convolution kernel size of the first causal convolution is k′, the receptive field of the lth layer is (k′-1)*l+1. The specific implementation of causal convolution is a normal convolution module plus a left padding data operation, such as Figure 8 As shown in the figure, the convolution kernel size of the causal convolution kernel is set to 3, and each time before inputting into the causal convolution module, the last dimension of the input data is left-padded with a padding dimension of 2. Through causal convolution, the output features at the current moment are only related to the input data at the current moment and before, and the timing information in the trajectory sequence is retained.

[0048] In this implementation, the causal convolution has the ability to integrate the information of the previous moment into the output of the current moment. However, the causal convolution module only performs ordinary convolution and integration on the historical time series information, and only has a simple aggregation operation on the time interaction information, and cannot select to focus on specific time information. Based on the above shortcomings, the temporal attention module is introduced in the temporal attention convolutional network.

[0049] In this embodiment, the time attention module selectively focuses on a part of the time point information, while ignoring or paying less attention to the rest of the time point information. The attention mechanism assigns more weight to the important information in the input data for feature extraction, and assigns less weight to the unimportant information. When processing time interaction information, the attention mechanism is that the trajectory prediction model focuses on the time period close to the current time point and / or the time period with obvious trajectory changes, and sets smaller weights for some distant time periods or time periods with less trajectory changes.

[0050] In a preferred manner, in order to avoid the traditional self-attention mechanism, when processing time series data, it will pay attention to both historical moments and future moments at the same time, which is not in line with the sequential characteristics of the time series. Based on the self-attention mechanism, the present invention further preferably uses a mask to shield the influence of future moment features on the current moment, that is, the time attention module uses a mask to shield the influence of future moment time interaction information on the current moment time interaction information. The mask can be a value of 0. In this way, the attention mechanism is used to integrate the influence of historical moments on the current moment, and automatically pay attention to the historical time series features with greater influence, such as Fig. 9 As shown, the specific process includes (the target feature processing process of each target is carried out according to the following steps):

[0051] First, the target features of the input time attention module are mapped to the key matrix through the key linear transformation matrix f, the query linear change matrix g, and the value linear change matrix h. Query Matrix Value Matrix represents the time series of the input l-th layer of the temporal attention convolutional network, The actual size is T obs The key linear transformation matrix f, query linear change matrix g, and value linear change matrix h are all of size 2×d k The key linear transformation matrix f, query linear change matrix g, and value linear change matrix h are all model variables of the temporal attention module. Before training the trajectory prediction model, the values ​​of the three matrices are initialized by the existing Gaussian distribution initialization method (such as the Gaussian distribution initialization method disclosed in the website https: / / zhuanlan.zhihu.com / p / 69026430). During the training of the trajectory prediction model, the key linear transformation matrix f, query linear change matrix g, and value linear change matrix h are continuously updated. The specific update method is the conventional setting in the temporal attention mechanism and will not be repeated here. The key matrix, query matrix, and value matrix obtained are all of size T obs ×d k Therefore, the dimensions of the key matrix, query matrix, and value matrix are d k .

[0052] After that, the first matrix Wl is obtained by multiplying the key matrix with the query matrix (l) , k (l)T Denotes the key matrix k (l) inversion.

[0053] After that, the values ​​of the lower triangular elements in the first matrix are retained, and the non-lower triangular elements in the first matrix are assigned 0, that is, the mask is 0, and the second matrix Wl′ is obtained(l) :

[0054] i′ and j′ represent the row and column numbers of the second matrix respectively.

[0055] Afterwards, the second matrix is ​​regularized by the first activation function to obtain the attention weight matrix Wa (l) ; The first activation function is preferably but not limited to the SoftMax activation function.

[0056] Finally, the numerical accumulation value of all elements in the attention weight matrix is ​​obtained, the numerical accumulation value is multiplied by the value matrix, and the multiplication result is used as the attention output feature SA output by the temporal attention module (l) .

[0057] In a preferred embodiment, in order to accurately capture the temporal correlation of the time series, a secondary temporal interaction information extraction is performed, such as Figure 7 As shown, the first branch also includes a second causal convolution module cascaded after the temporal attention module, and the attention output features extracted by the temporal attention module are used as input of the second causal convolution module.

[0058] In a preferred embodiment, relatively important information in the sequence is extracted and directly transmitted to the next layer. The trajectory sequence of the input encoder does not directly participate in the calculation of the regularized attention weight matrix, but is calculated by converting it into a value matrix. Because the mapping dimension of the value matrix is ​​usually smaller than the dimension of the input trajectory sequence, there may be a problem of information loss. Inspired by the residual module, we hope that attention can be directly applied to the input trajectory sequence. There are two main reasons. First, direct application to the input trajectory sequence can reduce information loss during forward propagation and solve the problem of gradient disappearance; second, in the case of neural network training, the attention weight matrix is ​​used to tell the network which content is relatively important. It will strengthen the learning of specific parts and help the model learn the features in the data faster. To this end, the temporal attention convolutional network provided by the present invention also includes a first residual branch, which is connected between the input end of the temporal attention convolutional network and the input end of the aggregation module of the temporal attention convolutional network. An enhanced residual module is provided on the first residual branch; the enhanced residual module converts the attention weight matrix Wa (l) Perform row summation to obtain the weight vector M t , and find the weight vector M t The target features of the temporal attention convolutional network input to this layer (set as layer l) Hadamard SR (l) , Hadamard SR (l) As the enhanced residual of the lth layer Where ⊙ represents the symbol for obtaining the Hadamard product. The specific process is as follows: Fig.10 As shown in Figure 1, the calculation process of the enhanced residual module is shown. The enhanced residual module directly multiplies the input sequence by using the attention weight matrix, which has the effect of retaining the input information, stabilizing the training process of the neural network, and accelerating convergence.

[0059] In a preferred embodiment, Figure 7 As shown in the figure, the temporal attention convolution network also includes a second residual branch Residual (i.e., a skip connection module), which is connected between the input end of the temporal attention convolution network and the input end of the aggregation module of the temporal attention convolution network, that is, the second residual branch Residual directly introduces the input information into the aggregation module. The purpose of adding the second residual branch Residual is to represent the output temporal interaction information of the temporal attention convolution network as a superposition of the original input data and the temporal interaction information of the input data. The reason for retaining the original input data is that it contains rich temporal interaction information, which makes the temporal interaction information of the aggregated data richer and the trajectory prediction more accurate.

[0060] In one embodiment of the present invention, the temporal attention convolutional network includes a first branch, a second branch, a first residual branch and a second residual branch. The specific structure is as follows: Figure 7 As shown in Figure 1, the main functions of the first causal convolution module and the temporal attention module are to extract temporal interaction information, but their specific functions are different. The combination of the two can extract temporal interaction information at multiple scales.

[0061] In this embodiment, the first causal convolution module directly processes the input relative trajectory data, learns and extracts the most obvious time correlation within the sequence, and processes the time dimension from a macro perspective; the time attention module can learn subtle time interaction information and integrate the information of the previous moment into the current moment. The two time processing modules work together to learn time interaction information at multiple scales. The enhanced residual module directly multiplies the input sequence by using the weight matrix of the time attention convolution module, which has the effect of retaining input information, stabilizing the training process of the neural network, and accelerating convergence. The present invention also uses a residual and skip connection module, the purpose of which is to represent the output time interaction information of the time attention convolution network as a superposition of the original input data and the time interaction information of the input data. The reason for retaining the original input data is that it contains rich time interaction information. The aggregation module superimposes the output features of the first branch, the second branch, the first residual branch, and the second residual branch. The feature output by the time attention convolution network is O:

[0062]

[0063] Among them, O represents the output of the l-th layer of temporal attention convolutional network, represents the input data of the l-th layer of the temporal attention convolutional network, SA(·) represents the output features of the temporal attention module, SR(·) represents the output features of the enhanced residual module, and C(·) represents the output features of the first causal convolutional module.

[0064] In a preferred embodiment, the target motion trajectory has diversity and uncertainty (more diversity and uncertainty when the target is a pedestrian). Even in the same scene and the same observed trajectory, due to the different intentions of the pedestrians themselves, their future trajectories will be diverse. Based on this insight, the proposed model of the present invention can generate a diverse predicted trajectory by adding random noise to the spatiotemporal interaction features on the basis of more accurate prediction of future trajectories. In order to make the trajectory output by the trajectory prediction model more in line with the actual situation, the decoder includes a noise adding module and a time output layer. The noise adding module adds random noise to the output features of the last layer of the temporal attention convolutional network, and transmits the features with added random noise to the time output layer, which is mapped to the predicted trajectory of the target in the next time period through the time output layer. The added random noise is preferably, but not limited to, Gaussian distribution noise of (0.1, 0). The time output layer is preferably, but not limited to, a fully connected layer.

[0065] In this embodiment, it is further preferred that in order to allow the trajectory prediction model to have a certain generation space, the present invention also discloses a trajectory prediction model training method. In the training of the trajectory prediction model, the noise adding module of the decoder adds different noises to the output features of the last layer of the temporal attention convolutional network, so as to obtain multiple features with different noises. The temporal output layer maps the multiple features with different noises to prediction trajectories respectively, and then obtains multiple different prediction trajectories; the difference between each prediction trajectory and the true trajectory is calculated, and the prediction trajectory with the smallest difference is back-propagated to update the linear mapping parameters of the temporal output layer. The difference is preferably but not limited to the mean square error value. Specifically, the temporal output layer may include five layers, each of which is a linear mapping.

[0066] In this embodiment, Gaussian distributed noise is added directly to the output features of the temporal convolutional neural network, and then a variety of relative position trajectory data of future moments is output through a multi-layer perceptron decoder network, and then the relative position coordinate trajectory is converted into an absolute position coordinate trajectory for the calculation of the loss function. The present invention uses mean square error as the loss function, and uses a diversity loss function in accordance with Social-GAN to encourage the prediction diversity and flexibility of the prediction model. The specific operation is that the proposed model obtains multiple noise data through multiple random samplings, aggregates them with the time interaction information data output by TACN, and then generates multiple prediction trajectories by the decoder. During the training process, we select the output trajectory that is closest to the real future moment trajectory to calculate the loss function. The specific meaning of the diversity loss function is that the model generates multiple prediction trajectories at the same time, and by calculating the mean square error between these prediction trajectories and the true value, a trajectory with the smallest error is selected for back propagation and training. The present invention also discloses a trajectory prediction device. In a preferred embodiment, the device includes: an acquisition module, which is used to acquire trajectory data of multiple targets in an observation area in the previous time period, and input the acquired trajectory data into the trajectory prediction module; a graph structure data construction module, which is used to construct graph structure data and input the graph structure data into a graph attention network module, wherein the nodes in the graph structure data correspond one-to-one to the targets, the initial value of the node feature is the initial value of the target feature of the corresponding target, and adjacent nodes are connected to form the edge of the graph structure data; the graph attention network module is configured to extract spatial interaction information between targets and update the node features of the graph structure data based on the spatial interaction information, that is, update the target features, and input the updated target features into the time attention network module; the time attention network module is configured to extract time interaction information from the updated target features and obtain the predicted trajectory of the target in the next time period based on the extracted time interaction information.

[0067] In one embodiment of the trajectory prediction method based on the dual attention mechanism proposed in the present invention, the target is pedestrians, and the trajectory prediction method is verified. The trajectory prediction model proposed in the present invention can further learn the spatial interaction information in the trajectory sequence of multiple pedestrians, and input the spatial interaction features of the trajectory data into the temporal attention convolutional network, so that the temporal attention module can generate spatiotemporal correlation information, and the causal convolution can continue to extract the temporal features in the sequence, effectively improving the completeness and accuracy of the model.

[0068] In this embodiment, the pedestrian motion datasets ETH and UCY are used to train and verify the proposed model. These two datasets have five sub-datasets, namely ETH, HOTEL, UNIV, ZARA1, and ZARA2. The input data dimension of the first layer of the graph attention network is 64, the output dimension is 16, and the number of multi-head attention is 2. The input data dimension of the second layer is 32, the output data dimension is 32, and this layer does not use multi-head attention. The output dimension of the position difference feature transformation function of pedestrians in the same scene is 64.

[0069] Two metrics are used to evaluate the accuracy of the proposed model, namely, the mean displacement error and the final displacement error. The lower the two metrics, the better the prediction model.

[0070] Average displacement error (Average displacement error, ADE): obtained by calculating the mean square error (MSE) between the predicted trajectory data of the N targets output by the model and the actual data, as shown in the following formula:

[0071]

[0072] in, is the predicted trajectory data output by the model, is the real trajectory data, T pred Forecast duration.

[0073] Final displacement error (FDE): obtained by performing mean square error calculation between the predicted trajectory data at the last moment output by the model and the real data, as shown in the following formula:

[0074]

[0075] The trajectory prediction accuracy of the Graph-TP-TACN model proposed in the present invention is compared. As shown in Table 1, Graph-TP-TACN can accurately predict trajectories.

[0076] Table 1 ADE / FDE of the trajectory prediction method Graph-TP-TACN proposed in this invention

[0077]

[0078]

[0079] After training with the ETH and UCY datasets, we compared the trajectory prediction and accuracy of the Graph-TP-TACN model (the prediction model of the present invention) with the trained Social-LSTM, Social-GAN and STGAT models on the self-made datasets, as shown in Table 2. It can be seen that in the actual campus scene, the average displacement error of the proposed model is still lower than that on the standard dataset, which shows that it has good accuracy, further illustrating that the proposed model has good generalization ability.

[0080] In addition, we selected five mainstream passenger trajectory prediction models for prediction accuracy comparison, namely:

[0081] (1) The SR-LSTM model extracts the intention of the crowd at every moment and jointly iteratively updates and refines the current status of all pedestrians through a message passing mechanism.

[0082] (2) The Sophie model introduces social attention and physical attention mechanisms based on the GAN model and uses LSTM for prediction.

[0083] (3) The trajectron model uses LSTM to extract spatiotemporal information and a conditional variational autoencoder

[58] to generate future trajectories.

[0084] (4) After the graph convolutional network extracts the spatiotemporal features, the Social-STGCNN model uses a time extrapolation convolutional neural network to directly operate on the spatiotemporal features to generate trajectories.

[0085] (5) The STAGT model uses LSTM to extract temporal interaction information and the original graph attention network to extract spatial interaction information, and aggregates and decodes the spatiotemporal interaction information to output the predicted trajectory.

[0086] Table 2 ADE / FDE of different algorithms on the square, supermarket and teaching building datasets

[0087]

[0088] Trajectory prediction tasks are usually used in real-time systems such as autonomous driving and video surveillance, so it is necessary to evaluate the real-time performance of the proposed algorithm. This paper compares the running speed of several trajectory prediction models based on recursive neural networks and several models based on graph convolutional networks, and evaluates the real-time performance on a simulation platform equipped with an Inter Core i7-10700K CPU and an Nvidia RTX3090 GPU. The results are shown in Table 3. From the running results, it can be seen that the algorithm proposed in this invention runs fast and has better real-time performance.

[0089] Table 3 Comparison of running speed of different trajectory prediction models

[0090]

[0091] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0092] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A trajectory prediction method based on a dual attention mechanism, characterized in that: include: Obtain trajectory data of multiple targets in the observation area in the previous time period; Preprocess the target's trajectory data and use the preprocessing result as the initial value of the target feature; Constructing graph structure data, in which nodes and targets correspond one to one, the initial value of the node feature is the initial value of the target feature of the corresponding target, and connecting adjacent nodes to form edges of the graph structure data; Inputting the graph structure data into a graph attention network module, wherein the graph attention network module is configured to extract spatial interaction information between targets and update node features of the graph structure data based on the spatial interaction information, that is, update target features; The updated target features are input into a temporal attention network module, which is configured to extract temporal interaction information from the updated target features and obtain a predicted trajectory of the target in the next time period based on the extracted temporal interaction information.

2. The trajectory prediction method based on the dual attention mechanism as claimed in claim 1, characterized in that: The target's trajectory data is preprocessed, and the preprocessing result is used as the initial value of the target feature. Specifically, the absolute position coordinates and relative position coordinates of each sampling point of the target in the previous time period are aggregated, and the aggregation results of all sampling points of the target are linearly transformed to obtain the initial value of the target feature.

3. The trajectory prediction method based on the dual attention mechanism as described in claim 1 or 2, characterized in that: The graph attention network module includes one or more layers of cascaded graph attention networks and a graph output layer; The graph attention network is configured to: extract attention correlation coefficients between adjacent targets based on graph structure data input into the graph attention network, perform regularization processing on the attention correlation coefficients to obtain spatial interaction information between adjacent targets, and update target features based on the spatial interaction information between adjacent targets; The graph output layer outputs the target features of each target after being updated by the last layer of graph attention network.

4. The trajectory prediction method based on the dual attention mechanism as claimed in claim 3, characterized in that: Before the first layer of graph attention network extracts the attention correlation coefficients between adjacent targets, the initial value of the target feature of the target is cross-multiplied with the shared weight matrix of the graph attention network to obtain the first feature, and the attention correlation coefficients between the targets are extracted based on the first feature of the target.

5. The trajectory prediction method based on the dual attention mechanism as claimed in claim 4, characterized in that: The graph attention network module also includes a gated activation module; The gated activation module processes the first feature of each target and uses the processing result as the second feature of the target, and updates the target feature based on the spatial interaction information between adjacent targets and the second feature of the target.

6. The trajectory prediction method based on the dual attention mechanism as claimed in claim 3, characterized in that: The position difference between adjacent targets is used as the graph edge feature prior knowledge to calculate the attention correlation coefficient.

7. The trajectory prediction method based on the dual attention mechanism as described in claim 4, 5 or 6, characterized in that: The graph attention network is a multi-head graph attention network, which respectively obtains spatial interaction information between adjacent targets. Each graph attention network updates the target features with its own spatial interaction information, and aggregates the target features updated by the multi-head graph attention networks to use the aggregated target features as the target features finally updated by the target.

8. The trajectory prediction method based on the dual attention mechanism as described in claim 4, 5 or 6, characterized in that: When the graph attention network module includes multiple layers of cascaded graph attention networks, at least one residual connection branch and / or at least one jumper connection branch is arranged between the multiple layers of graph attention networks.

9. The trajectory prediction method based on the dual attention mechanism as claimed in claim 1, characterized in that: The temporal attention network module extracts temporal interaction information of different scales from the updated target features through the temporal attention module and the first causal convolution module, and obtains the predicted trajectory of the target in the next time period based on the extracted temporal interaction information of different scales.

10. The trajectory prediction method based on the dual attention mechanism as claimed in claim 9, characterized in that: The temporal attention network module includes one or more cascaded temporal attention convolutional networks and a decoder; The temporal attention convolution network comprises a first branch, a second branch and an aggregation module, wherein the first branch and the second branch are both connected between an input end of the temporal attention convolution network and an input end of the aggregation module of the temporal attention convolution network, the first branch is provided with a temporal attention module, and the second branch is provided with a first causal convolution module; The decoder maps the temporal interaction information output by the aggregation module of the last layer of the temporal attention convolutional network into the predicted trajectory of the target in the next time period.

11. The trajectory prediction method based on the dual attention mechanism as claimed in claim 9, characterized in that: The temporal attention module uses a mask to shield the influence of temporal interaction information at future moments on the temporal interaction information at the current moment.

12. The trajectory prediction method based on the dual attention mechanism as claimed in claim 11, characterized in that: The processing process of the temporal attention module includes: Mapping the target features input into the temporal attention module into a key matrix, a query matrix, and a value matrix through a key linear transformation matrix, a query linear change matrix, and a value linear change matrix respectively; A first matrix is ​​obtained by performing a dot multiplication of a key matrix and a query matrix, the values ​​of the lower triangular elements in the first matrix are retained, and the non-lower triangular elements in the first matrix are assigned to 0 to obtain a second matrix; the second matrix is ​​regularized by a first activation function to obtain an attention weight matrix; Obtain the numerical cumulative value of all elements in the attention weight matrix, multiply the numerical cumulative value by the value matrix and use the multiplication result as the temporal attention output feature output by the temporal attention module.

13. The trajectory prediction method based on the dual attention mechanism as claimed in claim 12, characterized in that: The temporal attention convolutional network also includes a first residual branch, the first residual branch is connected between an input end of the temporal attention convolutional network and an input end of an aggregation module of the temporal attention convolutional network, and an enhanced residual module is arranged on the first residual branch; The enhanced residual module performs row summation on the attention weight matrix to obtain a weight vector, and obtains the Hadamard product of the weight vector and the target feature input into the temporal attention convolutional network.

14. The trajectory prediction method based on the dual attention mechanism as claimed in claim 10, 11, 12 or 13, characterized in that: The first branch also includes a second causal convolution module cascaded after the temporal attention module.

15. The trajectory prediction method based on the dual attention mechanism according to claim 9, 10, 11, 12 or 13, characterized in that: The temporal attention convolutional network also includes a second residual branch, which is connected between the input end of the temporal attention convolutional network and the input end of the aggregation module of the temporal attention convolutional network, and the second residual branch outputs the target features input into the temporal attention convolutional network to the input end of the aggregation module of the temporal attention convolutional network.

16. The trajectory prediction method based on the dual attention mechanism as claimed in claim 10, 11, 12 or 13, characterized in that: The decoder includes a noise adding module and a time output layer. The noise adding module adds random noise to the output feature sequence of the last layer of the time attention convolutional network, and transmits the feature sequence after adding random noise to the time output layer, which is mapped into the predicted trajectory of the target in the next time period through the time output layer.

17. A trajectory prediction device, characterized in that: include: An acquisition module is used to acquire trajectory data of multiple targets in the observation area in the previous time period, and input the acquired trajectory data into a trajectory prediction module; A graph structure data construction module, used to construct graph structure data and input the graph structure data into a graph attention network module, wherein nodes in the graph structure data correspond to targets one by one, the initial value of the node feature is the initial value of the target feature of the corresponding target, and adjacent nodes are connected to form edges of the graph structure data; A graph attention network module is configured to extract spatial interaction information between targets and update node features of graph structure data based on the spatial interaction information, that is, update target features, and input the updated target features into the temporal attention network module; The temporal attention network module is configured to extract temporal interaction information from the updated target features and obtain the predicted trajectory of the target in the next time period based on the extracted temporal interaction information.

Images