Transform-based pedestrian trajectory prediction method and device, and medium

Abstract

The invention relates to a pedestrian trajectory prediction method and device based on Transform and a medium, and the method comprises the following steps: inputting pedestrian trajectory data and interaction environment data into a trained pedestrian trajectory prediction model based on Transform, and outputting a pedestrian trajectory prediction result; wherein the pedestrian trajectory prediction model based on Transform comprises an encoder, a social interaction module, a vector network module, an anchor point generation module and a decoder, the encoder is used for carrying out feature extraction on input data, and the social interaction module is used for carrying out pedestrian-vehicle interaction analysis according to pedestrian trajectory features and vehicle trajectory features; the vector network module is used for carrying out human and environment interaction analysis according to pedestrian trajectory features and environment features, the anchor point generation module is used for generating intention anchor points of pedestrian advancing, and the decoder is used for embedding the intention anchor points into outputs of the social interaction module and the vector network module step by step to generate a pedestrian trajectory prediction result. Compared with the prior art, the accuracy of the prediction result can be further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a Transformer-based pedestrian trajectory prediction method, device and medium. Background Art

[0002] With the development of autonomous driving technology, urban autonomous driving is gradually becoming a reality, and the accuracy of pedestrian trajectory prediction has a great impact on the safety and reliability of the autonomous driving system. Pedestrians are the most vulnerable part of traffic participants with changeable behaviors. Their movement patterns are easily affected by the surrounding environment and have a high degree of uncertainty. For example, pedestrians may start, slow down, accelerate, stop suddenly, or even turn sharply when seeing vehicles approaching from a distance. These behaviors are diverse and difficult to predict, and all occur in a very short period of time, which brings huge challenges to accurately detecting pedestrian trajectories and achieving effective collision avoidance.

[0003] Existing pedestrian trajectory prediction methods are mainly divided into model-based methods and data-driven methods. Model-based methods can be further divided into physical model-based methods, behavioral model-based methods, and perceptual interaction model-based methods. Specifically, physical model-based methods predict outputs such as position, speed, and heading angle by considering historical speed, direction, and more additional information such as the vehicle's steering angle, acceleration, weight, and even road friction coefficient; behavioral model-based methods infer the behavioral intention of the current prediction object without considering the influence of other traffic participants, and use the intention as a priori condition for predicting future trajectories; perceptual interaction-based models represent pedestrians and vehicles as interactive dynamic objects, that is, the influence of the movement of other traffic participants in the scene on the movement of the current prediction object is introduced. Model-based methods require a lot of parameter calibration and are often only applicable to a certain scene. They also lack heterogeneity in pedestrian modeling. All pedestrians are modeled using the same parameters, which lacks generalization for various traffic scenes and different pedestrian movement patterns. Data-driven methods can automatically learn features in the data and can handle heterogeneity problems well. Data-driven methods include recurrent neural networks (RNN) and their variants, generative adversarial networks (GAN), graph neural networks (GCN), etc. However, they are still insufficient in dealing with uncertainty and dynamic changes, and it is difficult to accurately reflect the various potential behaviors of pedestrians. In addition, they often have limitations when dealing with the interaction between pedestrians and pedestrians and the environment, and it is difficult to accurately simulate the mutual influence between them. Therefore, it is necessary to design a pedestrian trajectory prediction method to further improve the accuracy of the prediction, thereby improving the safety of autonomous driving. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a pedestrian trajectory prediction method, device and medium based on Transformer, so as to further improve the accuracy of prediction and thus improve the safety of autonomous driving.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] The present invention provides a pedestrian trajectory prediction method based on Transformer, comprising the following steps:

[0007] Acquire videos of mixed-vehicle and pedestrian roads in real time, extract pedestrian trajectory data and interactive environment data, the interactive environment data includes vehicle trajectory data and environmental information, input the trained Transformer-based pedestrian trajectory prediction model, and output the pedestrian trajectory prediction result;

[0008] Among them, the Transformer-based pedestrian trajectory prediction model includes an encoder, a social interaction module, a vector network module, an anchor point generation module and a decoder. The encoder includes multiple stacked Transformer layers, which are used to extract features from the pedestrian trajectory data and the vehicle trajectory data respectively, and output pedestrian trajectory features, vehicle trajectory features and environmental features; the social interaction module is used to perform human-vehicle interaction analysis based on the pedestrian trajectory features and the vehicle trajectory features, including a social tensor encoding layer and a splicing layer connected in sequence; the vector network module is used to perform human-environment interaction analysis based on the pedestrian trajectory features and the environmental features, including a local vector network and a global vector network connected in sequence; the anchor point generation module is used to perform cluster analysis on the pedestrian trajectory features to generate pedestrian movement intention anchor points; the decoder includes multiple stacked Transformer layers, which are used to embed the intention anchor points step by step into the outputs of the social interaction module and the vector network module to generate pedestrian trajectory prediction results.

[0009] Furthermore, the pedestrian trajectory data and the interactive environment data are first converted into high-dimensional vector representations through a feature embedding layer and then input into the encoder.

[0010] Furthermore, the feature embedding layer is built based on a multi-layer perceptron, including a first fully connected layer, a ReLU activation function layer and a second fully connected layer connected in sequence.

[0011] Furthermore, in the encoder and the decoder, each Transformer layer includes a one-dimensional convolution, a multi-head self-attention and a feedforward network connected in sequence.

[0012] Furthermore, in the social interaction module, the social tensor encoding layer is an N0×N0 rectangular network, the trajectory features of the target pedestrian are set as the center of the bottom edge of the rectangular network, and then the rectangular network is filled based on the trajectory features of pedestrians and vehicles around the target pedestrian.

[0013] Furthermore, the output of the social tensor encoding layer passes through two convolutional neural networks and a pooling layer and is then input into the splicing layer for splicing along the time dimension.

[0014] Furthermore, in the vector network module, the local vector network is used to extract nodes of the local graph from the input data, and the global vector network is used to aggregate the nodes of the local graph.

[0015] Furthermore, the anchor point generation module performs cluster analysis on the pedestrian trajectory features according to a K-Means algorithm.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the above method when executing the program.

[0017] The present invention also provides a computer-readable storage medium on which a computer program is stored, and the program implements the above method when executed by a processor.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention provides a pedestrian trajectory prediction method based on Transformer. Firstly, pedestrian trajectory data and interaction environment data are extracted from a video of a mixed pedestrian and vehicle road. The interaction environment data includes vehicle trajectory data and environmental information. Then, a trained pedestrian trajectory prediction model based on Transformer is inputted. Finally, a pedestrian trajectory prediction result is outputted. The pedestrian trajectory prediction model based on Transformer includes an encoder, a social interaction module, a vector network module, an anchor point generation module and a decoder. The encoder includes a plurality of stacked Transformer layers, which can capture the long-range dependency in the time series, accurately and efficiently extract the features of pedestrian trajectory data and vehicle trajectory data, obtain pedestrian trajectory features, vehicle trajectory features and environmental features, and provide a rich information basis for subsequent analysis and prediction. The social interaction module includes a plurality of sequentially connected The social tensor encoding layer and splicing layer are used to analyze human-vehicle interaction based on the characteristics of pedestrian trajectories and vehicle trajectories, which helps the model understand the behavior patterns and decision-making processes of pedestrians in complex traffic environments, thereby improving the accuracy of pedestrian trajectory prediction; the vector network module includes a local vector network and a global vector network connected in sequence, which are used to analyze the interaction between people and the environment based on the characteristics of pedestrian trajectories and environmental characteristics, focusing on the interaction between pedestrians and the surrounding environment, so that the model can adapt to different traffic scenarios and conditions; the anchor point generation module is used to perform cluster analysis on pedestrian trajectory characteristics and generate pedestrian movement intention anchor points, which can reflect the pedestrian's potential destination or direction of travel; the decoder includes multiple stacked Transformer layers, which are used to embed the intention anchor points into the outputs of the social interaction module and the vector network module step by step to generate pedestrian trajectory prediction results, which can further improve the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the structure of the pedestrian trajectory prediction model based on Transformer. DETAILED DESCRIPTION

[0021] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0022] Example:

[0023] This embodiment provides a pedestrian trajectory prediction method based on Transformer, including the following steps:

[0024] First, real-time video of mixed pedestrian and vehicle traffic is acquired and data enhancement processing is performed to extract pedestrian trajectory data and interactive environment data. The pedestrian trajectory data includes the current position and speed of the pedestrian, and the interactive environment data includes vehicle trajectory data and environmental information. The vehicle trajectory data includes the current position and speed of the vehicle, and the environmental information includes lane lines, zebra crossings, and traffic signs.

[0025] Then, the above data is input into the trained Transformer-based pedestrian trajectory prediction model, and finally the pedestrian trajectory prediction result is output;

[0026] like Figure 1 As shown in the figure, the pedestrian trajectory prediction model based on Transformer includes an encoder, a social interaction module, a vector network module, an anchor point generation module and a decoder. The pedestrian trajectory data and the interactive environment data are first converted into a high-dimensional vector representation through the feature embedding layer. The feature embedding layer is built based on MLP (multi-layer perceptron), including the first fully connected layer, the ReLU activation function layer and the second fully connected layer connected in sequence. The output E of the feature embedding layer is a Th×D matrix, which is expressed as follows:

[0027] E=MLP(X)∈R Th×D

[0028] Among them, X represents the input data, D represents the dimension of the embedding vector, and Th represents the length of the historical trajectory.

[0029] Then the input position encoding layer further extracts the timing information and position relationship:

[0030]

[0031] in, and Represents the position encoding matrix PE X The elements in the t-th row and d-th column and the elements in the t-th row and d+1-th column, t represents the position number of the point in the historical trajectory, and d represents the dimension number of the embedded vector.

[0032] The output of the position encoding layer is input into the encoder, which includes multiple stacked Transformer layers. It is used to extract features from pedestrian trajectory data and vehicle trajectory data respectively, and outputs pedestrian trajectory features, vehicle trajectory features, and environmental features. Each Transformer layer includes a one-dimensional convolution, multi-head self-attention, and feedforward network connected in sequence. The Transformer layer has powerful parallel computing capabilities, can capture long-distance dependencies, and accurately and efficiently extract features.

[0033] The social interaction module is used to analyze human-vehicle interaction based on the characteristics of pedestrian and vehicle trajectories, understand the behavior patterns and decision-making processes of pedestrians in complex traffic environments, and thus improve the accuracy of pedestrian trajectory prediction. The social interaction module includes a social tensor encoding layer and a splicing layer connected in sequence. The social tensor encoding layer is an N0×N0 rectangular network. Since pedestrians' attention is mainly focused on the front, the trajectory characteristics of the target pedestrian are set as the center of the bottom edge of the rectangular network, and then the rectangular network is filled based on the trajectory characteristics of pedestrians and vehicles around the target pedestrian. The output of the social tensor encoding layer is passed through two convolutional neural networks (3×3 and 3×1, respectively) and a maximum pooling layer, and then input into the splicing layer for splicing along the time dimension.

[0034] The vector network module is used to analyze the interaction between people and the environment based on the characteristics of pedestrian trajectories and environmental features, focusing on the interaction between pedestrians and the surrounding environment, so that the model can adapt to different traffic scenarios and conditions. The vector network module includes a local vector network and a global vector network connected in sequence. Among them, the local vector network is used to extract the nodes of the local graph from the input data, and the global vector network is used to aggregate the nodes of the local graph. The node features obtained by aggregation can help the model understand the mutual influence between different nodes, thereby obtaining the interactive relationship between pedestrians and the environment.

[0035] The anchor point generation module is used to perform cluster analysis on pedestrian trajectory features and generate pedestrian travel intention anchor points, which can reflect the pedestrian's potential destination or travel direction. In this embodiment, the anchor point generation module performs cluster analysis on the pedestrian trajectory features according to the K-Means algorithm.

[0036] The structure of the decoder is similar to that of the encoder, and also includes multiple stacked Transformer layers. However, the decoder also includes an interactive attention layer to capture interactive information and integrate it into the model for use in subsequent Transformer layers. In addition, in each Transformer layer, the interactive attention layer processes the interaction between the anchor point and the historical trajectory, so that each anchor point can aggregate contextual information related to its own intention. In each layer of Transformer, the anchor point is processed by the self-attention mechanism and the feedforward neural network, and can be continuously adjusted and gradually refined. The decoder can embed the intention anchor point into the output of the social interaction module and the vector network module step by step to generate pedestrian trajectory prediction results. This can effectively decouple the uncertainty of pedestrians' future behavior and improve the stability and accuracy of the prediction.

[0037] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0038] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A pedestrian trajectory prediction method based on Transformer, characterized in that: The following steps are involved: Acquire videos of mixed-vehicle and pedestrian roads in real time, extract pedestrian trajectory data and interactive environment data, the interactive environment data includes vehicle trajectory data and environmental information, input the trained Transformer-based pedestrian trajectory prediction model, and output the pedestrian trajectory prediction result; Among them, the Transformer-based pedestrian trajectory prediction model includes an encoder, a social interaction module, a vector network module, an anchor point generation module and a decoder. The encoder includes multiple stacked Transformer layers, which are used to extract features from the pedestrian trajectory data and the vehicle trajectory data respectively, and output pedestrian trajectory features, vehicle trajectory features and environmental features; the social interaction module is used to perform human-vehicle interaction analysis based on the pedestrian trajectory features and the vehicle trajectory features, including a social tensor encoding layer and a splicing layer connected in sequence; the vector network module is used to perform human-environment interaction analysis based on the pedestrian trajectory features and the environmental features, including a local vector network and a global vector network connected in sequence; the anchor point generation module is used to perform cluster analysis on the pedestrian trajectory features to generate pedestrian movement intention anchor points; the decoder includes multiple stacked Transformer layers, which are used to embed the intention anchor points step by step into the outputs of the social interaction module and the vector network module to generate pedestrian trajectory prediction results.

2. The pedestrian trajectory prediction method based on Transformer according to claim 1, characterized in that: The pedestrian trajectory data and the interactive environment data are first converted into high-dimensional vector representations through a feature embedding layer and then input into the encoder.

3. The pedestrian trajectory prediction method based on Transformer according to claim 2 is characterized in that: The feature embedding layer is built based on a multi-layer perceptron, including a first fully connected layer, a ReLU activation function layer, and a second fully connected layer connected in sequence.

4. The pedestrian trajectory prediction method based on Transformer according to claim 1, characterized in that: In the encoder and the decoder, each Transformer layer includes a one-dimensional convolution, a multi-head self-attention and a feedforward network connected in sequence.

5. The pedestrian trajectory prediction method based on Transformer according to claim 1, characterized in that: In the social interaction module, the social tensor encoding layer is an N0×N0 rectangular network, the trajectory features of the target pedestrian are set as the center of the bottom edge of the rectangular network, and then the rectangular network is filled based on the trajectory features of pedestrians and vehicles around the target pedestrian.

6. The pedestrian trajectory prediction method based on Transformer according to claim 5, characterized in that: The output of the social tensor encoding layer passes through two convolutional neural networks and a pooling layer and then is input into the splicing layer for splicing along the time dimension.

7. The pedestrian trajectory prediction method based on Transformer according to claim 1, characterized in that: In the vector network module, the local vector network is used to extract nodes of the local graph from input data, and the global vector network is used to aggregate the nodes of the local graph.

8. The pedestrian trajectory prediction method based on Transformer according to claim 1, characterized in that: The anchor point generation module performs cluster analysis on the pedestrian trajectory features according to the K-Means algorithm.

9. An electronic device comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images