Pedestrian trajectory prediction method and device

Abstract

The invention provides a pedestrian trajectory prediction method and device. According to the pedestrian trajectory prediction method based on the hidden virtual interaction force field, calculation elements of the virtual interaction force field serve as feature input, and a graph neural network is combined to calculate an adjacent matrix so as to carry out trajectory prediction; in addition, deep learning models such as a multi-head attention mechanism and a graph neural network are utilized to construct a pedestrian trajectory prediction model. Compared with a traditional pedestrian trajectory prediction technology based on learning, the pedestrian trajectory prediction method has higher interpretability; meanwhile, due to a hidden virtual interaction site calculation mode, unnecessary calculation amount is reduced, and the model operation efficiency and the real-time performance of trajectory prediction are improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving and machine learning technology, and specifically to a method and device for predicting pedestrian trajectories. Background Art

[0002] With the development of autonomous driving technology, the scenarios involved are becoming more and more diverse. Among them, in urban scenarios, there are a large number of specific interaction scenarios, including the process of interaction between vehicles and humans. Therefore, predicting the trajectory of pedestrians to be interacted has become a basic requirement of autonomous driving technology. Pedestrian trajectory prediction technology in related technologies mainly includes the following two types: rule-based trajectory prediction technology and learning-based trajectory prediction technology.

[0003] Among them, rule-based trajectory prediction technology mainly uses methods such as linear functions to fit pedestrian trajectories and predict their future trajectories. However, considering that pedestrian trajectories are relatively flexible and that there are many factors such as surrounding pedestrians and the environment that affect the specific direction of pedestrian movement, this technology is difficult to handle pedestrian trajectory prediction problems under strong interactions. Learning-based trajectory prediction technology mainly uses deep neural networks to learn the characteristics of pedestrian actions based on existing data sets and predict pedestrian future trajectories. This technology can effectively predict pedestrian trajectories under complex conditions, but it lacks interpretability and is prone to incomprehensible trajectory prediction results.

[0004] At the same time, there are also a lot of studies on pedestrians and their interactions. Among them, the more common technology is mainly virtual interaction force field (see Wang, Jianqiang, Jian Wu, and Yang Li. "The driving safety field based on driver–vehicle–road interactions." IEEE Transactions on Intelligent Transportation Systems 16.4 (2015): 2203-2214). This type of method mainly constructs a pedestrian interaction model by calculating the interaction force generated by surrounding pedestrians or vehicles on the predicted pedestrian or calculating the interaction force field generated in the area. From the results, the virtual interaction force field model can more intuitively reflect the interaction relationship between pedestrians, but its calculation process is relatively complicated. It is necessary to calculate the virtual interaction force field generated by each pedestrian or vehicle in the area at each point in the area, which consumes a lot of calculations. Summary of the invention

[0005] At least one embodiment of the present application provides a pedestrian trajectory prediction method and device, which can reduce the amount of calculation in trajectory prediction and improve the efficiency and real-time performance of trajectory prediction.

[0006] According to one aspect of the present application, at least one embodiment provides a pedestrian trajectory prediction method, including:

[0007] Acquire the target pedestrian's data to be predicted, the data to be predicted including the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, the first sampling moment being the last data sampling moment in the first time period;

[0008] The to-be-predicted data of the target pedestrian is input into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0009] Optionally, the historical trajectory data of the target pedestrian is a vector matrix composed of at least one pedestrian posture vector of the target pedestrian, and the pedestrian posture vector is a vector composed of position data and speed data of the pedestrian at the same data sampling moment;

[0010] The surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians closest to each other;

[0011] The implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of nearby pedestrians are vectors composed based on the relative distance and relative speed between the target pedestrian and the nearby pedestrians.

[0012] Optionally, the relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane.

[0013] Optionally, the implicit virtual interaction force vector of the surrounding pedestrians is a vector formed by the square of the relative distance and the square of the relative speed between the target pedestrian and the surrounding pedestrians.

[0014] Optionally, before obtaining the target pedestrian's data to be predicted, the method further includes:

[0015] Acquire a training set, the training set including multiple training data, each training data including: training trajectory data and real trajectory data of the same historical pedestrian, posture data of pedestrians around the historical pedestrian at a second sampling moment, and implicit virtual interaction force data, wherein the training trajectory data and the real trajectory data are the historical trajectory data of the historical pedestrian in a second time period and a third time period, respectively, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period;

[0016] The pedestrian trajectory prediction model is trained based on the training set to obtain a trained pedestrian trajectory prediction model.

[0017] Optionally, the pedestrian trajectory prediction model includes a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module, a pedestrian interaction feature extraction module and a model loss calculation module; training the pedestrian trajectory prediction model based on the training data set includes:

[0018] Inputting the historical pedestrian training trajectory data into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module;

[0019] Inputting the pedestrian trajectory features and the implicit virtual interaction force data of the historical pedestrians into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module;

[0020] Inputting the virtual interaction features and the posture data of pedestrians around the historical pedestrian into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module;

[0021] Inputting the predicted trajectory data and the real trajectory data of the historical pedestrian into the model loss calculation module, and calculating the training loss through the model loss calculation module;

[0022] According to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met to obtain a trained pedestrian trajectory prediction model.

[0023] Optionally, the training loss is calculated by the model loss calculation module, including:

[0024] Calculating a first average distance deviation between the position data at corresponding data sampling moments in the actual trajectory data and the predicted trajectory data of the historical pedestrian;

[0025] Calculating a second average distance deviation between position data of a corresponding feature point in the real trajectory data and the predicted trajectory data of the historical pedestrian, the feature point being a predetermined data sampling moment;

[0026] Calculating a third distance deviation between the actual trajectory data and the end point of the predicted trajectory data of the historical pedestrian;

[0027] The training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation.

[0028] Optionally, the pedestrian trajectory prediction model further includes a prediction trajectory testing module;

[0029] The predicted trajectory testing module is used to calculate the first average distance deviation and the third distance deviation based on the predicted trajectory data generated by the test set, and generate evaluation data of the predicted trajectory data according to the first average distance deviation and the third distance deviation.

[0030] According to another aspect of the present application, at least one embodiment provides a pedestrian trajectory prediction device, including:

[0031] A first acquisition module is used to acquire the target pedestrian's to-be-predicted data, wherein the to-be-predicted data includes the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, wherein the first sampling moment is the last data sampling moment in the first time period;

[0032] The prediction module is used to input the to-be-predicted data of the target pedestrian into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0033] Optionally, the historical trajectory data of the target pedestrian is a vector matrix composed of at least one pedestrian posture vector of the target pedestrian, and the pedestrian posture vector is a vector composed of position data and speed data of the pedestrian at the same data sampling moment;

[0034] The surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians that are closest to each other;

[0035] The implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of nearby pedestrians are vectors composed based on the relative distance and relative speed between the target pedestrian and the nearby pedestrians.

[0036] Optionally, the relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane.

[0037] Optionally, the implicit virtual interaction force vector of the surrounding pedestrians is a vector formed by the square of the relative distance and the square of the relative speed between the target pedestrian and the surrounding pedestrians.

[0038] Optionally, the above device further comprises:

[0039] A second acquisition module is used to acquire a training set, wherein the training set includes multiple training data, each of which includes: training trajectory data and real trajectory data of the same historical pedestrian, and surrounding pedestrian posture data and implicit virtual interaction force data of the historical pedestrian at a second sampling moment, wherein the training trajectory data and the real trajectory data are the historical trajectory data of the historical pedestrian in a second time period and a third time period, respectively, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period;

[0040] The training module is used to train the pedestrian trajectory prediction model based on the training set to obtain a trained pedestrian trajectory prediction model.

[0041] Optionally, the pedestrian trajectory prediction model includes a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module, a pedestrian interaction feature extraction module and a model loss calculation module; the training module is further used to:

[0042] Inputting the historical pedestrian training trajectory data into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module;

[0043] Inputting the pedestrian trajectory features and the implicit virtual interaction force data of the historical pedestrians into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module;

[0044] Inputting the virtual interaction features and the posture data of pedestrians around the historical pedestrian into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module;

[0045] Inputting the predicted trajectory data and the real trajectory data of the historical pedestrian into the model loss calculation module, and calculating the training loss through the model loss calculation module;

[0046] According to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met to obtain a trained pedestrian trajectory prediction model.

[0047] Optionally, the training module is further used to:

[0048] Calculating a first average distance deviation between the position data at corresponding data sampling moments in the actual trajectory data and the predicted trajectory data of the historical pedestrian;

[0049] Calculating a second average distance deviation between position data of a corresponding feature point in the real trajectory data and the predicted trajectory data of the historical pedestrian, the feature point being a predetermined data sampling moment;

[0050] Calculating a third distance deviation between the actual trajectory data and the end point of the predicted trajectory data of the historical pedestrian;

[0051] The training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation.

[0052] According to another aspect of the present application, at least one embodiment provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described above are implemented.

[0053] Compared with the prior art, the pedestrian trajectory prediction method and device provided in the embodiment of the present application realizes a pedestrian trajectory prediction method based on an implicit virtual interactive force field, which uses the calculation elements of the virtual interactive force field as feature inputs, and combines the graph neural network to calculate the adjacency matrix for trajectory prediction; in addition, the embodiment of the present application uses deep learning models such as multi-head attention mechanism and graph neural network to construct a pedestrian trajectory prediction model. Compared with traditional learning-based pedestrian trajectory prediction technology, the embodiment of the present application has stronger interpretability; at the same time, the implicit virtual interactive stance calculation method reduces unnecessary calculations, improves model operation efficiency and the real-time performance of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0055] Figure 1 A schematic diagram of a flow chart of a pedestrian trajectory prediction method according to an embodiment of the present application;

[0056] Figure 2Another schematic diagram of a flow chart of a pedestrian trajectory prediction method according to an embodiment of the present application;

[0057] Figure 3 A diagram showing an example of a structure of a pedestrian trajectory prediction model according to an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the training process of the pedestrian trajectory prediction model according to an embodiment of the present application;

[0059] Figure 5 Another structural example diagram of a pedestrian trajectory prediction model according to an embodiment of the present application;

[0060] Figure 6 A schematic diagram of pedestrian trajectory data preprocessing in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the structure of a pedestrian trajectory prediction device according to an embodiment of the present application;

[0062] Figure 8 Another structural schematic diagram of the pedestrian trajectory prediction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0064] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. "And / or" in the specification and claims represents at least one of the connected objects.

[0065] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the functions and arrangements of the elements discussed without departing from the spirit and scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0067] In view of the limitations of the current pedestrian trajectory prediction methods, the embodiment of the present application designs a pedestrian trajectory prediction method based on an implicit virtual interaction force field. This method is based on a deep learning model that combines a multi-head attention mechanism and a graph neural network. At the same time, it uses the idea of ​​a virtual interaction force field to construct an implicit virtual interaction force field as a model feature input to enhance the interpretability of the model. Compared with traditional learning-based pedestrian trajectory prediction technology, this method benefits from a more intuitive virtual interaction force field and has stronger interpretability. At the same time, compared with traditional virtual interaction force field calculation methods, this method can save a lot of computing power, while avoiding the loss of precision caused by processes such as rasterization, thereby improving the efficiency and accuracy of calculations.

[0068] Specifically, the current virtual interaction force field calculation method is relatively complex, and a certain degree of accuracy will be lost when rasterized. The embodiment of the present application designs a deep learning input method for extracting virtual interaction force field features. This method removes the linear calculation process of the virtual interaction force field, retains only the distance and relative speed square features of the intelligent agent as the feature input of the deep learning model, and constructs the adjacency matrix of the virtual interaction force field through the graph neural network to construct an implicit virtual interaction force field.

[0069] The embodiment of the present application also designs a pedestrian trajectory prediction model and training method based on deep learning. The embodiment of the present application designs a pedestrian trajectory prediction model based on deep learning, which applies a deep learning input method for extracting virtual interactive force field features and combines deep learning methods such as multi-head attention mechanism and graph neural network; at the same time, the embodiment of the present application also designs a training method for the model, including a pedestrian data preprocessing method and a training method.

[0070] The embodiment of the present application also designs an experiment and performance evaluation method for a pedestrian trajectory prediction model. The embodiment of the present application designs an experiment for the pedestrian trajectory prediction model and designs a corresponding performance evaluation method to evaluate the performance of the pedestrian trajectory prediction model. The method divides the data set into a training set and a test set, uses the test set for training, and calculates the average distance deviation and the end point distance deviation between the predicted trajectory and the true trajectory as a performance evaluation method.

[0071] Figure 1 A flow chart of the pedestrian trajectory prediction method of the embodiment of the present application includes two parts: pedestrian trajectory data preprocessing and pedestrian trajectory prediction. After calculation of these two parts, the pedestrian trajectory prediction result is obtained.

[0072] Figure 2 Another flow chart of the pedestrian trajectory prediction method of the present application embodiment is as follows: Figure 2 As shown, the method includes:

[0073] Step 21, obtaining the target pedestrian's data to be predicted, the data to be predicted including the target pedestrian's historical trajectory data within a first time period, surrounding pedestrian posture data of multiple surrounding pedestrians closest to the target pedestrian at a first sampling moment, and implicit virtual interaction force data, wherein the first sampling moment is the last data sampling moment of the first time period.

[0074] Here, the target pedestrian is a pedestrian whose trajectory is to be predicted. The embodiment of the present application can collect trajectory data of the target pedestrian and its surrounding pedestrians according to a preset sampling period (for example, once every 0.2 seconds). In this way, the data of a trajectory point of the pedestrian can be obtained at each data sampling moment, and the data of the trajectory point can specifically include information such as the position coordinates of the trajectory point and the speed of the pedestrian at the trajectory point. The position coordinates may include coordinates in the x-direction and coordinates in the y-direction, and the speed may include speed in the x-direction and speed in the y-direction. The x-direction and the y-direction are two perpendicular directions on a predefined two-dimensional plane. For example, the length direction of the road is taken as the x-direction, and the width direction of the road is taken as the y-direction, so as to collect the trajectory data of the pedestrian. In this way, the data to be predicted for trajectory prediction can be obtained by preprocessing the collected trajectory data to predict the trajectory of the target pedestrian. The data to be predicted specifically includes:

[0075] (1) Historical trajectory data of the target pedestrian in the first time period.

[0076] Assuming that the trajectory of the target pedestrian after the first sampling moment is to be predicted, the time period up to the first sampling moment can be taken as the first time period to obtain the historical trajectory data of the target pedestrian in the first time period. In other words, the first sampling moment is the last data sampling moment of the first time period.

[0077] Here, the historical trajectory data of a pedestrian is a vector matrix composed of at least one pedestrian pose vector of the pedestrian in a period of time. For example, the historical trajectory data of the target pedestrian in the first time period can be a matrix composed of the various pedestrian pose vectors of the target pedestrian at various data collection moments in the first time period. The pedestrian pose vector of a pedestrian is a vector composed of the position data and speed data of the pedestrian at the same data sampling moment. For example, assuming that the coordinate of the pedestrian in the x direction at a certain data sampling moment is x i , the position coordinate in the y direction is y i , the velocity in the x direction is v xi , the velocity in the y direction is v yi , then the pedestrian pose vector of the pedestrian at the data sampling time can be expressed as [x i y i v xi v yi ].

[0078] (2) The posture data of pedestrians around the target pedestrian at the first sampling moment.

[0079] Here, the surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians that are closest to the target pedestrian. The embodiment of the present application can calculate the relative distance between pedestrians based on the trajectory data of each pedestrian obtained at the first sampling moment, and then determine the multiple surrounding pedestrians that are closest to the target pedestrian at the first sampling moment. The specific number of the multiple surrounding pedestrians can be a preset fixed value, such as 8 or more. Then, the pedestrian posture vectors of the multiple surrounding pedestrians at the first sampling moment are obtained, and a vector matrix is ​​constructed based on the pedestrian posture vectors of these surrounding pedestrians, that is, the surrounding pedestrian posture data of the target pedestrian at the first sampling moment. When the number of pedestrians surrounding the target pedestrian is less than the fixed value, appropriate surrounding pedestrian posture data can be generated by zero padding for ease of calculation.

[0080] (3) Implicit virtual interaction force data of the target pedestrian at the first sampling moment.

[0081] Here, the implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of the nearby pedestrians are vectors composed of the relative distance and relative speed between the target pedestrian and the nearby pedestrians. The relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane. The relative distance in a certain direction can be obtained by subtracting the position coordinates in that direction, and the relative speed in a certain direction can also be obtained by subtracting the speeds in that direction.

[0082] For example, the implicit virtual interaction force vector of the surrounding pedestrians can be formed by the square of the relative distance between the target pedestrian and the surrounding pedestrians and the square of the relative speed. Of course, the implicit virtual interaction force vector of the surrounding pedestrians can also be formed by the fourth power (or sixth power) of the relative distance between the target pedestrian and the surrounding pedestrians and the fourth power (or sixth power) of the relative speed. Then, based on the implicit virtual interaction force vectors of these surrounding pedestrians, a vector matrix is ​​constructed, that is, the implicit virtual interaction force data of the target pedestrian at the first sampling moment.

[0083] Step 22: input the target pedestrian's to-be-predicted data into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0084] Here, the pedestrian trajectory prediction model can be a deep learning model based on a multi-head attention mechanism and a graph neural network. By inputting the target pedestrian's to-be-predicted data into the pedestrian trajectory prediction model, an adjacency matrix calculated by the pedestrian trajectory prediction model is obtained. The adjacency matrix is ​​the predicted trajectory data, which specifically includes the position data of the target pedestrian for a period of time after the first time period. The predicted trajectory data includes the predicted position data of the target pedestrian at multiple predicted moments after the first time period.

[0085] Figure 3 A structural schematic diagram of a pedestrian trajectory prediction model of an embodiment of the present application is provided, including: a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module and a pedestrian interaction feature extraction module.

[0086] In step 22, the historical trajectory data of the target pedestrian is input into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features of the target pedestrian extracted by the pedestrian trajectory feature extraction module.

[0087] For example, Figure 3 In the method, the pedestrian trajectory feature extraction module includes two levels of multi-head attention mechanisms, each of which includes a multi-head attention module (Multi-head Attention), a regularization layer (Dropout) and a fully connected layer (Full Connected Layer), and the historical trajectory data of the target pedestrian is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the first-level multi-head attention mechanism. The output of the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the fully connected layer in the second-level multi-head attention mechanism outputs the pedestrian trajectory features of the target pedestrian.

[0088] Then, the pedestrian trajectory characteristics of the target pedestrian and the implicit virtual interaction force data of the target pedestrian at the first sampling moment are input into the virtual interaction feature extraction module to obtain the virtual interaction features of the target pedestrian extracted by the virtual interaction feature extraction module.

[0089] For example, Figure 3In the example, the virtual interaction feature extraction module also includes a similar two-level multi-head attention mechanism, in which the pedestrian trajectory features of the target pedestrian are simultaneously used as the Q and K inputs of the multi-head attention module in the first-level multi-head attention mechanism, and the implicit virtual interaction force data of the target pedestrian at the first sampling moment is used as the V input of the multi-head attention module in the first-level multi-head attention mechanism. The output of the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the fully connected layer in the second-level multi-head attention mechanism outputs the virtual interaction features of the target pedestrian.

[0090] Then, the virtual interaction features of the target pedestrian and the posture data of pedestrians around the target pedestrian at the first sampling moment are input into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the target pedestrian generated by the pedestrian interaction feature extraction module.

[0091] For example, Figure 3 In the method, the pedestrian interaction feature extraction module also includes a two-level multi-head attention mechanism and a linear layer, wherein the first-level attention mechanism includes a multi-head attention module and a regularization layer, and the second-level attention mechanism includes a multi-head attention module, a regularization layer, and a full connected layer. The virtual interaction features of the target pedestrian are simultaneously used as the Q and K inputs of the multi-head attention module in the first-level multi-head attention mechanism, and the surrounding pedestrian posture data of the target pedestrian at the first sampling time is used as the V input of the multi-head attention module in the first-level multi-head attention mechanism. The output of the regularization layer in the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the adjacency matrix output by the fully connected layer in the second-level multi-head attention mechanism is used as the input of the linear layer. The linear layer converts the adjacency matrix into the matrix size corresponding to the predicted trajectory data to obtain the predicted trajectory data.

[0092] It should be noted that Figure 3 The network structure shown is only a specific structure of a deep learning model that can be used in the embodiment of the present application, and the embodiment of the present application is not limited thereto. Those skilled in the art can appropriately increase or decrease the specific layer structure in the above model as needed, and can even use other types of neural network models, which can all be applied to the present application.

[0093] Through the above steps, the embodiment of the present application uses the calculation elements of the virtual interactive force field as feature input, and combines the graph neural network to calculate the adjacency matrix for trajectory prediction; in addition, the embodiment of the present application uses deep learning models such as multi-head attention mechanism and graph neural network to build a pedestrian trajectory prediction model. Compared with traditional learning-based pedestrian trajectory prediction technology, the embodiment of the present application has stronger interpretability; at the same time, the implicit virtual interactive stance calculation method reduces unnecessary calculations, improves model operation efficiency and the real-time performance of trajectory prediction.

[0094] Before step 21, the embodiment of the present application may pre-train the pedestrian trajectory prediction model. The training of the pedestrian trajectory prediction model is described below. Figure 4 As shown, in the embodiment of the present application, the training of the pedestrian trajectory prediction model includes:

[0095] Step 41, obtaining a training set, wherein the training set includes multiple training data, each training data includes: training trajectory data and real trajectory data of the same historical pedestrian, surrounding pedestrian posture data of the historical pedestrian at the second sampling moment, and implicit virtual interaction force data.

[0096] Here, the training trajectory data is the historical trajectory data of the historical pedestrian in the second time period, and the real trajectory data is the historical trajectory data of the historical pedestrian in the third time period. The historical trajectory data is a vector matrix composed of at least one pedestrian posture vector in the corresponding time period. Here, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period. The second time period and the third time period each include the same number of data sampling moments.

[0097] Step 42: Train the pedestrian trajectory prediction model based on the training set to obtain a trained pedestrian trajectory prediction model.

[0098] refer to Figure 3 The pedestrian trajectory prediction model of the embodiment of the present application, in addition to the pedestrian trajectory feature extraction module, the virtual interaction feature extraction module and the pedestrian interaction feature extraction module, also includes a model loss calculation module.

[0099] During training, the historical pedestrian training trajectory data is input into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module. Figure 3In the above, the training trajectory data of the historical pedestrians are simultaneously used as the Q, K, and V inputs of the multi-head attention module in the first-level multi-head attention mechanism. The output of the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the fully connected layer in the second-level multi-head attention mechanism outputs the pedestrian trajectory features of the historical pedestrians.

[0100] The pedestrian trajectory features of the historical pedestrians and the implicit virtual interaction force data of the historical pedestrians are input into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module. For example, Figure 3 In the method, the pedestrian trajectory features of the historical pedestrians are simultaneously used as the Q and K inputs of the multi-head attention module in the first-level multi-head attention mechanism, and the implicit virtual interaction force data of the historical pedestrians are used as the V input of the multi-head attention module in the first-level multi-head attention mechanism. The output of the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the fully connected layer in the second-level multi-head attention mechanism outputs the virtual interaction features of the historical pedestrians.

[0101] The virtual interaction features of the historical pedestrian and the posture data of the pedestrians around the historical pedestrian are input into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module. Figure 3 In the method, the virtual interaction features of the historical pedestrians are simultaneously used as the Q and K inputs of the multi-head attention module in the first-level multi-head attention mechanism, and the posture data of the pedestrians around the historical pedestrians are used as the V input of the multi-head attention module in the first-level multi-head attention mechanism. The output of the regularization layer in the first-level multi-head attention mechanism is simultaneously used as the Q, K, and V inputs of the multi-head attention module in the second-level multi-head attention mechanism, and the adjacency matrix output by the fully connected layer in the second-level multi-head attention mechanism is used as the input of the linear layer. The linear layer converts the adjacency matrix into a matrix size corresponding to the predicted trajectory data to obtain the predicted trajectory data of the historical pedestrian. The predicted trajectory data of the historical pedestrian includes the predicted position data of the historical pedestrian at multiple data sampling moments within the third time period.

[0102] The predicted trajectory data and the real trajectory data of the historical pedestrians are input into the model loss calculation module, and the training loss is calculated by the model loss calculation module. For example, Figure 3In the method, a first average distance deviation (average trajectory deviation) between the position data of the corresponding data sampling moments in the real trajectory data and the predicted trajectory data of the historical pedestrian is calculated; a second average distance deviation (feature point trajectory deviation) between the position data of the corresponding feature points in the real trajectory data and the predicted trajectory data of the historical pedestrian is calculated, wherein the feature points are predetermined data sampling moments, for example, when the weights of certain data sampling moments are set larger during model training, these data sampling moments are feature points; a third distance deviation (end point trajectory deviation) between the end points of the real trajectory data and the predicted trajectory data of the historical pedestrian is calculated; and then, the training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation, for example, the above three deviations are added to obtain the training loss.

[0103] Then, according to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met, thereby obtaining a trained pedestrian trajectory prediction model.

[0104] like Figure 5 As shown, in an embodiment of the present application, the pedestrian trajectory prediction model also includes a prediction trajectory testing module. The prediction trajectory testing module is used to calculate the first average distance deviation and the third distance deviation based on the prediction trajectory data generated by the test set, and generate evaluation data of the prediction trajectory data according to the first average distance deviation and the third distance deviation. For example, the first average distance deviation and the third distance deviation are added, and the smaller the result, the better the authenticity of the prediction trajectory data. Specifically, the test set includes multiple test data, and the test data is similar to the training data of the training set. The predicted trajectory data is obtained by inputting the test data into the pedestrian trajectory prediction model, and the first average distance deviation and the third distance deviation are calculated based on the predicted trajectory data and the corresponding true trajectory data. The above process is similar to the training process and will not be repeated here.

[0105] In order to better understand the above method embodiment, the above method is further explained below through more specific examples.

[0106] In this example, historical trajectory data of pedestrians on the road is obtained in advance.

[0107] The historical trajectory of pedestrians on the road is the input of the pedestrian trajectory prediction method, which mainly includes the historical position coordinates (x, y), speed (v x , v y) information, and the position coordinates and speed information of pedestrians around it. The trajectory can be obtained through perception devices such as roadside perception devices, or it can be obtained from existing pedestrian trajectory data sets. When the pedestrian trajectory is obtained and applied through the roadside perception device, the roadside perception device first obtains the pedestrian's point cloud, image and other perception information through its perception device (such as lidar, camera, etc.), and processes the perception information through the computing unit to obtain the specific position and speed information of the pedestrian, and then sends the information to the intelligent connected vehicle through the vehicle-road-cloud architecture. The pedestrian trajectory prediction method of this example deployed on the intelligent connected vehicle takes the obtained information as input, and outputs the trajectory prediction result through this method.

[0108] In this example, pedestrian trajectory data is preprocessed. The pedestrian trajectory data preprocessing mainly refers to vectorizing the collected pedestrian data or data sets to meet the requirements of the pedestrian trajectory prediction model. Figure 6 Shown is a schematic diagram of pedestrian trajectory preprocessing.

[0109] like Figure 6 As shown in the figure, pedestrian trajectory data preprocessing includes two cases. When training the pedestrian trajectory prediction model, the pedestrian trajectory dataset needs to be processed into training data; when applying the pedestrian trajectory prediction model for trajectory prediction, the pedestrian trajectory data needs to be processed according to the application data structure, that is, the actual trajectory of the pedestrian is not needed.

[0110] Specifically, the processing of pedestrian historical trajectories (or pedestrian real trajectories) requires constructing the pedestrian's speed and position into vectors:

[0111] P t =[x t y t v xt v yt ⑴

[0112] Where P t represents the pedestrian pose vector at the data sampling time t. t is the x-direction position of the pedestrian at the data sampling time t, and y t is the y-direction position of the pedestrian at the data sampling time t, v xt is the x-direction speed of the pedestrian at the data sampling time t, v yt is the y-direction speed of the pedestrian at the data sampling time t. In this example, the pedestrian trajectory prediction model uses the pedestrian's posture vectors at the past 8 moments and combines other features to predict the pedestrian's position at the next 8 moments. So assuming that the earliest moment of the historical trajectory is the 0th moment, the vector matrix P of the pedestrian's historical trajectory can be constructed h :

[0113] P h =[P0P1…P6P7] T⑵

[0114] At the same time, under this assumption, the 8th moment is the earliest moment of the predicted trajectory. Therefore, when constructing training data, the pedestrian's real trajectory vector matrix can also be constructed:

[0115] P f =[P8P9…P 14 P 15 ] T ⑶

[0116] The pose data of surrounding pedestrians needs to construct a vector matrix based on the pose vectors of surrounding pedestrians, where the pose vector A of surrounding pedestrians is n It is constructed based on the speed and position of surrounding pedestrians:

[0117] A n =[x n y n v xn v yn ⑷

[0118] In the formula, A n represents the pose vector of the surrounding pedestrians at the current moment (the 7th moment according to the previous assumption). n is the x-direction position coordinate of the nth pedestrian around, y n is the y-direction position coordinate of the nth pedestrian around, v xn is the x-direction speed of the nth pedestrian, v yn is the y-direction speed of the nth surrounding pedestrian. In this example, the pedestrian trajectory prediction model requires the posture information of the eight surrounding pedestrians closest to the predicted pedestrian to construct the vector matrix A of the surrounding pedestrian posture data t When the number of pedestrians around is less than 8, use 0 vector to supplement:

[0119] A t =[A0A1…A6A7] T ⑸

[0120] Implicit virtual interaction force requires extracting the virtual interaction force calculation elements of surrounding pedestrians and constructing them into vectors:

[0121] V n =[x n 2 y n 2 v xn 2 v yn 2 ⑹

[0122] Considering that the virtual interaction force is mainly calculated using the square of relative speed and relative distance, this example also mainly uses these two factors when constructing the vector. Considering the convenience of calculation, the square of distance is used instead of distance for calculation. At the same time, considering the consistency of model input, the implicit virtual interaction force vector and matrix construction method is similar to the construction method of surrounding pedestrian posture data. In formula 6, V n represents the implicit virtual interaction force vector at the current moment (the 7th moment according to the previous assumption). n is the relative position of the nth pedestrian in the x direction, y n is the relative position of the nth pedestrian in the y direction, v xn is the relative speed of the nth pedestrian in the x direction, v yn is the relative speed of the nth pedestrian in the y direction. The model needs the implicit virtual interaction force vectors of the 8 pedestrians closest to the predicted pedestrian to construct the implicit virtual interaction force vector matrix of the surrounding pedestrians. When the number of surrounding pedestrians is less than 8, the 0 vector is used to supplement:

[0123] V t =[V0V1…V6V7] T ⑺

[0124] Thus, we can obtain the four input matrices required for the training of the pedestrian trajectory prediction model and the three input matrices required for application.

[0125] The pedestrian trajectory prediction model in this example refers to building a deep learning model and training the model using preprocessed training data. After training, the predicted pedestrian trajectory can be output by inputting application data. The structure of the pedestrian trajectory prediction model can be referred to Figure 5 , mainly includes 5 different modules. Among them, when training the model, the pedestrian trajectory feature extraction module, virtual interaction feature extraction module, pedestrian interaction feature extraction module and model loss calculation module are used for training; when testing, the pedestrian trajectory feature extraction module, virtual interaction feature extraction module, pedestrian interaction feature extraction module and predicted trajectory test module are used; when applying, the pedestrian trajectory feature extraction module, virtual interaction feature extraction module and pedestrian interaction feature extraction module are used to obtain the predicted pedestrian trajectory of the next 8 moments.

[0126] In this example, the pedestrian trajectory feature extraction module mainly uses deep learning modules such as the multi-head attention mechanism to extract the features of the pedestrian's historical trajectory. Figure 5In the figure, Multi-head Attention represents the multi-head attention mechanism module. In this example, the number of heads in this module is 4; Dropout is a regularization layer used to reduce the overfitting phenomenon of the neural network. In this model, the dropout rate is 0.25; Full Connected Layer represents the fully connected layer, which includes two linear layers and two regularization layers.

[0127] Pedestrian trajectory feature extraction mainly extracts the pedestrian's historical trajectory features to predict the pedestrian's future trajectory.

[0128] In this example, the virtual interaction feature extraction module is similar to the pedestrian trajectory feature extraction module. It mainly uses deep learning modules such as the multi-head attention mechanism to extract the virtual force field interaction features between pedestrians and surrounding pedestrians. Unlike the pedestrian trajectory feature extraction module, the first multi-head attention mechanism input V of the virtual interaction feature extraction is the implicit virtual interaction force matrix input, which is used to extract virtual interaction features.

[0129] Virtual interaction feature extraction mainly extracts the virtual force field interaction features between pedestrians and surrounding pedestrians to predict the future trajectory of pedestrians.

[0130] In this example, the pedestrian interaction feature extraction module is similar to the virtual interaction feature extraction module, which uses direct information about the surrounding pedestrians' positions to extract pedestrian interaction features. Its main structure is similar to that of the virtual interaction feature module. At the end of the model, a Linear Layer is added to represent the linear layer to obtain the final pedestrian prediction trajectory. The output of this module is the pedestrian trajectory prediction trajectory. When applied, only the first three modules need to be applied to obtain the pedestrian prediction trajectory.

[0131] In this example, the model loss calculation module mainly calculates the training loss of the model and serves the stochastic gradient descent training method. The model loss mainly includes three modules, including average trajectory deviation, feature point trajectory deviation and end point trajectory deviation. Among them, the average trajectory deviation is the average distance deviation between the predicted trajectory and each point of the true trajectory:

[0132]

[0133] Where AED (Average Error Distance) represents the average distance deviation. i is the x-direction distance of the predicted trajectory at the i-th moment, x ti is the x-direction distance of the real trajectory at the i-th moment; y i is the y-direction distance of the predicted trajectory at the i-th moment, y ti is the y-direction distance of the true trajectory at the i-th moment.

[0134] The feature point trajectory deviation is the average distance deviation between the feature point on the trajectory and the corresponding point on the true trajectory:

[0135]

[0136] In the formula, KED (key point error distance) represents the feature point distance deviation. i is the x-direction distance of the predicted trajectory at the i-th moment, x ti is the x-direction distance of the real trajectory at the i-th moment; y i is the y-direction distance of the predicted trajectory at the i-th moment, y ti is the y-direction distance of the real trajectory at the i-th moment. The feature points used in this model are the 2nd, 4th, 6th, and 8th points.

[0137] End point trajectory deviation The distance deviation between the end point on the trajectory and the end point of the actual trajectory:

[0138]

[0139] Where, FED (final error distance) represents the final distance deviation. Where x7 is the x-direction distance of the predicted trajectory at the last moment, x t7 is the x-direction distance of the actual trajectory at the last moment; y7 is the y-direction distance of the predicted trajectory at the last moment, t7 is the y-direction distance of the actual trajectory at the last moment.

[0140] In this example, the predicted trajectory test module mainly uses the average distance deviation and end point distance deviation modules in the model loss calculation to test the trajectory prediction results. The smaller the average distance deviation and end point distance deviation, the more realistic the predicted trajectory.

[0141] In this example, the pedestrian trajectory prediction result is the output of the pedestrian trajectory prediction model, which includes the location information of the pedestrian to be predicted at 8 moments in the future. The collection of location information constitutes the predicted trajectory of the pedestrian.

[0142] The inventors also used the ETH pedestrian dataset to test the feasibility and accuracy of the method of the embodiment of the present application. During implementation, training was performed according to the above method to obtain test and application results. Based on the relevant test results, the pedestrian trajectory prediction model of the embodiment of the present application can better predict the pedestrian trajectory, and for relatively similar trajectories, different features in the trajectory can be extracted to obtain accurate prediction results. In addition, the predicted trajectory test module can also obtain trajectory test results. For example, one prediction test result is: the average deviation of the predicted trajectory is 0.5739 meters, and the end point deviation is 0.7976 meters, which shows that the trajectory prediction result of the embodiment of the present application is good and can meet relevant needs.

[0143] In summary, it can be seen that the pedestrian trajectory prediction method of the embodiment of the present application can predict the pedestrian trajectory more accurately and achieve the purpose of pedestrian trajectory prediction; in addition, the embodiment of the present application has strong interpretability. Compared with the virtual interaction force calculation method, the embodiment of the present application requires less calculation amount and has a higher calculation effect.

[0144] Please refer to Figure 7 , a structure of a pedestrian trajectory prediction device provided in an embodiment of the present application includes:

[0145] A first acquisition module 71 is used to acquire the target pedestrian's data to be predicted, wherein the target pedestrian's data to be predicted includes the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, wherein the first sampling moment is the last data sampling moment in the first time period;

[0146] The prediction module 72 is used to input the to-be-predicted data of the target pedestrian into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0147] Optionally, the historical trajectory data of the target pedestrian is a vector matrix composed of at least one pedestrian posture vector of the target pedestrian, and the pedestrian posture vector is a vector composed of position data and speed data of the pedestrian at the same data sampling moment;

[0148] The surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians that are closest to each other;

[0149] The implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of nearby pedestrians are vectors composed based on the relative distance and relative speed between the target pedestrian and the nearby pedestrians.

[0150] Optionally, the relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane.

[0151] Optionally, the implicit virtual interaction force vector of the surrounding pedestrians is a vector formed by the square of the relative distance and the square of the relative speed between the target pedestrian and the surrounding pedestrians.

[0152] Optionally, the above device further comprises:

[0153] A second acquisition module is used to acquire a training set, wherein the training set includes multiple training data, each of which includes: training trajectory data and real trajectory data of the same historical pedestrian, and surrounding pedestrian posture data and implicit virtual interaction force data of the historical pedestrian at a second sampling moment, wherein the training trajectory data and the real trajectory data are the historical trajectory data of the historical pedestrian in a second time period and a third time period, respectively, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period;

[0154] The training module is used to train the pedestrian trajectory prediction model based on the training set to obtain a trained pedestrian trajectory prediction model.

[0155] Optionally, the pedestrian trajectory prediction model includes a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module, a pedestrian interaction feature extraction module and a model loss calculation module; the training module is further used to:

[0156] Inputting the historical pedestrian training trajectory data into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module;

[0157] Inputting the pedestrian trajectory features and the implicit virtual interaction force data of the historical pedestrians into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module;

[0158] Inputting the virtual interaction features and the posture data of pedestrians around the historical pedestrian into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module;

[0159] Inputting the predicted trajectory data and the real trajectory data of the historical pedestrian into the model loss calculation module, and calculating the training loss through the model loss calculation module;

[0160] According to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met to obtain a trained pedestrian trajectory prediction model.

[0161] Optionally, the training module is further used to:

[0162] Calculating a first average distance deviation between the position data at corresponding data sampling moments in the actual trajectory data and the predicted trajectory data of the historical pedestrian;

[0163] Calculating a second average distance deviation between position data of a corresponding feature point in the real trajectory data and the predicted trajectory data of the historical pedestrian, the feature point being a predetermined data sampling moment;

[0164] Calculating a third distance deviation between the actual trajectory data and the end point of the predicted trajectory data of the historical pedestrian;

[0165] The training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation.

[0166] Optionally, the pedestrian trajectory prediction model further includes a prediction trajectory testing module;

[0167] The predicted trajectory testing module is used to calculate the first average distance deviation and the third distance deviation based on the predicted trajectory data generated by the test set, and generate evaluation data of the predicted trajectory data according to the first average distance deviation and the third distance deviation.

[0168] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above pedestrian trajectory prediction method, and the implementation methods in the above embodiments are all applicable to the embodiments of the device, and can also achieve the same technical effects. The above device provided in the embodiment of the present application can implement all the method steps implemented in the above method embodiment, and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment will not be specifically described here.

[0169] Please refer to Figure 8 , a schematic diagram of the structure of another pedestrian trajectory prediction device provided in an embodiment of the present application, the device includes: a processor 801, a transceiver 802, a memory 803, a user interface 804 and a bus interface.

[0170] In the embodiment of the present application, the device also includes: a program stored in the memory 803 and executable on the processor 801 .

[0171] The transceiver 802 is used to send and receive data under the control of the processor;

[0172] The processor 801 is configured to read the computer program in the memory and perform the following operations:

[0173] Acquire the target pedestrian's data to be predicted, the data to be predicted including the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, the first sampling moment being the last data sampling moment in the first time period;

[0174] The to-be-predicted data of the target pedestrian is input into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0175] It can be understood that in the embodiment of the present application, when the computer program is executed by the processor 801, each process of the above-mentioned pedestrian trajectory prediction method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0176] exist Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 801 and various circuits of memory represented by memory 803 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 802 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 804 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0177] The processor 801 is responsible for managing the bus architecture and general processing, and the memory 803 can store data used by the processor 801 when performing operations.

[0178] It should be noted that the device in this embodiment is a device corresponding to the above-mentioned pedestrian trajectory prediction method, and the implementation methods in the above-mentioned embodiments are all applicable to the embodiments of the device, and can also achieve the same technical effects. In the device, the transceiver 802 and the memory 803, as well as the transceiver 802 and the processor 801 can be connected through a bus interface communication, and the function of the processor 801 can also be implemented by the transceiver 802, and the function of the transceiver 802 can also be implemented by the processor 801. It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effects, and the parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

[0179] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored, and when the program is executed by a processor, the following steps are implemented:

[0180] Acquire the target pedestrian's data to be predicted, the data to be predicted including the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, the first sampling moment being the last data sampling moment in the first time period;

[0181] The to-be-predicted data of the target pedestrian is input into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

[0182] When the program is executed by the processor, it can implement all the implementation methods of the above-mentioned pedestrian trajectory prediction method and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0183] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0188] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A pedestrian trajectory prediction method, characterized in that: include: Acquire the target pedestrian's data to be predicted, the data to be predicted including the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, the first sampling moment being the last data sampling moment in the first time period; The to-be-predicted data of the target pedestrian is input into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

2. The method according to claim 1, characterized in that The historical trajectory data of the target pedestrian is a vector matrix composed of at least one pedestrian posture vector of the target pedestrian, and the pedestrian posture vector is a vector composed of position data and speed data of the pedestrian at the same data sampling moment; The surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians that are closest to each other; The implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of nearby pedestrians are vectors composed based on the relative distance and relative speed between the target pedestrian and the nearby pedestrians.

3. The method according to claim 2, characterized in that The relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane.

4. The method according to claim 3, characterized in that The implicit virtual interaction force vector of the surrounding pedestrians is a vector formed by the square of the relative distance and the square of the relative speed between the target pedestrian and the surrounding pedestrians.

5. The method according to claim 1, characterized in that Before obtaining the target pedestrian's to-be-predicted data, the method further includes: Acquire a training set, the training set including multiple training data, each training data including: training trajectory data and real trajectory data of the same historical pedestrian, posture data of pedestrians around the historical pedestrian at a second sampling moment, and implicit virtual interaction force data, wherein the training trajectory data and the real trajectory data are the historical trajectory data of the historical pedestrian in a second time period and a third time period, respectively, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period; The pedestrian trajectory prediction model is trained based on the training set to obtain a trained pedestrian trajectory prediction model.

6. The method according to claim 5, characterized in that The pedestrian trajectory prediction model includes a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module, a pedestrian interaction feature extraction module and a model loss calculation module; Training the pedestrian trajectory prediction model based on the training data set includes: Inputting the historical pedestrian training trajectory data into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module; Inputting the pedestrian trajectory features and the implicit virtual interaction force data of the historical pedestrians into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module; Inputting the virtual interaction features and the posture data of pedestrians around the historical pedestrian into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module; Inputting the predicted trajectory data and the real trajectory data of the historical pedestrian into the model loss calculation module, and calculating the training loss through the model loss calculation module; According to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met to obtain a trained pedestrian trajectory prediction model.

7. The method according to claim 6, characterized in that The training loss is calculated by the model loss calculation module, including: Calculating a first average distance deviation between the position data at corresponding data sampling moments in the actual trajectory data and the predicted trajectory data of the historical pedestrian; Calculating a second average distance deviation between position data of a corresponding feature point in the real trajectory data and the predicted trajectory data of the historical pedestrian, the feature point being a predetermined data sampling moment; Calculating a third distance deviation between the actual trajectory data and the end point of the predicted trajectory data of the historical pedestrian; The training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation.

8. The method according to claim 7, characterized in that The pedestrian trajectory prediction model also includes a prediction trajectory testing module; The predicted trajectory testing module is used to calculate the first average distance deviation and the third distance deviation based on the predicted trajectory data generated by the test set, and generate evaluation data of the predicted trajectory data according to the first average distance deviation and the third distance deviation.

9. A pedestrian trajectory prediction device, characterized in that: include: A first acquisition module is used to acquire the target pedestrian's to-be-predicted data, wherein the to-be-predicted data includes the target pedestrian's historical trajectory data in a first time period, the target pedestrian's surrounding pedestrian posture data at a first sampling moment, and implicit virtual interaction force data, wherein the first sampling moment is the last data sampling moment in the first time period; The prediction module is used to input the to-be-predicted data of the target pedestrian into a pre-trained pedestrian trajectory prediction model to obtain the predicted trajectory data of the target pedestrian after the first time period output by the pedestrian trajectory prediction model.

10. The device according to claim 9, characterized in that The historical trajectory data of the target pedestrian is a vector matrix composed of at least one pedestrian posture vector of the target pedestrian, and the pedestrian posture vector is a vector composed of position data and speed data of the pedestrian at the same data sampling moment; The surrounding pedestrian posture data is a vector matrix composed of pedestrian posture vectors of multiple surrounding pedestrians that are closest to each other; The implicit virtual interaction force data is a vector matrix composed of implicit virtual interaction force vectors of multiple nearby pedestrians, and the implicit virtual interaction force vectors of nearby pedestrians are vectors composed based on the relative distance and relative speed between the target pedestrian and the nearby pedestrians.

11. The device according to claim 10, characterized in that The relative distance includes a first relative distance in the x direction and a second relative distance in the y direction; the relative speed includes a first relative speed in the x direction and a second relative speed in the y direction; wherein the x direction and the y direction are two perpendicular directions on a predefined two-dimensional plane.

12. The device according to claim 11, characterized in that The implicit virtual interaction force vector of the surrounding pedestrians is a vector formed by the square of the relative distance and the square of the relative speed between the target pedestrian and the surrounding pedestrians.

13. The device according to claim 9, characterized in that Also includes: A second acquisition module is used to acquire a training set, wherein the training set includes multiple training data, each of which includes: training trajectory data and real trajectory data of the same historical pedestrian, and surrounding pedestrian posture data and implicit virtual interaction force data of the historical pedestrian at a second sampling moment, wherein the training trajectory data and the real trajectory data are the historical trajectory data of the historical pedestrian in a second time period and a third time period, respectively, the third time period is after the second time period and adjacent to the second time period, and the second sampling moment is the last data sampling moment of the second time period; The training module is used to train the pedestrian trajectory prediction model based on the training set to obtain a trained pedestrian trajectory prediction model.

14. The device according to claim 13, characterized in that The pedestrian trajectory prediction model includes a pedestrian trajectory feature extraction module, a virtual interaction feature extraction module, a pedestrian interaction feature extraction module and a model loss calculation module; the training module is also used for: Inputting the historical pedestrian training trajectory data into the pedestrian trajectory feature extraction module to obtain the pedestrian trajectory features extracted by the pedestrian trajectory feature extraction module; Inputting the pedestrian trajectory features and the implicit virtual interaction force data of the historical pedestrians into the virtual interaction feature extraction module to obtain the virtual interaction features extracted by the virtual interaction feature extraction module; Inputting the virtual interaction features and the posture data of pedestrians around the historical pedestrian into the pedestrian interaction feature extraction module to obtain the predicted trajectory data of the historical pedestrian generated by the pedestrian interaction feature extraction module; Inputting the predicted trajectory data and the real trajectory data of the historical pedestrian into the model loss calculation module, and calculating the training loss through the model loss calculation module; According to the training loss, the pedestrian trajectory prediction model is optimized until a preset training end condition is met to obtain a trained pedestrian trajectory prediction model.

15. The device according to claim 14, characterized in that The training module is also used for: Calculating a first average distance deviation between the position data at corresponding data sampling moments in the actual trajectory data and the predicted trajectory data of the historical pedestrian; Calculating a second average distance deviation between position data of a corresponding feature point in the real trajectory data and the predicted trajectory data of the historical pedestrian, the feature point being a predetermined data sampling moment; Calculating a third distance deviation between the actual trajectory data and the end point of the predicted trajectory data of the historical pedestrian; The training loss is determined according to the first average distance deviation, the second average distance deviation and the third distance deviation.

Images