A Multimodal Pedestrian Trajectory Prediction Method and System Based on a Predefined Tree

Abstract

The present invention discloses a multi-modal pedestrian trajectory prediction method and system based on a predefined tree. The method includes the following steps: obtaining an observation trajectory sequence of a pedestrian; constructing a coarse-grained trajectory tree based on the movement speed in the observation trajectory sequence of the pedestrian; optimizing each branch of the coarse-grained trajectory tree by using a pre-trained neural network based on the observation trajectory sequence and the coarse-grained trajectory tree to obtain an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result. In the present invention, based on the general movement pattern of pedestrians, a variety of possible future trajectories are represented as a trajectory tree, and each path from the root node to the leaf node is represented as a possible path, which has good interpretability and unique movement characteristics; stable and more accurate multi-modal future trajectories can be predicted.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, relates to the field of pedestrian trajectory prediction, and particularly relates to a multi-modal pedestrian trajectory prediction method and system based on a predefined tree. Background Art

[0002] Trajectory prediction technology is an important part of driverless technology and has a great impact on the path planning of driverless. Due to the multi-modal problem of future trajectories, that is, pedestrians may move towards multiple different future trajectories, the trajectory prediction task still faces challenges.

[0003] Currently, existing methods mostly use generative models for multi-modal trajectory prediction, and there are still the following problems:

[0004] (1) Existing methods use a latent variable in an implicit space to represent different future trajectories. However, the latent variable faces the problem of being unexplainable.

[0005] (2) Existing methods use a continuous distribution to represent the implicit space and sample multiple times in the implicit space to represent multi-modal future trajectories; affected by the sampling operation, this method will generate unstable future trajectories.

[0006] (3) Existing methods use generative models (exemplarily, such as generative adversarial networks) and face the problem of difficult training (exemplarily, such as model collapse). Summary of the Invention

[0007] The purpose of the present invention is to provide a multi-modal pedestrian trajectory prediction method and system based on a predefined tree to solve one or more of the above-mentioned technical problems. In the present invention, based on the general motion pattern of pedestrians, a variety of possible future trajectories are represented as a trajectory tree, and each path from the root node to the leaf node represents a possible path, which has good interpretability and unique motion characteristics; stable and more accurate multi-modal future trajectories can be predicted.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A multi-modal pedestrian trajectory prediction method based on a predefined tree provided by the present invention includes the following steps:

[0010] Obtain the observed trajectory sequence of the pedestrian;

[0011] Based on the movement speed in the observed trajectory sequence of the pedestrian, construct a coarse-grained trajectory tree; wherein, the branches of the coarse-grained trajectory tree are used to respectively represent different modes of the future trajectory of the pedestrian.

[0012] Based on the observed trajectory sequence and the coarse-grained trajectory tree, use a pre-trained neural network to optimize each branch of the coarse-grained trajectory tree to obtain an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result.

[0013] A further improvement of the method of the present invention lies in that the step of constructing the coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of the pedestrian specifically includes:

[0014] 1) For the observed trajectory sequence with length T 1 , obtain its movement speed V;

[0015] 2) Set the length of the trajectory sequence to be predicted as T 2 , the number of tree splitting directions is N, and the splitting length is K times the movement speed obtained in step 1), K ∈ Z, 0 < K ≤ T 2 ; set the splitting matrix as W(α), where α ∈ [0, 2π];

[0016] 3) According to the movement speed V obtained in step 1), the tree splitting length, splitting direction and splitting matrix W(α) set in step 2), set N splitting direction vectors of the tree, where the nth splitting direction vector is W(αn)VK, where n ∈ [0, N - 1];

[0017] 4) Using the last time point of the observed trajectory sequence as the root node, use the N splitting direction vectors set in step 3) to recursively split the tree to construct the coarse-grained trajectory tree.

[0018] A further improvement of the method of the present invention lies in that in step 1), the movement speed V is the average speed, maximum speed or the speed at the last time point of the observed trajectory sequence.

[0019] A further improvement of the method of the present invention lies in that, characterized in that the structure of the neural network includes:

[0020] The first multi-layer perceptron is used to input the observed trajectory sequence and output the observed features;

[0021] The second multi-layer perceptron is used to input the coarse-grained trajectory tree to be optimized and output the branch features;

[0022] The self-attention mechanism is used to input the observed features and output the interaction features;

[0023] The attention mechanism is used to input the branch features and the interaction features and output the attention scores of the branches of the coarse-grained trajectory tree to be optimized;

[0024] The third multi-layer perceptron is used to input the pre-obtained coarse-grained future real trajectory sequence and output the future features;

[0025] The fourth multi-layer perceptron is configured to input the branch feature corresponding to the branch with the highest branch score and output an optimized branch;

[0026] The fifth multi-layer perceptron is configured to input the mixed feature and output the future predicted trajectory corresponding to the observed trajectory sequence; wherein, the mixed feature is obtained by adding the future feature and the interaction feature.

[0027] A further improvement of the method of the present invention lies in that the obtaining step of the pre-trained neural network includes:

[0028] Obtain a training sample set; wherein, each sample includes: an observed trajectory sequence, a coarse-grained trajectory tree, a future true trajectory sequence, a coarse-grained future true trajectory sequence, and a coarse-grained trajectory label;

[0029] Use the training samples for training, including:

[0030] Use the first multi-layer perceptron to extract observed features on the observed trajectory sequence;

[0031] Use the second multi-layer perceptron to extract branch features on each branch of the coarse-grained trajectory tree;

[0032] Use the self-attention mechanism to extract interaction features on the observed features;

[0033] Calculate the attention score with the interaction feature on the branch feature of each coarse-grained trajectory tree to obtain the attention score of each branch;

[0034] Use the third multi-layer perceptron to extract future features on the coarse-grained future true trajectory sequence; add the future feature and the interaction feature to obtain the mixed feature;

[0035] Use the fourth multi-layer perceptron on the branch corresponding to the highest attention score to obtain an optimized branch;

[0036] Use the fifth multi-layer perceptron on the mixed feature to obtain the future predicted trajectory;

[0037] Adopt a loss function, and complete the training when the preset convergence condition is reached to obtain the pre-trained neural network;

[0038] The expression of the loss function is, where λ 1 , λ 2 , λ 3 is used to reconcile the entire loss function; is the cross-entropy loss function, which is used to calculate the cross-entropy loss between the attention score and the coarse-grained trajectory tree label to optimize the attention score; is the Huber loss function, which is used to calculate the mean square error between the optimized branch and the coarse-grained future true trajectory sequence, and optimize the coarse-grained trajectory tree; is the Huber loss function, which is used to calculate the mean square error between the predicted trajectory and the future true trajectory, and optimize the future predicted trajectory.

[0039] A further improvement of the method of the present invention lies in that the steps of constructing the coarse-grained trajectory tree labels specifically include:

[0040] (1) For each true future trajectory sequence of length T in the training set 2 , take a point every K time steps;

[0041] (2) Use the trajectory sequence composed of the points taken in step (1) as the coarse-grained future true trajectory sequence;

[0042] (3) Calculate the distance between each branch of the coarse-grained trajectory tree and the coarse-grained future true trajectory described in step (2), and encode the serial number of the branch with the closest distance as a 0 / 1 vector, which is used as the coarse-grained trajectory tree label.

[0043] A multi-modal pedestrian trajectory prediction system based on a predefined tree provided by the present invention includes:

[0044] An observed trajectory sequence acquisition module, which is used to acquire the observed trajectory sequence of a pedestrian;

[0045] A coarse-grained trajectory tree acquisition module, which is used to construct a coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of a pedestrian; wherein, the branches of the coarse-grained trajectory tree are used to represent different modalities of the future trajectory of the pedestrian respectively;

[0046] A prediction module, which is used to optimize each branch of the coarse-grained trajectory tree by using a pre-trained neural network based on the observed trajectory sequence and the coarse-grained trajectory tree, and obtain an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result.

[0047] A further improvement of the system of the present invention lies in that the steps of constructing a coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of a pedestrian specifically include:

[0048] 1) For an observed trajectory sequence of length T 1 , obtain its movement speed V;

[0049] 2) Set the length of the trajectory sequence to be predicted as T 2 , the number of tree splitting directions as N, the splitting length as K times the movement speed obtained in step 1), K ∈ Z, 0 < K ≤ T 2 ; set the splitting matrix as W(α), where α ∈ [0, 2π];

[0050] 3) Based on the motion speed V obtained in step 1) and the splitting length, splitting direction, and splitting matrix W(α) of the tree set in step 2), set N splitting direction vectors of the tree, where the nth splitting direction vector is W(αn)VK, where n ∈ [0, N - 1];

[0051] 4) Using the last time point of the observation trajectory sequence as the root node, perform tree splitting recursively using the N splitting direction vectors set in step 3) to construct a coarse-grained trajectory tree.

[0052] A further improvement of the system of the present invention is that the structure of the neural network includes:

[0053] A first multi-layer perceptron for inputting the observation trajectory sequence and outputting observation features;

[0054] A second multi-layer perceptron for inputting the to-be-optimized coarse-grained trajectory tree and outputting branch features;

[0055] A self-attention mechanism for inputting the observation features and outputting interaction features;

[0056] An attention mechanism for inputting the branch features and the interaction features and outputting the attention scores of the branches of the to-be-optimized coarse-grained trajectory tree;

[0057] A third multi-layer perceptron for inputting the pre-obtained coarse-grained future real trajectory sequence and outputting future features;

[0058] A fourth multi-layer perceptron for inputting the branch features corresponding to the branch with the highest branch score and outputting the optimized branch;

[0059] A fifth multi-layer perceptron for inputting the mixed features and outputting the future predicted trajectory corresponding to the observation trajectory sequence; wherein, the mixed features are obtained by adding the future features and the interaction features.

[0060] A further improvement of the system of the present invention is that the obtaining steps of the pre-trained neural network include:

[0061] Obtain a training sample set; wherein, each sample includes: an observation trajectory sequence, a coarse-grained trajectory tree, a future real trajectory sequence, a coarse-grained future real trajectory sequence, and a coarse-grained trajectory label;

[0062] Use the training samples for training, including:

[0063] Extract observation features using the first multi-layer perceptron on the observation trajectory sequence;

[0064] Extract branch features using the second multi-layer perceptron on each branch of the coarse-grained trajectory tree;

[0065] Use the self-attention mechanism on the observed features to extract interaction features;

[0066] Calculate the attention scores with the interaction features on the branch features of each coarse-grained trajectory tree to obtain the attention scores of each branch;

[0067] Use a third multi-layer perceptron to extract future features on the coarse-grained future true trajectory sequence; Add the future features and the interaction features to obtain a mixed feature;

[0068] Use a fourth multi-layer perceptron on the branch corresponding to the highest attention score to obtain an optimized branch;

[0069] Use a fifth multi-layer perceptron on the mixed feature to obtain a future predicted trajectory;

[0070] Adopt a loss function, complete the training when the preset convergence condition is reached, and obtain the pre-trained neural network;

[0071] The expression of the loss function is where λ 1 and λ 2 and λ 3 are used to reconcile the entire loss function; is the cross-entropy loss function, which is used to calculate the cross-entropy loss between the attention scores and the coarse-grained trajectory tree labels to optimize the attention scores; is the Huber loss function, which is used to calculate the mean square error between the optimized branch and the coarse-grained future true trajectory sequence to optimize the coarse-grained trajectory tree; is the Huber loss function, which is used to calculate the mean square error between the predicted trajectory and the future true trajectory to optimize the future predicted trajectory.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] In the multi-modal pedestrian trajectory prediction method based on a predefined tree specifically provided by the present invention, based on the general motion pattern of pedestrians, a variety of possible future trajectories are represented as a trajectory tree, and each path from the root node to the leaf node represents a possible path, which has good interpretability and unique motion characteristics, and can predict stable and more accurate multi-modal future trajectories.

[0074] Specifically, the present invention uses the path from the root node to the leaf node in the tree as the possible future trajectory, which can describe the motion state of the future trajectory (exemplarily, go straight first and then turn right), and has good interpretability compared with latent variables; since the tree is predefined, the motion characteristics of the path of the tree are not affected by the average mode of the data, which is more conducive to representing the multi-modal nature and avoiding the model from collapsing to the average mode of the data; by scoring and electing through different paths of the tree, stable prediction results can be generated compared with multiple samplings. Brief Description of the Drawings

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art; obviously, the drawings in the following description are some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0076] Figure 1 It is a schematic flowchart of a multi-modal pedestrian trajectory prediction method based on a predefined tree according to an embodiment of the present invention;

[0077] Figure 2 It is a schematic diagram of a coarse-grained trajectory tree in an embodiment of the present invention;

[0078] Figure 3 It is a schematic diagram of the visualization result of the method provided by the embodiment of the present invention under the ETH and UCY data sets. Detailed Embodiments

[0079] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0080] It should be noted that the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present invention are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0081] The present invention will be further described in detail below with reference to the accompanying drawings:

[0082] Please refer to Figure 1 , a multi-modal pedestrian trajectory prediction method based on a predefined tree according to an embodiment of the present invention, includes the following steps:

[0083] Step 1, construct a coarse-grained trajectory tree:

[0084] 1) Define that a pedestrian has N motion patterns at each time point. For example, going straight, turning at different angles, etc.

[0085] 2) Set the length T of the trajectory sequence to be predicted 2 , according to the N motion patterns defined in step 1), there are N possible motion patterns for a pedestrian at each time point. Then all possible future trajectories can be regarded as a full N-ary tree with a depth of T 2 and generate future possible trajectories.

[0086] 3) Since the number of paths of the tree increases exponentially with the increase of the depth of the tree, splitting the tree at each time point will increase the time and space complexity of the tree. Therefore, a coarse-grained trajectory tree is constructed. Referring to the previous assumption, a pedestrian will not change the motion trend greatly in a short time, and split multiple time points each time to reduce the time and space complexity. Assume that K time points are split each time, then the depth of the tree is

[0087] 4) Set the length of the observed trajectory sequence to be T 1 and obtain its motion speed V, which can be the average speed, maximum speed, speed at the last time point, etc. of the observed trajectory sequence.

[0088] 5) The number of splitting directions of the tree is N, and the splitting length is K times the motion speed obtained in step 1), where K ∈ Z, 0 < K ≤ T 2. Set the splitting matrix as W(α), where α ∈ [0, 2π].

[0089] 6) According to the motion speed V obtained in step 1), the splitting length, splitting direction, and splitting matrix W(α) of the tree set in step 2), set the nth splitting direction vector of the tree as W(αn)VK, where n ∈ N. 4) Using the root node at the last time point of the observed trajectory sequence, perform tree splitting recursively using the N splitting direction vectors set in step 3).

[0090] Step 2, construct the coarse-grained trajectory tree label:

[0091] 1) For each true future trajectory sequence of length T 2 in the training set, take a point every K time points.

[0092] 2) Use the trajectory sequence composed of the points taken in step 1) as the coarse-grained future true trajectory sequence.

[0093] 3) Calculate the distance between each branch of the coarse-grained trajectory tree and the coarse-grained future true trajectory described in step 2), and encode the serial number of the branch with the closest distance as a 0 / 1 vector as the coarse-grained trajectory tree label.

[0094] Step 3, feature extraction:

[0095] 1) Observation feature: Use a multi-layer perceptron to extract the observation trajectory features of each person in the scene;

[0096] 2) Interaction feature: Use the self-attention mechanism to obtain the interaction features between people;

[0097] 3) Element feature: Use a multi-layer perceptron to extract features for each element in the constructed structured coarse-grained trajectory tree.

[0098] Step 4, element scoring and election:

[0099] 1) Element scoring: Integrate the observation feature and the interaction feature to obtain the scene feature, and use the attention mechanism for the scene feature and the element feature to obtain the attention score vector, which is the score for each element;

[0100] 2) Coarse-grained future true trajectory: Perform high-order sampling on the future true trajectory, that is, corresponding to the elements of the coarse-grained trajectory tree, select a trajectory point every S time points to obtain the coarse-grained future true trajectory;

[0101] 3) Election target: Use the squared distance error to calculate the distance between each element and the coarse-grained future true trajectory, and record the element with the closest distance as the target element, which is the target of the election.

[0102] 4) Election Optimization: Encode the position of the target element as a one-hot vector, and use the cross-entropy loss function to optimize the attention score vector, thereby optimizing the election. The cross-entropy loss function is:

[0103]

[0104] where p(x i ) represents the one-hot vector, q(x i ) represents the predicted probability distribution, that is, the attention score, and n represents the vector dimension.

[0105] Step 5, Greedy Optimization of the Coarse-Grained Trajectory Tree: For the element with the highest attention score, use the Huber loss function to greedily approximate the true future trajectory of the coarse-grained; the Huber loss function is:

[0106]

[0107] where, represents the highest branch corresponding to the selected attention score, and y represents the true future trajectory of the coarse-grained.

[0108] Step 6, Pedestrian Multimodal Trajectory Prediction:

[0109] 1) After obtaining the optimized coarse-grained trajectory tree, define a re-optimization module to re-optimize the optimized closest element to approximate the true future trajectory;

[0110] 2) In the training stage, use the teacher-guided method, that is, use the true future trajectory of the coarse-grained to replace the optimized closest element, and use the Huber loss function to make it approximate the true future trajectory; finally, the network uses an end-to-end method that combines multiple loss functions to train the model; the Huber loss function is:

[0111]

[0112] where, represents the true future trajectory of the coarse-grained, and y represents the true future trajectory.

[0113] 3) In the test stage, select the elements corresponding to the top-k attention scores for re-optimization to obtain multiple predicted trajectories, that is, multimodal pedestrian prediction trajectories.

[0114] A method for multimodal pedestrian trajectory prediction based on a predefined tree according to an embodiment of the present invention includes the following steps:

[0115] Step 1: Construct a coarse-grained trajectory tree: 1) Define the general motion patterns of pedestrians as: going straight, turning left, and turning right; where standing still and turning around are regarded as special cases of going straight and turning left (right), respectively. 2) According to the general motion patterns of pedestrians, there are three alternative directions for pedestrians at each time point. Assuming that the future T time points are predicted, according to the splitting process of the tree, all possible future trajectories can be regarded as a full ternary tree with a depth of T, and 3 T future possible trajectories are generated. 3) Since the number of paths in the tree increases exponentially with the increase of the tree depth, splitting the tree at each time point will increase the time and space complexity of the tree. Therefore, a coarse-grained trajectory tree is constructed. Referring to the previous assumption, pedestrians will not change their motion trends greatly in a short period of time. Split multiple time points each time to reduce the time and space complexity. Assuming that S time points are split each time, the depth of the tree is 4) Regarding the path from the root node to the leaf node as an element, the constructed coarse-grained trajectory tree can construct a coarse-grained trajectory tree.

[0116] Step 2: Extract observation trajectory features, interaction features, and spatial element features: 1) Observation features: Use a multi-layer perceptron to extract the observation trajectory features of each person in the scene; 2) Interaction features: Use a self-attention mechanism to obtain the interaction features between people; 3) Element features: Use a multi-layer perceptron to extract features for each element in the constructed coarse-grained trajectory tree.

[0117] Step 3: Element scoring and election: 1) Element scoring: Fuse the observation features and interaction features to obtain the scene features, and use the attention mechanism for the scene features and element features to obtain the attention score vector, which is the score for each element; 2) Coarse-grained future real trajectory: Perform high-order sampling on the future real trajectory, that is, corresponding to the elements of the coarse-grained trajectory tree, select a trajectory point every S time points to obtain the coarse-grained future real trajectory; 3) Election target: Use the squared distance error to calculate the distance between each element and the coarse-grained future real trajectory, and record the element with the closest distance as the target element, which is the target of the election. 4) Election optimization: Encode the position of the target element as a one-hot vector, and use the cross-entropy loss function to optimize the attention score vector, so as to optimize the election.

[0118] Step 4: Greedily optimize the coarse-grained trajectory tree: For the element with the highest attention score, use the Huber loss function to greedily approximate the coarse-grained future real trajectory;

[0119] Step 5: Multi-modal pedestrian trajectory prediction: 1) After obtaining the optimized coarse-grained trajectory tree, define a re-optimization module to re-optimize the optimized closest element to approximate the future real trajectory. 2) In the training phase, use the teacher-guided method, that is, use the coarse-grained future real trajectory to replace the optimized closest element, and use the Huber loss function to make it approximate the future real trajectory; finally, the network uses an end-to-end method that combines multiple loss functions to train the model. 3) In the test phase, select the elements corresponding to the top-k attention scores for re-optimization to obtain multiple predicted trajectories, that is, multi-modal pedestrian prediction trajectories.

[0120] In the embodiment of the present invention, in step 1, the straight-ahead splitting direction of the tree is the offset vector v of the last S time instants of the observed trajectory -s , the left splitting direction is the product of the rotation matrix M(θ) and v -s , and the right splitting direction is the product of the rotation matrix M(-θ) and v -s .

[0121] In the embodiment of the present invention, in step 5, the multiple loss functions combined are:

[0122] where λ 1 , λ 2 , λ 3 are used to reconcile the entire loss function, is the cross-entropy loss function used to optimize the attention scores in 4) of step 3, is the Huber loss function used to optimize the closest element in 1) of step 4 so as to greedily optimize the coarse-grained trajectory tree in 4) of step 1. is the Huber loss function used to re-optimize the closest element in 1) of step 5.

[0123] In the embodiment of the present invention, when obtaining the observed trajectory sequence of each pedestrian in the video, the existing pedestrian tracking method is used to obtain the trajectory coordinates.

[0124] Please refer to Figure 2 , Figure 2 which is the construction process of the coarse-grained trajectory tree in the method of the present invention. For the convenience of drawing, the parameters in the figure are all reference parameters. For other forms of coarse-grained trajectory trees, such as different depths or different numbers of splitting time points, they can be drawn according to the same principle.

[0125] Please refer to Figure 3 , Figure 3It is the prediction result using the method of the embodiment of the present invention. It can be seen that the constructed coarse-grained trajectory tree can well cover the future action space of pedestrians, and the selected path can briefly explain the future action direction of the behavior. Finally, the predicted trajectory has a small error.

[0126] Please refer to Table 1. Table 1 shows the experimental results of the method of the embodiment of the present invention under the ETH and UCY datasets. ETH and HOTEL respectively include some sub-datasets; among them, ETH includes the ETH and HOTEL datasets, and UCY includes the UNIV, ZARA1, and ZARA2 datasets. Here, for the five data subsets, the leave-one-out method is used for experiments. The embodiment of the present invention observes the trajectory of 3.2 seconds of the experimental observed trajectory and predicts the next 4.8 seconds of the trajectory, with a rate of 0.4, that is, observes the trajectory of 8 time points and predicts the trajectory of 12 time points.

[0127] Table 1. Experimental results of the method under the ETH and UCY datasets

[0128]

[0129]

[0130] ADE / FDE is used as the evaluation index in the experiment. Select the top-20 sample trajectories from the constructed coarse-grained trajectory tree. ADE is the average displacement error, that is, the average error between the 20 sample trajectories and the real trajectory at each time point. FDE is the final displacement error, that is, the average error between the 20 sample trajectories and the real trajectory at the last time point. The splitting time length S = 4, and the rotation angle θ = π / 12. It can be seen from Table 1 that except for the ADE index on HOTEL, the method of the present invention reaches the optimal in both the ADE and FDE indexes; on average of the indexes, the method of the embodiment of the present invention also reaches the optimal.

[0131] In addition, without any optimization, just using the original coarse-grained trajectory tree, the coarse-grained trajectory tree in the method of the embodiment of the present invention can be close to or even exceed some methods based on deep neural networks. For example, Raw Tree (d = 0) represents that a 12-time-point straight-line walking trajectory extended according to the speed of the last time point of the observed trajectory can exceed Sophie (CVPR 2019), and Raw Tree (d = 1) represents that the splitting time length S = 12, generating three 12-time-point trajectories, which can exceed the method of the deep neural network PITF; as the depth d increases, for example, d = 3, the performance of the constructed coarse-grained tree can reach 0.39 / 0.75, which can already be close to or even exceed the recent deep neural network methods such as TPNMS (AAAI 2021), Social-STGCNN (CVPR 2020), etc.

[0132] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0133] In another embodiment of the present invention, a multi-modal pedestrian trajectory prediction system based on a predefined tree of the present invention embodiment includes:

[0134] An observed trajectory sequence acquisition module, configured to acquire an observed trajectory sequence of a pedestrian;

[0135] A coarse-grained trajectory tree acquisition module, configured to construct a coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of the pedestrian; wherein, the branches of the coarse-grained trajectory tree are used to respectively represent different modalities of the future trajectory of the pedestrian;

[0136] A prediction module, configured to optimize each branch of the coarse-grained trajectory tree based on the observed trajectory sequence and the coarse-grained trajectory tree by using a pre-trained neural network to obtain an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result.

[0137] In summary, the present invention discloses a multi-modal pedestrian trajectory prediction method based on a predefined tree, belonging to the field of computer vision. In the embodiments of the present invention, aiming at the multi-modal characteristics of pedestrian trajectories, general walking rules are defined, and various possible futures are mapped into a coarse-grained trajectory tree, and a different data structure (tree) from before is used to construct this coarse-grained trajectory tree. Due to the pre-definition of the tree, the trajectories composed of the paths from the root node to the leaf nodes can maintain independent motion characteristics, thus ensuring multi-modality. High-order sampling is performed on the future real trajectory to obtain a coarse-grained future real trajectory, and it is used to greedily optimize the coarse-grained trajectory tree. At the same time, scoring and election operations are performed on the elements in the coarse-grained trajectory tree to find the future trajectories with high possibilities, generating stable prediction results. Finally, a teacher-guided method is used to predict the final fine-grained multi-modal future trajectories.

[0138] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0140] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific embodiments of the present invention, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A multi-modal pedestrian trajectory prediction method based on a predefined tree, characterized in that, it includes the following steps: Obtain the observed trajectory sequence of the pedestrian; Construct a coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of the pedestrian; wherein, the branches of the coarse-grained trajectory tree are used to represent different modalities of the future trajectory of the pedestrian respectively; Based on the observed trajectory sequence and the coarse-grained trajectory tree, use a pre-trained neural network to optimize each branch of the coarse-grained trajectory tree to obtain an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result; Wherein, the step of constructing the coarse-grained trajectory tree based on the movement speed in the observed trajectory sequence of the pedestrian specifically includes: 1) For the observation trajectory sequence of length T 1 , obtain its motion speed V; 2) Set the length of the trajectory sequence to be predicted as T 2 , the number of splitting directions of the tree is N, and the splitting length is K times the motion speed obtained in step 1), where K ∈ Z and 0 < K ≤ T 2 ; Set the splitting matrix as W(α), where α ∈ [0, 2π]; 3) According to the movement speed V obtained in step 1), the splitting length, splitting direction and splitting matrix W(α) of the tree set in step 2), set N splitting direction vectors of the tree, where the nth splitting direction vector is W(αn)VK, where n ∈ [0, N - 1]; 4) Using the last time point of the observed trajectory sequence as the root node, use the N splitting direction vectors set in step 3) to recursively split the tree to construct the coarse-grained trajectory tree; The structure of the neural network includes: The first multi-layer perceptron is used to input the observed trajectory sequence and output the observed features; The second multi-layer perceptron is used to input the coarse-grained trajectory tree to be optimized and output the branch features; The self-attention mechanism is used to input the observed features and output the interaction features; The attention mechanism is used to input the branch features and the interaction features and output the attention scores of the branches of the coarse-grained trajectory tree to be optimized; The third multi-layer perceptron is used to input the pre-obtained coarse-grained future real trajectory sequence and output the future features; The fourth multi-layer perceptron is used to input the branch features corresponding to the branch with the highest branch score and output the optimized branch; The fifth multi-layer perceptron is used to input the mixed features and output the future predicted trajectory corresponding to the observed trajectory sequence; wherein, the mixed features are obtained by adding the future features and the interaction features; The obtaining step of the pre-trained neural network includes: Obtain a training sample set; wherein, each sample includes: an observed trajectory sequence, a coarse-grained trajectory tree, a future real trajectory sequence, a coarse-grained future real trajectory sequence and a coarse-grained trajectory label; Use the training samples for training, including: Extract the observed features using the first multi-layer perceptron on the observed trajectory sequence; Extract the branch features using the second multi-layer perceptron on each branch of the coarse-grained trajectory tree; Extract the interaction features using the self-attention mechanism on the observed features; Calculate the attention scores with the interaction features on the branch features of each coarse-grained trajectory tree to obtain the attention scores of each branch; Extract the future features using the third multi-layer perceptron on the coarse-grained future real trajectory sequence; Add the future features and the interaction features to obtain the mixed features; Use the fourth multi-layer perceptron on the branch corresponding to the highest attention score to obtain the optimized branch; Use the fifth multi-layer perceptron on the mixed features to obtain the future predicted trajectory; Using a loss function, when the preset convergence condition is reached, the training is completed, and the pre-trained neural network is obtained; The expression of the loss function is where λ 1 , λ 2 , λ 3 is used to reconcile the entire loss function; is the cross - entropy loss function, which is used to calculate the cross - entropy loss between the attention scores and the coarse - grained trajectory tree labels to optimize the attention scores; is the Huber loss function, which is used to calculate the mean squared error between the optimized branch and the coarse - grained future true trajectory sequence to optimize the coarse - grained trajectory tree; is the Huber loss function, which is used to calculate the mean squared error between the predicted trajectory and the future true trajectory to optimize the future predicted trajectory.

2. A multi-modal pedestrian trajectory prediction method based on a predefined tree according to claim 1, wherein, in step 1), the motion speed V is the average speed, the maximum speed or the speed at the last time point of the observed trajectory sequence.

3. A multi-modal pedestrian trajectory prediction method based on a predefined tree according to claim 1, wherein, the steps of constructing the coarse-grained trajectory tree label specifically include: (1) For each true future trajectory sequence of length T in the training set 2 take a point every K time steps; (2) The trajectory sequence composed of the points taken in step (1) is used as the coarse-grained future true trajectory sequence; (3) Calculate the distance between each branch of the coarse-grained trajectory tree and the coarse-grained future true trajectory described in step (2), and encode the serial number of the branch with the closest distance as a 0 / 1 vector, which is used as the coarse-grained trajectory tree label.

4. A multi-modal pedestrian trajectory prediction system based on a predefined tree, wherein, it includes: An observed trajectory sequence acquisition module for acquiring the observed trajectory sequence of a pedestrian; A coarse-grained trajectory tree acquisition module for constructing a coarse-grained trajectory tree based on the motion speed in the observed trajectory sequence of a pedestrian; wherein, the branches of the coarse-grained trajectory tree are used to represent different modalities of the future trajectory of the pedestrian respectively; A prediction module for optimizing each branch of the coarse-grained trajectory tree by using the pre-trained neural network based on the observed trajectory sequence and the coarse-grained trajectory tree, and obtaining an optimized trajectory tree; wherein, the branches of the optimized trajectory tree are used to represent the final multi-modal pedestrian trajectory prediction result; wherein, the steps of constructing the coarse-grained trajectory tree based on the motion speed in the observed trajectory sequence of a pedestrian specifically include: 1) For the observation trajectory sequence of length T 1 obtain its moving speed V; 2) Set the length of the trajectory sequence to be predicted as T 2 , the number of splitting directions of the tree is N, and the splitting length is K times the motion speed obtained in step 1), where K ∈ Z and 0 < K ≤ T 2 ; Set the splitting matrix as W(α), where α ∈ [0, 2π]; 3) According to the motion speed V obtained in step 1), the splitting length, splitting direction and splitting matrix W(α) of the tree set in step 2), set N splitting direction vectors of the tree, where the nth splitting direction vector is W(αn)VK, where n ∈ [0, N-1]; 4) Using the last time point of the observed trajectory sequence as the root node, use the N splitting direction vectors set in step 3) to perform tree splitting recursively to construct a coarse-grained trajectory tree; The structure of the neural network includes: The first multi-layer perceptron is used to input the observed trajectory sequence and output the observed features; The second multi-layer perceptron is used to input the coarse-grained trajectory tree to be optimized and output the branch features; The self-attention mechanism is used to input the observed features and output the interaction features; The attention mechanism is used to input the branch features and the interaction features and output the attention scores of the branches of the coarse-grained trajectory tree to be optimized; The third multi-layer perceptron is used to input the pre-obtained coarse-grained future true trajectory sequence and output the future features; The fourth multi-layer perceptron is used to input the branch features corresponding to the branch with the highest branch score and output the optimized branch; The fifth multi-layer perceptron is used to input the mixed features and output the future prediction trajectory corresponding to the observed trajectory sequence; wherein, the mixed features are obtained by adding the future features and the interaction features; The steps for obtaining the pre-trained neural network include: Obtain a training sample set; where each sample includes: an observed trajectory sequence, a coarse-grained trajectory tree, a future true trajectory sequence, a coarse-grained future true trajectory sequence, and a coarse-grained trajectory label; Use the training samples for training, including: Extract observed features using a first multi-layer perceptron on the observed trajectory sequence; Extract branch features using a second multi-layer perceptron on each branch of the coarse-grained trajectory tree; Extract interaction features using a self-attention mechanism on the observed features; Calculate the attention scores with the interaction features on the branch features of each coarse-grained trajectory tree to obtain the attention scores for each branch; Extract future features using a third multi-layer perceptron on the coarse-grained future true trajectory sequence; add the future features and the interaction features to obtain hybrid features; Use a fourth multi-layer perceptron on the branch corresponding to the highest attention score to obtain an optimized branch; Use a fifth multi-layer perceptron on the hybrid features to obtain a future predicted trajectory; Adopt a loss function, complete the training when the preset convergence condition is reached, and obtain the pre-trained neural network; The expression of the loss function is where λ 1 , λ 2 , λ 3 is used to reconcile the entire loss function; is the cross-entropy loss function, which is used to calculate the cross-entropy loss between the attention score and the coarse-grained trajectory tree label, and optimize the attention score; is the Huber loss function, which is used to calculate the mean squared error between the optimized branch and the coarse-grained future true trajectory sequence, and optimize the coarse-grained trajectory tree; is the Huber loss function, which is used to calculate the mean squared error between the predicted trajectory and the future true trajectory, and optimize the future predicted trajectory.

Images