Pedestrian trajectory prediction method, device and equipment based on progressive refinement strategy

Abstract

The invention relates to the field of pedestrian trajectory prediction, in particular to a pedestrian trajectory prediction method, device and equipment based on a progressive refinement strategy. The prediction method comprises the following steps: predicting a current historical trajectory by using a trained prediction model, and outputting a future prediction trajectory; wherein the trained prediction model is obtained by training a built model through a training data set, the training data set comprises multiple groups of trajectory data, and each group of trajectory data comprises a historical trajectory X and a future real trajectory Y; the model comprises a motion mode clustering block, a self-adaptive motion mode refining block, an embedded splicing block, a Transform encoder, a Transform decoder, a track refining block and a classifier. According to the method, the motion mode and the prediction track are gradually refined through the two refining blocks, the adaptability of the model to different scenes is enhanced, and the prediction of complex motion is improved.

Description

Technical Field

[0001] The present invention relates to the field of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method, device and equipment based on a progressive refinement strategy. Background Art

[0002] Pedestrian trajectory prediction is to predict the future movement of pedestrians based on their historical trajectories. Deep neural networks, with their powerful ability to express data features, have made significant breakthroughs in many visual recognition tasks such as image classification, object detection, and semantic segmentation. This success is attributed to the ability of deep neural networks to learn task patterns through a large number of model parameters, but this requires a large amount of labeled data for model training. In traditional visual recognition tasks, people will artificially adjust the category distribution of labeled data so that the number of samples in different categories remains balanced, that is, there is no obvious difference in the number of samples.

[0003] However, in practical applications, naturally collected data categories usually present a long-tail distribution, that is, the minority category has a large number of samples, while the majority category has only a small number of samples. This class imbalance problem brings difficulties to the training of deep neural networks. Models trained under long-tail data tend to be biased towards the majority category in the training data, because the feature space of the majority category is usually larger than the feature space of the minority category, and the classification decision boundary will shift towards the minority category to ensure better classification of the majority category. This phenomenon often causes deep models to perform poorly on minority categories with limited data. Directly using the empirical risk minimization method to train deep models cannot effectively deal with the long-tail class imbalance problem in practical applications.

[0004] The long-tail distribution of pedestrian trajectory data leads to an imbalance in the number of samples of different motion pattern categories. The imbalanced instances ultimately degrade the performance of the head samples. Due to the long-tail distribution of trajectory data, the decoder often considers the head and tail samples comprehensively when decoding features, resulting in the output prediction often being the average result, leading to large errors in the tail samples. When the application scenarios are different, there are differences between the general motion pattern and the future true trajectory. This may amplify the error, thereby affecting the prediction of the tail samples in different scenarios. Due to the different distribution of motion patterns in different scenarios, the average prediction is only applicable to the training scenario, hindering the generalization ability of the model. Therefore, the long-tail distribution problem always affects the generalization ability of the pedestrian trajectory prediction model. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a pedestrian trajectory prediction method based on a progressive refinement strategy, which gradually refines the motion pattern and predicted trajectory through two refinement blocks, enhances the adaptability of the model to different scenes and improves the prediction of complex motion.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a pedestrian trajectory prediction method based on a progressive refinement strategy, comprising: using a trained prediction model to predict the current historical trajectory and output a future predicted trajectory; wherein the trained prediction model is obtained by training a model built by a training data set, and the training data set includes multiple groups of trajectory data, each group of trajectory data includes a historical trajectory X and a future real trajectory Y; the model includes:

[0007] The motion pattern clustering block is used to perform K-means clustering on the normalized trajectory data to generate L future general motion patterns; and to take the average value of the historical trajectories X belonging to the same class to generate L historical general motion patterns;

[0008] The adaptive motion pattern refinement block is used to encode the difference between the historical trajectory X and each historical general motion pattern, obtain the differential feature, and concatenate the differential feature with each future general motion pattern and input it into the fully connected layer σ pattern , obtain the refined future general motion pattern;

[0009] The embedding and splicing block is used to embed and splice each historical trajectory X with L refined future general motion patterns;

[0010] Transformer encoder, which takes the output of the embedding splicing module as input to obtain the motion feature F;

[0011] Transformer decoder, which takes motion features F and neighbor embeddings of historical trajectories X as input and generates raw predictions

[0012] The trajectory refinement block is used to calculate the original prediction P orig and motion feature F as input, calculate the coordinate offset △ref of the predicted trajectory;

[0013] The refined prediction trajectory generation module is used to add the coordinate offset △ref to the original prediction P orig , generate L refined prediction trajectories;

[0014] The classifier is used to take the motion feature F as input and output the probabilities of L refined prediction trajectories, wherein the first K prediction trajectories with the largest probabilities are selected from the L prediction trajectories, which are the future prediction trajectories.

[0015] Furthermore, the difference between the historical trajectory X and the historical general movement pattern is encoded to obtain the differential features;

[0016] The specific formula is:

[0017] Δ h =σ s (XH)

[0018] where σ s represents a three-layer multilayer perceptron MLP with sigmoid activation, and H is the historical general motion pattern.

[0019] Further, the coordinate offset △ref of the predicted trajectory is calculated; the specific formula is:

[0020] Δ ref =σ proposal (P orig ,F)

[0021] where σ proposal Represents a three-layer MLP.

[0022] Furthermore, the classifier is a three-layer MLP with a tanh activation function, and the loss function of the classifier is:

[0023]

[0024] in, represents the probability of L predicted trajectories, Crossetropy represents the cross entropy loss, and Y represents the future true trajectory.

[0025] The present invention also relates to a pedestrian trajectory prediction device based on a progressive refinement strategy, which is used to predict the future trajectory of pedestrians by using a pedestrian trajectory prediction method based on a progressive refinement strategy.

[0026] The present invention also relates to a device comprising:

[0027] Memory for storing computer programs;

[0028] A processor is used to execute the computer program, and when the computer program is executed by the processor, the method for predicting pedestrian trajectory based on the progressive refinement strategy is used to predict the future trajectory of the pedestrian.

[0029] The present invention also relates to a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a pedestrian trajectory prediction method based on a progressive refinement strategy are implemented.

[0030] After adopting the above technical solution, the present invention proposes a progressive refinement strategy to adaptively refine the general motion pattern based on the historical trajectory X to provide more accurate guidance; the predicted trajectory is further adjusted using the motion feature F to generate a prediction that is more consistent with the future real trajectory. Therefore, the present invention adopts a progressive refinement strategy to gradually refine the motion pattern and predicted trajectory through two refinement blocks, which can enhance the adaptability of the model to different scenarios and improve the prediction of complex motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A framework diagram of a model of the present invention;

[0032] Figure 2 A detailed diagram of the adaptive mode refinement block of the present invention;

[0033] Figure 3 Detailed diagram of the trajectory refinement block of the present invention. DETAILED DESCRIPTION

[0034] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0035] like Figure 1 , Figure 2 and Figure 3 As shown, a pedestrian trajectory prediction method based on a progressive refinement strategy includes: using a trained prediction model to predict the current historical trajectory and output a future predicted trajectory; wherein the trained prediction model is obtained by training a model built by a training data set, and the training data set includes multiple groups of trajectory data, each group of trajectory data includes a historical trajectory X and a future real trajectory Y; each group of trajectory data has m consecutive frames, the historical trajectory X is the first m1 frame, and the future real trajectory Y is the last m2 frame, m = m1 + m2. In this embodiment, each group of trajectory data has 20 consecutive frames, including the first 8 frames of the historical trajectory X and the last 12 frames of the future real trajectory Y.

[0036] The model includes a motion feature encoding module and a probabilistic trajectory prediction module.

[0037] The motion feature encoding module includes a motion pattern clustering block, an adaptive motion pattern refinement block, an embedding splicing block, and a Transformer encoder. The probabilistic trajectory prediction module includes a Transformer decoder, a trajectory refinement block, a refined prediction trajectory generation module, and a classifier.

[0038] like Figure 1 As shown in FIG. 1 , the motion pattern clustering block is used to perform K-means clustering on the normalized trajectory data to generate L future general motion patterns, and to take the average value of the historical trajectories X belonging to the same category to generate L historical general motion patterns; the specific working process of the motion pattern clustering block is as follows:

[0039] First, multiple trajectory data are normalized to a fixed area through translation and rotation;

[0040] Then, k-means clustering is applied to the future true trajectory Y to generate the future general motion pattern set

[0041] where C = {c1, c2, ..., cL},c i represents the ith future general motion pattern, and L represents the category number. The future general motion pattern set C essentially exhibits a long-tail distribution;

[0042] Then take the average value of the historical trajectories X belonging to the same category to generate the historical general motion pattern set where H = {h1,h2,...,h L}, and h1 represents the i-th historical general motion pattern, the complete general motion pattern set M p It is generated by connecting C and H.

[0043] like Figure 2 As shown in the figure, the working process of the adaptive motion mode refinement block is:

[0044] First, encode the difference between the historical trajectory X and the historical general motion pattern set to obtain the differential feature Δ h ;

[0045] Encode the difference between H and X as follows:

[0046] Δ h =σ s (XH)

[0047] where σ s Represents a three-layer multilayer perceptron (MLP) with sigmoid activation.

[0048] Then the differential feature Δ h It is concatenated with the future general motion pattern set C to obtain the cascade feature. In practical applications, the real future trajectory of pedestrians is unknown. We can only adjust and refine the future general motion pattern according to the difference between the observed trajectory and the historical general motion pattern;

[0049] Finally, the cascaded features are input into the fully connected layer σ pattern , obtain the refined future general motion pattern set C ref , to alleviate the impact of motion pattern differences in different scenes on motion feature extraction, which can more accurately reflect the future motion of pedestrians.

[0050] Cascade feature input to fully connected layer σ pattern , as shown below:

[0051] C ref =σ pattern (Δ h ||C)

[0052] Here, || represents a concatenation operation.

[0053] like Figure 1As shown, the embedding block is used to embed and splice each historical trajectory X with L refined future general motion patterns respectively; this is to provide rough guidance information for the prediction model, and copy each historical trajectory X into a set of refined future general motion patterns C ref The same number of patterns (L) is obtained, and then concatenated with each motion pattern (the size is bs*L×20×2, 2 represents 2 dimensions), and finally linearly embedded into the Transformer encoder to obtain the motion feature F (the size is bs*L*feature dimension). The historical trajectory X is respectively combined with the refined future general motion pattern set C ref The various refined future general motion patterns in are embedded and spliced; specifically including:

[0054] Calculate the linear embedding E of the historical trajectory X respectively o and the linear embedding of the refined future general motion model E c , for the linear embedding E o and E c To splice; the formula is,

[0055]

[0056] E=(E o ||E c )

[0057] in is a linear transformation, | represents a separator; || represents concatenation; W o and W c They represent learnable parameters respectively; V represents the relative displacement between two adjacent frames in the historical trajectory X; V ref It represents the relative displacement between two adjacent frames in the refined future general motion pattern. E represents the pedestrian motion pattern.

[0058] like Figure 1 As shown, the Transformer encoder is used to take the output E of the embedding splicing module as input to obtain the motion feature F.

[0059] like Figure 1 As shown, the Transformer decoder is used to take the motion features F and the neighbor embeddings of the historical trajectory X as input to generate the original prediction Here, the historical trajectories whose Euclidean distance to the historical trajectory X of the target pedestrian is less than 2 are defined as adjacent trajectories, that is, neighbor embedding.

[0060] like Figure 1 and Figure 3 As shown, the trajectory refinement block is used to calculate the original prediction P orig And motion feature F are used as input to calculate the coordinate offset △ref of the predicted trajectory; the calculation is as follows:

[0061] Δ ref =σ proposal (P orig ,F)

[0062] where σ proposal Represents a three-layer MLP.

[0063] The fine prediction trajectory generation module is used to convert the difference Δ ref Add to the original prediction P orig In the above example, L refined prediction trajectories are generated. As shown below:

[0064]

[0065] like Figure 1 As shown, the classifier It is used to take motion feature F as input and output the probability of L refined predicted trajectories to help the agent select from multi-modal trajectories. The probability is calculated as follows:

[0066]

[0067] in is a three-layer MLP with tanh activation function, represents the probability of L predicted trajectories. The nearest neighbor assumption is adopted, that is, the predicted trajectory that is most similar to the future true trajectory has the highest probability. Therefore, c i The soft probability p of is obtained by normalizing the distance between the future general motion pattern set C and the future true trajectory Y, as follows:

[0068]

[0069] The present invention uses the cross entropy loss function to train as follows:

[0070]

[0071] In the inference phase, the top K predicted trajectories with the highest probability are selected to achieve accurate multimodal prediction.

[0072] This embodiment uses a motion pattern clustering block to obtain a general future motion pattern as a guide, so that the tail motion pattern can be better identified. However, there are inherent differences between the general motion pattern and the future real trajectory, resulting in prediction errors. These errors are further amplified when the application scenario changes. In this module, the present invention proposes an adaptive pattern refinement block, which can adaptively refine the future general motion pattern based on the observed trajectory. It provides accurate guidance to better predict the future trajectory of the tail sample in different scenarios.

[0073] The probabilistic trajectory prediction module in this embodiment takes F as input and outputs refined multimodal trajectory predictions and their probabilities. In this module, a trajectory refinement block is proposed, which forms a progressive refinement with the adaptive mode refinement block, which uses the motion features F to adjust the original predictions to make them closer to the real situation. It alleviates the impact of the long-tail distribution on the decoder and further improves the generalization ability of the model.

[0074] It should be noted that during the training phase, the prediction model:

[0075] (1) Input historical trajectory X and refined future motion pattern set C ref ; X is the future motion mode set C after dimension expansion and refinement ref Splicing;

[0076] (2) The concatenation is linearly embedded and sent to the transformer encoder to obtain the motion feature F

[0077] (3) F is fed into the classifier for training to obtain L probabilities.

[0078] (5) On the other hand, F is sent to the transformer decoder to generate the predicted trajectory and trajectory refinement block (bs*L*12*2), which is trained to generate L refined predicted trajectories;

[0079] After training, the prediction model is in the prediction (validation) stage:

[0080] (1) Input the historical trajectory X and the refined future motion pattern set C into the trained transformer encoder ref , directly use the trained transformer encoder to encode the input features and obtain the motion feature F;

[0081] (2) The motion feature F is fed into the trained classifier to obtain L probabilities.

[0082] (3) The motion feature F is fed into the trained transformer decoder and trajectory refinement block. The Transformer decoder receives F and the neighbor embedding E. nei To generate the original prediction P orig At the same time, the trajectory refinement block is based on P orig With motion feature F as input, the output coordinate difference Δref. Δref and P orig By adding the elements together, L refined prediction trajectories are generated, and the top K prediction trajectories with the highest probability are selected from the L refined prediction trajectories as the final prediction.

[0083] In one embodiment, a pedestrian trajectory prediction device based on a progressive refinement strategy is used to predict the future trajectory of a pedestrian using the pedestrian trajectory prediction method based on a progressive refinement strategy in the above embodiment.

[0084] In one embodiment, a device includes:

[0085] Memory for storing computer programs;

[0086] A processor is used to execute the computer program, and when the computer program is executed by the processor, the method for predicting pedestrian trajectory based on the progressive refinement strategy in the above embodiment is used to predict the future trajectory of the pedestrian.

[0087] In one embodiment, a readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the pedestrian trajectory prediction method based on the progressive refinement strategy in the above embodiment are implemented.

[0088] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A pedestrian trajectory prediction method based on progressive refinement strategy, It is characterized in that It includes: using the trained prediction model to predict the current historical trajectory and output the future predicted trajectory; wherein the trained prediction model is obtained by training the model built by the training data set, and the training data set includes multiple groups of trajectory data, each group of trajectory data includes the historical trajectory X and the future real trajectory Y; the model includes: The motion pattern clustering block is used to perform K-means clustering on the normalized trajectory data to generate L future general motion patterns; and to take the average value of the historical trajectories X belonging to the same class to generate L historical general motion patterns; The adaptive motion pattern refinement block is used to encode the difference between the historical trajectory X and each historical general motion pattern, obtain the differential feature, and concatenate the differential feature with each future general motion pattern and input it into the fully connected layer σ pattern , obtain the refined future general motion pattern; The embedding and splicing block is used to embed and splice each historical trajectory X with L refined future general motion patterns; Transformer encoder, which takes the output of the embedding splicing module as input to obtain the motion feature F; Transformer decoder, which takes motion features F and neighbor embeddings of historical trajectories X as input and generates raw predictions The trajectory refinement block is used to calculate the original prediction P orig and motion feature F as input, calculate the coordinate offset △ref of the predicted trajectory; The refined prediction trajectory generation module is used to add the coordinate offset △ref to the original prediction P orig , generate L refined prediction trajectories; The classifier is used to take the motion feature F as input and output the probabilities of L refined prediction trajectories, wherein the first K prediction trajectories with the largest probabilities are selected from the L prediction trajectories, which are the future prediction trajectories.

2. The pedestrian trajectory prediction method based on the progressive refinement strategy according to claim 1 is characterized in that: Encode the difference between the historical trajectory X and the historical general movement pattern to obtain the differential feature; The specific formula is: D h =s s (XH) where σ s represents a three-layer multilayer perceptron MLP with sigmoid activation, and H is the historical general motion pattern.

3. The pedestrian trajectory prediction method based on the progressive refinement strategy according to claim 1 is characterized in that: Calculate the coordinate offset △ref of the predicted trajectory; the specific formula is: D ref =s proposal (P orig ,F) where σ proposal Represents a three-layer MLP.

4. The pedestrian trajectory prediction method based on the progressive refinement strategy according to claim 1, characterized in that: The classifier is a three-layer MLP with a tanh activation function, and the loss function of the classifier is: in, represents the probability of L predicted trajectories, Crossetropy represents the cross entropy loss, and Y represents the future true trajectory.

5. A pedestrian trajectory prediction device based on a progressive refinement strategy, characterized in that: Used to predict the future trajectory of pedestrians using the pedestrian trajectory prediction method based on the progressive refinement strategy described in any one of claims 1-4.

6. A device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the method for predicting pedestrian trajectory based on the progressive refinement strategy according to any one of claims 1 to 4 is used to predict the future trajectory of pedestrians.

7. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the pedestrian trajectory prediction method based on the progressive refinement strategy as described in any one of claims 1 to 4 are implemented.

Images