Multi-modal pedestrian trajectory prediction method and system based on mixed skewed distribution

Abstract

The invention relates to a pedestrian trajectory prediction technology, in particular to a multi-modal pedestrian trajectory prediction method and system based on mixed skewed distribution. The method comprises the following steps: introducing group trajectory standardization, calculating an average trajectory starting point and an average displacement of a group, and unifying coordinate representations of trajectories of different groups; time sequence dependence and space dependence of individual trajectories are modeled through a space-time encoder, and mixed skewed distribution of modeling trajectory end point uncertainty is output; sampling a plurality of trajectory end points from the mixed skewed distribution, performing linear interpolation on the trajectory end points and the trajectory starting points to obtain a preliminary prediction trajectory, and correcting a prediction result through a correction network; and through mean square loss of a prediction result and negative logarithm likelihood optimization model parameters of mixed skewed distribution, a model with a minimum objective function value on a verification set is used as a final model, and a future trajectory of a group is predicted. According to the method, mixed skewed distribution is introduced to model position distribution of individuals, and asymmetric advancing trends and different advancing intentions of the individuals are effectively modeled.

Description

Technical Field

[0001] The present invention relates to a pedestrian trajectory prediction technology, and in particular to a multimodal pedestrian trajectory prediction method and system based on mixed skewed distribution. Background Art

[0002] In urban transportation systems, vehicles, pedestrians, and other individuals form a group. Due to the varying travel intentions of individuals, complex interactions between them, and the complex feasible environments within which they operate, individual trajectories are highly uncertain and diverse. Therefore, predicting multiple possible future trajectories based on an individual's historical trajectory—multimodal trajectory prediction—is of great practical significance. Accurately predicting the future trajectories of surrounding individuals helps plan the future trajectory of the central individual, avoiding collisions and improving safety. This approach has broad applications in areas such as autonomous driving and social robot navigation.

[0003] Taking the individual's position at the last historical moment as the starting point of the trajectory, multi-step trajectory prediction can be achieved by iteratively predicting the position offset of the next moment relative to the current moment. Currently, a representative method uses a normal distribution to model the uncertainty of the offset of each future time step, and predicts the individual's future trajectory by accumulating the offset. This type of method has two major shortcomings. First, because the normal distribution is a symmetric distribution, this type of method fails to effectively model the asymmetric movement trends of the individual, which may cause most of the prediction results to deviate significantly from the actual trajectory. Second, under the conditional independence assumption, the multi-step prediction results obtained by this type of method still obey the normal distribution, which greatly limits the model's ability to model complex distributions. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal pedestrian trajectory prediction method and system based on mixed skewed distribution. The method unifies the coordinate representation of the trajectories of different groups through group trajectory normalization, uses a spatiotemporal encoder to encode the historical trajectories of the group to obtain a mixed skewed distribution estimated trajectory endpoint, and obtains a preliminary predicted trajectory by linear interpolation of the trajectory starting point and trajectory endpoint. The preliminary predicted trajectory is input into a correction network to obtain a more accurate prediction result.

[0005] In an embodiment of the present invention, a multimodal pedestrian trajectory prediction method based on mixed skew distribution is used to predict the trajectory of a group of N individuals in T h historical trajectory of historical time steps Predict all individuals in the future T f There are M possible trajectories for each historical time step, and each future trajectory is represented as Make it consistent with the real trajectory The gap between them should be as small as possible;

[0006] The prediction method comprises the following steps:

[0007] S1. Calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group, and unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory;

[0008] S2. Use a spatiotemporal encoder to encode the historical trajectory of the group in step S1, model the temporal dependency of the individual historical trajectory and the spatial dependency of the interaction between individuals to obtain a latent layer representation, and convert it into parameters of a mixed skewed distribution, including the parameters of each skewed distribution and the mixing coefficient; output the mixed skewed distribution;

[0009] Among them, the mixed normal distribution is composed of the weighted sum of several normal distribution components according to different mixing coefficients. The skewness parameter is introduced into the components of the mixed normal distribution on the basis of the symmetric normal distribution, so that the components of the mixed distribution have the ability to model asymmetry, so as to generalize the mixed skewed distribution.

[0010] S3, sampling multiple possible trajectory endpoints from the mixed skewed distribution obtained in step S2 by stratified sampling;

[0011] S4. Obtain a preliminary predicted trajectory by performing linear interpolation between the trajectory starting point and the trajectory end point obtained in step S3, and input the preliminary predicted trajectory into the correction network to obtain a prediction result that is closer to the actual trajectory;

[0012] S5. Build a prediction model based on the spatiotemporal encoder and the correction network, and train the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain a final prediction model;

[0013] S6. Use the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

[0014] In an embodiment of the present invention, a multimodal pedestrian trajectory prediction system based on mixed skewed distribution is implemented based on the above prediction method. The prediction system includes the following modules:

[0015] The group trajectory normalization module is used to calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group, and unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory;

[0016] The spatiotemporal encoder is used to encode the historical trajectory of the group, model the temporal dependence of the historical trajectory of individuals and the spatial dependence of the interactions between individuals to obtain the hidden layer representation, and convert it into the parameters of the mixed skew distribution, including the parameters of each skew distribution and the mixing coefficient; output the mixed skew distribution;

[0017] Among them, the mixed normal distribution is composed of the weighted sum of several normal distribution components according to different mixing coefficients. The skewness parameter is introduced into the components of the mixed normal distribution on the basis of the symmetric normal distribution, so that the components of the mixed distribution have the ability to model asymmetry, so as to generalize the mixed skewed distribution.

[0018] The sampling module samples multiple possible trajectory endpoints from the mixed skewed distribution output by the spatiotemporal encoder through stratified sampling;

[0019] The correction network obtains a preliminary predicted trajectory by linearly interpolating the trajectory starting point and the trajectory end point obtained by the sampling module. The preliminary predicted trajectory is input into the correction network to obtain a prediction result that is closer to the actual trajectory.

[0020] The prediction model training module builds a prediction model based on the spatiotemporal encoder and the correction network, and trains the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain the final prediction model;

[0021] The prediction module uses the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

[0022] Compared with the prior art, the beneficial effects of the technical solution of the present invention include:

[0023] 1. The present invention introduces group trajectory normalization, obtains personalized translation and rotation transformations based on the group's historical trajectory, unifies the group's trajectory center and displacement direction, and helps the model learn individual motion laws that are independent of absolute coordinates.

[0024] 2. The present invention introduces a mixed skewed distribution model to model the position distribution of individuals, effectively modeling the asymmetric movement trends and different movement intentions of individuals.

[0025] 3. The present invention has the characteristics of plug-and-play and can be applied to existing multimodal trajectory prediction methods based on normal distribution or mixed normal distribution to improve the model's ability to model complex asymmetric distributions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of a multimodal pedestrian trajectory prediction method according to an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of the model structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] This invention is used to predict multiple possible future trajectories for different traffic participants in urban transportation systems. Technically, it introduces group trajectory normalization to unify the coordinate representation of trajectories for different groups, helping the model learn individual motion patterns that are independent of absolute coordinates. Using a spatiotemporal encoder, the invention learns a mixed skewed distribution for each individual, modeling the asymmetric distribution of trajectory endpoints. This allows the model to accurately estimate trajectory endpoints and, based on this, accurately predict the individual's future trajectory.

[0029] This paper evaluates pedestrian trajectory prediction datasets, specifically the ETH-UCY and SDD datasets. The ETH-UCY dataset contains five different scenes, and the SDD dataset contains eight different scenes. This plug-and-play approach is validated by using the advanced pedestrian trajectory prediction method GraphTERN (proposed by Inhwan Bae et al. at the AAAI 2023 conference) as the backbone neural network.

[0030] The problem in this invention is defined as: given a group of N individuals in T h historical trajectory of historical time steps Predict all individuals in the future T f There are M possible trajectories for each historical time step, and each predicted trajectory (i.e., future trajectory) is expressed as Make it consistent with the real trajectory The gap between them should be as small as possible.

[0031] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments, but the embodiments of the present invention are not limited thereto.

[0032] Example

[0033] This embodiment provides a multimodal trajectory prediction method based on mixed skew distribution, which is a deep learning method. The prediction model mainly includes a spatiotemporal encoder and a correction network in structure. Figure 1 、 2 As shown in the figure, for the input trajectory data of multiple individuals, group trajectory normalization is used to unify the coordinate representation of the trajectories of different groups. The input spatiotemporal encoder models the temporal and spatial dependencies of individual trajectories, and the output reflects the uncertainty of the trajectory endpoints. Multiple trajectory endpoints are sampled from the mixed skewed distribution, and linear interpolation is performed on the trajectory starting point and the trajectory endpoint to obtain a preliminary predicted trajectory. The preliminary predicted trajectory is input into a correction network to obtain a more accurate predicted trajectory.

[0034] Specifically, the prediction method of this embodiment includes the following steps:

[0035] S1. Calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group. Unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory.

[0036] Specifically, the starting point of the trajectory is translated to the coordinate origin, the average displacement direction is rotated to be consistent with the positive direction of the coordinate x-axis, and the coordinate representation of the trajectories of different groups is unified, which is conducive to the model learning individual motion laws that are independent of absolute coordinates.

[0037] In this step S1, represents the position of individual i at time t. First, calculate the average position of all individuals at the first and last historical moments, and use the average position at the last historical moment as the starting point of the average trajectory, that is:

[0038]

[0039] Where, represents the position of individual i at the first historical moment, represents the average position of all individuals at the first historical moment; Represents individual i at the last historical moment T h The average position of Indicates that all individuals at the last historical moment T h The average position of .

[0040] The average displacement of different individuals is The angle between the individual and the positive half axis of the x-axis is:

[0041] θ=arctan2(r x ,r y ) (3)

[0042] Where r x is the average displacement of the individual along the x-axis, r y is the average displacement of the individual along the y-axis.

[0043] On this basis, the average trajectory starting point of all individuals Translate to the coordinate origin and rotate clockwise around the coordinate origin by angle θ, that is:

[0044]

[0045] S2. Use a spatiotemporal encoder to encode the historical trajectory of the group in step S1, model the temporal dependency of the individual historical trajectory and the spatial dependency of the interaction between individuals to obtain a latent layer representation, and convert it into parameters of a mixed skewed distribution, including the parameters of each skewed distribution and the mixing coefficient; output the mixed skewed distribution.

[0046] In this embodiment, the mixture of skew-normal distribution is a "generalization" of the commonly used mixture of normal distribution. The mixture of normal distribution can be regarded as a special case of the mixture of skew-normal distribution. The mixture of normal distribution is composed of the weighted sum of several components of normal distribution according to different mixing coefficients. The components of the mixture of normal distribution are generalized from normal distribution to skew distribution to obtain the mixture of skew-normal distribution. Specifically, the skewness parameter is introduced into the components of the mixture of normal distribution on the basis of the symmetric normal distribution, so that the components of the mixture distribution have the ability to model asymmetry to obtain the mixture of skew-normal distribution.

[0047] Specifically, the probability density function of a bivariate mixed normal distribution is:

[0048]

[0049] Where z represents a two-dimensional random variable that obeys a mixed normal distribution, λ k represents the mixing coefficient of the kth normal distribution; represents the probability density function of the kth normal distribution, specifically:

[0050]

[0051] where μ k is the mean of the normal distribution, π represents the circumference of a circle, ∑ k is the covariance matrix of the normal distribution. From the above formula, we can see that the probability density function of the normal distribution is about the point z = μ k Symmetric, therefore, it fails to model the asymmetry of the distribution. When the distribution is asymmetric, the mixed normal distribution requires more components to fit, which increases the number of parameters in the model and is prone to overfitting.

[0052] In this embodiment, the mixed skewed distribution overcomes the above-mentioned overfitting problem by replacing each component with a skewed distribution. The probability density function of the mixed skewed distribution obtained by generalizing the mixed normal distribution is:

[0053]

[0054] where m k represents the mixing coefficient of the kth skewed distribution; Represents the probability density function of the kth skewed distribution, specifically:

[0055]

[0056] Where z represents a two-dimensional random variable that obeys a skewed distribution; is the location parameter, which is used to control the aggregation position of the random variable z in space; is the correlation matrix, which is a 2×2 semi-positive definite matrix used to control the uncertainty of the random variable z in different dimensions; is the skewness parameter, which is used to control the degree of skewness of the random variable z in different dimensions. The following terms of formula (8) specifically describe the difference between skew distribution and normal distribution:

[0057] (1) Represents the probability density function of the 2D normal distribution, The parameter μ in expression (6) k and ∑ k are replaced by the position parameters ξ k and the incidence matrix Ω k , we can calculate the probability density value of the 2D normal distribution. Different from the normal distribution, due to the skewness parameter α k The influence of the position parameter ξ k and the incidence matrix Ω k no longer correspond to the mean and covariance matrix of a normal distribution.

[0058] (2) represents the cumulative distribution function of the 1-dimensional normal distribution, where is the incidence matrix Ω k The diagonal matrix composed of the diagonal elements of represents the scale of the random variable z in different dimensions, and I is a 2×2 identity matrix; is a normalized vector that translates the original random variable z by ξ k To eliminate the influence of the location parameter, and by dividing by ω k To eliminate the influence of scale. When the skewness parameter α k With normalized vector When the angle is less than 90°, the distribution is skewed. The probability density value is smaller than the corresponding normal distribution The probability density value of is large, and vice versa, it is small, thus breaking the normal distribution about the point z=μ k The symmetry of allows the model to better model the asymmetry in the data distribution.

[0059] The mixed skewed distribution has plug-and-play properties and can be applied to trajectory prediction methods based on normal distributions and mixed normal distributions. In this step, a spatiotemporal encoder is used to capture the temporal and spatial dependencies in historical trajectories and to parse the latent representation of the trajectory into the parameters of the mixed skewed distribution.

[0060] Specifically, the spatiotemporal encoder transforms the historical trajectory Encoded as the following hidden layer representation

[0061]

[0062] The hidden layer representation can be parsed as the parameters of the mixed skew distribution, including K skew distribution parameters and the corresponding mixing coefficients. Without loss of generality, the present invention uses the hidden layer representation matrix of an individual As an example, this paper introduces the analytical method of mixed skewed distribution parameters.

[0063] The first column of the hidden layer representation matrix is used to calculate the mixing coefficients of K skew distributions [m1,…m k …,m K ]:

[0064] [m1,…m k …,m K ]=softmax(H[:,0]) (10)

[0065] Where 1≤k≤K.

[0066] The 2nd to 7th columns of the hidden layer representation matrix are used to calculate the location parameters ξ of different skew distributions k , correlation matrix Ω l and the skewness parameter α k :

[0067]

[0068] Each mixing coefficient m k and the parameter of each skewed distribution (ξ k ,Ω k ,α k ) is substituted into the expression of the probability density function of the aforementioned mixed skewed distribution to calculate the probability density value.

[0069] S3. Sample multiple possible trajectory endpoints from the mixed skewed distribution obtained in step S2 by stratified sampling.

[0070] In step S3, a skewed distribution is first sampled according to the mixing coefficient, and then samples are sampled from the selected skewed distribution. Each skewed distribution can be converted to a normalized skewed distribution through translation and scaling transformations. By sampling the normalized skewed distribution and performing an inverse transformation on the sampled samples, a sampling result of any skewed distribution can be obtained.

[0071] In this embodiment, the specific process of stratified sampling in step S3 includes:

[0072] S31. Sample a two-dimensional skewed distribution according to the mixing coefficient of the mixed skewed distribution.

[0073] Specifically, according to the K mixing coefficients [m1,…m k …,m K ] Define a categorical distribution from which sampling an integer k represents the kth skewed distribution.

[0074] S32. Normalize the selected skewed distribution so that its location parameter is zero and the diagonal elements of the correlation matrix are 1.

[0075] Specifically, for the selected skewed distribution Construct the corresponding normalized skewed distribution in is a normalized correlation matrix with all diagonal elements being 1; u represents a two-dimensional random variable that obeys the normalized skewed distribution.

[0076] S33. Construct a 3D normal distribution based on the normalized skewed distribution and sample a vector v from it:

[0077]

[0078] in Is a normalized skewness parameter. To sample vector v, we can first sample from a 3D standard normal distribution Then the incidence matrix Ω * Perform Cholesky decomposition to obtain the lower triangular matrix L, ie Ω * =LL T ; Finally, the sampling result is obtained through linear transformation v=Lw.

[0079] S34. Perform conditional transformation on the vector v sampled in step S33 to obtain a two-dimensional vector u that conforms to the normalized skewed distribution.

[0080] Specifically, the sampled vector v is split into a scalar v0 and a two-dimensional vector v1, and the two-dimensional vector u that conforms to the normalized skewed distribution is obtained through the following conditional transformation:

[0081]

[0082] S35. Perform scaling and translation transformations on the vector u obtained by the conditional transformation in step S34 to obtain a vector z that conforms to the original skewed distribution.

[0083] Specifically, by performing scaling and translation transformation on the vector u, a vector z=ξ+ωu that obeys the original skewed distribution is obtained.

[0084] This step transforms sampling from an arbitrary 2-dimensional skewed distribution into sampling from a normalized 2-dimensional skewed distribution through translation and scaling transformations; and further transforms sampling from a normalized 2-dimensional skewed distribution into sampling from a 3-dimensional normal distribution.

[0085] S4. A preliminary predicted trajectory is obtained by linear interpolation between the starting point of the trajectory and the end point of the trajectory obtained in step S3. The preliminary predicted trajectory is input into the correction network to obtain a prediction result that is closer to the actual trajectory.

[0086] In this step S4, the position of the predicted trajectory end point relative to the last historical moment is set The offset is Z, then the position and the predicted trajectory endpoint The prediction results after linear interpolation are:

[0087]

[0088] Here, τ represents the τth prediction time step.

[0089] The prediction results corrected by the calibration network are:

[0090]

[0091] The Refine correction network uses the same network as GraphTERN, consisting of a single-layer graph convolutional neural network and a three-layer temporal convolutional neural network. This network simultaneously models the spatiotemporal correlations between the initial predicted trajectories of all individuals, resulting in more accurate correction results.

[0092] S5. Build a prediction model based on the spatiotemporal encoder and the correction network, and train the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain the final prediction model.

[0093] This embodiment calculates the mean square error between the predicted result and the true trajectory, and calculates the negative log-likelihood of the mixed skewed distribution inferred by the spatiotemporal encoder in step S2 at the true endpoint. The sum of the mean square error and the negative log-likelihood is used as the objective function to optimize the parameters of the prediction model; the prediction model is trained multiple times until convergence, and the prediction model with the smallest objective function value on the validation set is taken as the final prediction model.

[0094] In step S5, the negative log-likelihood of the trajectory endpoint and the mean square loss of the predicted trajectory relative to the true trajectory are used as the objective function of the prediction model. The objective function is specifically:

[0095]

[0096] in, and They represent the actual position and predicted position of individual i at time t respectively; Indicates that individual i is in T h +T f The position at the moment is the end point of the trajectory.

[0097] According to the calculated objective function value, the parameters of the prediction model are updated using the gradient-based optimizer Adam.

[0098] In this embodiment, the above steps S1 to S5 can be repeated 512 times in total, the prediction model with the smallest loss function value on the validation set is saved, and the final prediction model is tested on the test set and the results are recorded.

[0099] S6. Use the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

[0100] This embodiment further provides a multimodal pedestrian trajectory prediction system based on mixed skewed distribution, which is implemented based on the above prediction method. The prediction system includes the following modules:

[0101] The group trajectory normalization module is used to calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group, and unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory;

[0102] The spatiotemporal encoder is used to encode the historical trajectory of the group, model the temporal dependence of the historical trajectory of individuals and the spatial dependence of the interactions between individuals to obtain the hidden layer representation, and convert it into the parameters of the mixed skew distribution, including the parameters of each skew distribution and the mixing coefficient; output the mixed skew distribution;

[0103] Among them, the mixed normal distribution is composed of the weighted sum of several normal distribution components according to different mixing coefficients. The skewness parameter is introduced into the components of the mixed normal distribution on the basis of the symmetric normal distribution, so that the components of the mixed distribution have the ability to model asymmetry, so as to generalize the mixed skewed distribution.

[0104] The sampling module samples multiple possible trajectory endpoints from the mixed skewed distribution output by the spatiotemporal encoder through stratified sampling;

[0105] The correction network obtains a preliminary predicted trajectory by linearly interpolating the trajectory starting point and the trajectory end point obtained by the sampling module. The preliminary predicted trajectory is input into the correction network to obtain a prediction result that is closer to the actual trajectory.

[0106] The prediction model training module builds a prediction model based on the spatiotemporal encoder and the correction network, and trains the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain the final prediction model;

[0107] The prediction module uses the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

[0108] The above modules are used to implement steps S1-S6 respectively. Please refer to each step for detailed implementation process.

[0109] In general, the present invention can accurately predict the future trajectory of a group based on its historical trajectory. Specifically, group trajectory normalization is introduced to calculate the group's average trajectory starting point and average displacement, based on which translation and rotation transformations are performed to unify the coordinate representation of the trajectories of different groups. A spatiotemporal encoder is used to model the temporal and spatial dependencies of individual trajectories, outputting a mixed skewed distribution of the uncertainty of the modeled trajectory endpoints. Multiple trajectory endpoints are sampled from the mixed skewed distribution and linearly interpolated with the trajectory starting point to obtain a preliminary predicted trajectory, which is then corrected using a correction network. Model parameters are optimized using the mean squared loss of the predicted results and the negative log-likelihood of the mixed skewed distribution. The model with the smallest objective function value on the validation set is selected as the final model to predict the group's future trajectory.

[0110] The present invention uses two evaluation indicators to measure the prediction model and the predicted future trajectory:

[0111] 1. ADE (Average Displacement Error): The average of the Euclidean distances between the predicted position and the true position at each step;

[0112] 2. FDE (end displacement error): the Euclidean distance between the predicted end point and the true end point.

[0113] This paper compares a variety of methods based on generative models, including: 1) Social-STGCNN based on normal distribution

[0114] and SGCN; 2) Trajectron++ and GraphTERN based on mixed normal distribution; 3) SoPhie based on generative adversarial network; 4) MID based on diffusion model; 5) FlowChain based on standardized flow model.

[0115] The final experimental results are shown in the following table:

[0116] Table 1 ADE / FDE (m) experimental results of the present invention on the ETH-UCY dataset

[0117]

[0118] Table 2 ADE / FDE (pixel) experimental results of the present invention on the SDD dataset

[0119] index Pixel SoPhie 16.27 / 29.38 Social-STGCNN 14.03 / 22.87 Trajectron++ 12.75 / 24.23 SGCN 11.20 / 20.66 MID 10.31 / 17.37 FlowChain 9.93 / 17.17 GraphTERN 9.52 / 16.91 The present invention 7.86 / 12.97

[0120] The experimental results above demonstrate that the proposed method significantly improves upon other methods on most datasets. In particular, the proposed method effectively reduces the FDE, demonstrating that using a mixed skewed distribution to estimate position distribution can effectively reduce the cumulative error of multi-step trajectory prediction.

[0121] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal pedestrian trajectory prediction method based on mixed skewed distribution, characterized in that: For a group composed of a given N individuals in T h historical trajectory of historical time steps Predict all individuals in the future T f There are M possible trajectories for each historical time step, and each future trajectory is represented as Make it consistent with the real trajectory The gap between them should be as small as possible; The prediction method comprises the following steps: S1. Calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group, and unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory; S2. Use a spatiotemporal encoder to encode the historical trajectory of the group in step S1, model the temporal dependency of the individual historical trajectory and the spatial dependency of the interaction between individuals to obtain a latent layer representation, and convert it into parameters of a mixed skewed distribution, including the parameters of each skewed distribution and the mixing coefficient; output the mixed skewed distribution; Among them, the mixed normal distribution is composed of the weighted sum of several normal distribution components according to different mixing coefficients. The skewness parameter is introduced into the components of the mixed normal distribution on the basis of the symmetric normal distribution, so that the components of the mixed distribution have the ability to model asymmetry, so as to generalize the mixed skewed distribution. S3, sampling multiple possible trajectory endpoints from the mixed skewed distribution obtained in step S2 by stratified sampling; S4. Obtain a preliminary predicted trajectory by performing linear interpolation between the trajectory starting point and the trajectory end point obtained in step S3, and input the preliminary predicted trajectory into the correction network to obtain a prediction result that is closer to the actual trajectory; S5. Build a prediction model based on the spatiotemporal encoder and the correction network, and train the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain a final prediction model; S6. Use the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

2. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In step S1, first calculate the average position of all individuals at the first historical moment and the last historical moment, and use the average position of the last historical moment as the starting point of the average trajectory; then calculate the average trajectory starting point of all individuals Translate to the origin of the coordinate system and rotate clockwise around the origin of the coordinate system by an angle θ, where θ is the angle between the individual and the positive half axis of the x-axis.

3. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In step S2, the spatiotemporal encoder converts the historical trajectory Encoded as hidden layer representation The hidden layer representation can be parsed as parameters of a mixed skewed distribution, including K skewed distribution parameters and corresponding mixing coefficients.

4. The multimodal pedestrian trajectory prediction method according to claim 3, characterized in that: For the hidden layer representation matrix The analytical method for the parameters of the mixed skewed distribution is: The first column of the hidden layer representation matrix is used to calculate the mixing coefficients of K skew distributions [m1,…m k …,m k ]: [m1,…m k …,m K ]=softmax(H[:,0]) Where 1≤k≤K; The 2nd to 7th columns of the hidden layer representation matrix are used to calculate the location parameters ξ of different skew distributions k , correlation matrix Ω k and the skewness parameter α k : Each mixing coefficient m k and the parameter of each skewed distribution (ξ k ,Ω k ,α k ) into the expression for the probability density function of the mixed skewed distribution to calculate the probability density value.

5. The multimodal pedestrian trajectory prediction method according to claim 4, characterized in that: In step S2, the probability density function of the binary mixed normal distribution is: Where z represents a two-dimensional random variable that obeys a mixed normal distribution, λ k represents the mixing coefficient of the kth normal distribution; Represents the probability density function of the kth normal distribution: where μ k is the mean of the normal distribution, π represents the circumference of a circle, ∑ k is the covariance matrix of the normal distribution; the probability density function of the mixed skewed distribution obtained by generalizing the mixed normal distribution is: where m k represents the mixing coefficient of the kth skewed distribution; Represents the probability density function of the kth skewed distribution: Where z represents a two-dimensional random variable that obeys a skewed distribution; is the location parameter, which is used to control the aggregation position of the random variable z in space; is the correlation matrix, which is a 2×2 semi-positive definite matrix used to control the uncertainty of the random variable z in different dimensions; is the skewness parameter, which is used to control the degree of skewness of the random variable z in different dimensions; Represents the probability density function of the 2D normal distribution; represents the cumulative distribution function of the 1-dimensional normal distribution, where is the incidence matrix Ω k The diagonal matrix composed of the diagonal elements of represents the scale of the random variable z in different dimensions, and I is a 2×2 identity matrix; is a normalized vector that translates the original random variable z by ξ k To eliminate the influence of the location parameter, and by dividing by ω k To eliminate the effect of scale.

6. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In the stratified sampling of step S3, a skewed distribution is first sampled according to the mixing coefficient, and then sampling is performed from the selected skewed distribution; wherein each skewed distribution is converted into a normalized skewed distribution through translation and scaling transformations, and the sampling results of any skewed distribution are obtained by sampling the normalized skewed distribution and performing an inverse transformation on the sampled samples.

7. The multimodal pedestrian trajectory prediction method according to claim 6, characterized in that: Step S3 stratified sampling includes the following steps: S31. sampling a 2-dimensional skewed distribution according to a mixing coefficient of the mixed skewed distribution; S32. normalizing the selected skewed distribution so that its location parameter is zero and the diagonal elements of the correlation matrix are 1; S33. Construct a 3D normal distribution based on the normalized skewed distribution, and sample a vector v from it; S34, performing conditional transformation on the vector v sampled in step S33 to obtain a two-dimensional vector u that conforms to the normalized skewed distribution; S35. Perform scaling and translation transformations on the vector u obtained by the conditional transformation in step S34 to obtain a vector z that conforms to the original skewed distribution.

8. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In step S4, the position of the predicted trajectory endpoint relative to the last historical moment is set The offset is Z, then the position and the predicted trajectory endpoint The prediction results after linear interpolation are: Among them, τ represents the τth prediction time step; The prediction results corrected by the calibration network are: Among them, the correction network includes 1 layer of graph convolutional neural network and 3 layers of temporal convolutional neural network.

9. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In step S5, the mean square error between the predicted result and the true trajectory is calculated, and the negative log-likelihood of the mixed skewed distribution inferred by the spatiotemporal encoder in step S2 at the true endpoint is calculated. The sum of the mean square error and the negative log-likelihood is used as the objective function to optimize the parameters of the prediction model; the prediction model is trained multiple times until convergence, and the prediction model with the minimum objective function value on the validation set is selected as the final prediction model; The objective function is: in, and They represent the actual position and predicted position of individual i at time t respectively; Indicates that individual i is in T h +T f The position at the moment is the end point of the trajectory.

10. A multimodal pedestrian trajectory prediction system based on mixed skewed distribution, implemented based on the prediction method according to any one of claims 1 to 9, characterized in that: The prediction system includes the following modules: The group trajectory normalization module is used to calculate the average trajectory starting point and average displacement of different individuals based on the historical trajectory of the group, and unify the coordinate representation of the trajectories of different groups through translation and rotation transformation to normalize the group trajectory; The spatiotemporal encoder is used to encode the historical trajectory of the group, model the temporal dependence of the historical trajectory of individuals and the spatial dependence of the interactions between individuals to obtain the hidden layer representation, and convert it into the parameters of the mixed skew distribution, including the parameters of each skew distribution and the mixing coefficient; output the mixed skew distribution; Among them, the mixed normal distribution is composed of the weighted sum of several normal distribution components according to different mixing coefficients. The skewness parameter is introduced into the components of the mixed normal distribution on the basis of the symmetric normal distribution, so that the components of the mixed distribution have the ability to model asymmetry, so as to generalize the mixed skewed distribution. The sampling module samples multiple possible trajectory endpoints from the mixed skewed distribution output by the spatiotemporal encoder through stratified sampling; The correction network obtains a preliminary predicted trajectory by linearly interpolating the trajectory starting point and the trajectory end point obtained by the sampling module. The preliminary predicted trajectory is input into the correction network to obtain a prediction result that is closer to the actual trajectory. The prediction model training module builds a prediction model based on the spatiotemporal encoder and the correction network, and trains the parameters of the prediction model, including the parameters of the spatiotemporal encoder and the correction network, to obtain the final prediction model; The prediction module uses the final prediction model to predict the future trajectory of each individual based on the historical trajectory data of the group.

Images