A pedestrian trajectory prediction method based on multi-disconnected mode learning

Abstract

The application relates to a pedestrian trajectory prediction method based on a multi-disconnected mode learning, a prediction trajectory is generated based on a social disconnected mode generation adversarial network model, an encoder is used to extract visual features and observed pedestrian trajectory features, physical scene and social attention features are obtained based on an attention module; a structured graph sequence based on attention labeling is established by using the social attention, and the structured graph sequence is encoded by using a space-time encoder to extract transient changes of a physical background and pedestrian movement; the physical scene attention, the social attention and space-time encoder output features are spliced and input into a multi-generator architecture, and the future trajectory of a predicted pedestrian is output; a generator selector is used for prior learning of the multi-generator, and a spectral trajectory clustering module is used for updating an upper limit of the number of generators in the prior learning process. Compared with the prior art, the application can capture transient changes of space-time information, reduce model redundancy, and be flexibly adapted to multiple prediction scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method based on multi-disconnected mode learning. BACKGROUND

[0002] In prediction learning, predicting the future behavior of humans is very important in artificial intelligence systems. Trajectory prediction in real-world scenarios is a challenging complex problem. The moving bodies in the trajectory prediction problem often have complex interaction relationships. For a pedestrian in a certain traffic scene, their progress is not independent of each other. They react and adjust their strategy according to their own strategy. For example, pedestrians will adjust their future behavior when encountering obstacles or other pedestrians. The former generally comes from information in the visual background, and the latter refers to the social interaction between pedestrians. The distribution of the future trajectory of the pedestrian is complex and diverse, and the pedestrian can walk out of multiple possible and reasonable future trajectories. Imagine a pedestrian arriving at an intersection, he can probably go straight, turn left or right. Moreover, no matter which road the pedestrian takes, if any two roads are connected, it means that the pedestrian can go to a trajectory that does not actually exist, i.e. an out-of-distribution (OOD) trajectory. In other words, the pedestrian trajectory data falls on the union of multiple disjoint modes. Figure 1 The above-mentioned pedestrian trajectory prediction scenario involving complex interaction and multiple modes is shown in the figure, and the area between the upper and middle trajectories in the figure is an impassable obstacle. If the trajectory distribution is a single mode that is globally connected, the model can easily predict an out-of-distribution trajectory that crosses the obstacle.

[0003] For example, Chinese patent applications CN114580715A and CN114580715A, many previous works use generative adversarial networks to generate diverse trajectories, and some works use a multi-generator architecture, where each generator is specialized in learning one of the multiple disconnected modes to model real trajectories that are mixtures of multiple modes. However, this method has the problem of blindly selecting a fixed number of generators, which can cause problems of mode loss and redundancy of the model if the number of generators is too large, and cannot fully cover the real trajectory distribution if the number of generators is too small, which will generate a large number of OOD samples and adversely affect the quality of the generated samples, and it is difficult to adapt to various prediction scenarios. SUMMARY

[0004] The purpose of the present application is to provide a pedestrian trajectory prediction method based on multi-disconnected mode learning.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A pedestrian trajectory prediction method based on multi-disconnected mode learning, the method generates a predicted trajectory by generating an adversarial network model based on a social disconnected mode, wherein the input of the model includes observed pedestrian i trajectory coordinate data in the scene and scene image I containing visual information, and the specific steps of the model generating a predicted trajectory include:

[0007] extracting visual features using a physical scene encoder and extracting observed pedestrian trajectory features using a social encoder; obtaining physical scene attention features and social attention features through physical scene attention and social attention based on the outputs of the physical scene encoder and the social encoder;

[0008] establishing an attention-labeled structured graph sequence based on social attention features, and encoding and extracting physical background and transient changes of pedestrian movement using a spatio-temporal encoder on the structured graph sequence;

[0009] inputting the concatenation of physical scene attention, social attention, and spatio-temporal encoder output features into a multi-generator architecture to output the future trajectory of the predicted pedestrian; each generator in the multi-generator architecture receives the concatenated input and learns one of the multiple disconnected modes of the trajectory distribution separately;

[0010] learning the prior of the multiple generators using a generator selector to select a generator with a larger prior value to generate the future trajectory, and updating the upper limit of the number of generators using a spectral trajectory clustering module during the prior learning process.

[0011] Further, the physical scene encoder extracts visual features, and the specific steps include:

[0012] extracting visual features V from scene image I p ;

[0013] augmenting the original data, calculating the speed according to the absolute coordinates, and representing the acceleration;

[0014] obtaining pedestrian i trajectory at time step t.

[0015] Further, the social encoder takes the sequence of pedestrian trajectory as input to learn the feature representation V s (i) of the observed trajectory, and the specific steps include:

[0016] obtaining a high-dimensional embedding of the sequence of pedestrian trajectory through a multi-layer perceptron;

[0017] inputting the embedding sequence through LSTM to obtain pedestrian i trajectory feature

[0018] Further, the physical scene attention input is visual features V p and social encoder LSTM last observation time step t obs under the trajectory encoding hidden state The physical scene attention applies soft attention to obtain context vector C ph (i);

[0019] The social attention calculates the attention score between any two adjacent pedestrians based on the distance and azimuth angle between them; the pedestrian trajectory features V en (i) obtained by the social encoder LSTM embedding are weighted to obtain social features V so (i); the social features V so (i) and t obs The average hidden state of LSTM decoder pedestrian i at time t is input into the social attention network ATT s , and the social attention vector C so (i) is output.

[0020] Further, the structured spatio-temporal graph is constructed, and the specific steps include:

[0021] A local image centered on the target pedestrian is cropped from the global scene image I;

[0022] A graph with the same length and width as the cropped image is constructed, the constructed graph has an additional channel dimension, and the additional channel dimension is divided into a two-dimensional grid, and the position of the pedestrian in the graph at time step t is represented by the distance between each real trajectory coordinate X t The nearest grid cell;

[0023] The pedestrian trajectory features V en (i) and the social attention vector C so (i) are spliced to obtain the annotation vector of pedestrian i at time step t;

[0024] In all grid cells containing pedestrians, the annotation vectors of the target pedestrian and other pedestrians in the local area are filled in the additional channel dimension.

[0025] Further, the multi-generator architecture includes:

[0026] n G generators, all generators share the same network structure but use different weights; each generator includes an LSTM decoder, and the generator accepts pedestrian trajectory features V en (i), physical scene attention features C ph(i) social attention feature C so (i) spatio-temporal encoding C st (i) and random noise vector z and concatenate the receptive features to get the generator input C gen (i), the generator generates the future trajectory of pedestrian i

[0027] discriminator, which gets the discriminator input C by concatenating the pedestrian trajectory feature, the physical scene attention feature, the social attention feature and the spatio-temporal encoding obtained from the complete sequence containing the observation sequence and the prediction sequence dis (i), wherein the prediction sequence can be the real trajectory Y or the generated future time period trajectory using a multi-layer perceptron to calculate the discriminator score the discriminator score is a binary classification score, used to represent whether the sample is real or not;

[0028] classifier, which receives the same input C as the discriminator dis (i), using a multi-layer perceptron to calculate the classification score the is an n G category classification score, used to represent the probability that the generated trajectory belongs to each generator.

[0029] Further, the model uses an EM algorithm to learn the prior, and the specific training steps include:

[0030] E-step: generator selector training, learning the prior probability s(g; ζ) of each generator to approximate the optimal prior p(g);

[0031] M-step: multi-classifier architecture GAN training, learning the probability q(g|Y) that the future trajectory Y belongs to each generator G g to approximate the real posterior probability p(g|Y).

[0032] Further, the multi-classifier architecture GAN training is as follows:

[0033] For the multi-generator structure, the goal of each generator is to maximize the error of the common discriminator, wherein the loss function of the GAN training includes:

[0034] adversarial loss using the original adversarial loss, considering the distribution of n G generators, the calculation is weighted for each generator;

[0035] trajectory diversity loss using the best multiple loss calculation method to encourage each generator to generate diverse trajectories;

[0036] Classification loss The cross-entropy is used to calculate the loss between the generator index of the classifier output and the true generator index of the generated trajectory.

[0037] Further, the generator selector is trained as follows:

[0038] Using the learning result of the classifier, the cross-entropy between the model distribution s(g) and the true posterior p(g|Y) is calculated to train the generator selector, wherein q(g|Y) is used to approximate p(g|Y);

[0039] On the basis of the cross-entropy loss , the model distribution entropy H(s(g)) is added, and the decay rate alpha changing with the training time is set t to ensure smoothness;

[0040] During the training process, the trajectory spectrum clustering module is used to add an additional upper limit to the number of generators, and more than the upper limit of the generator will be discarded and continue to be trained.

[0041] Further, the trajectory spectrum clustering module adaptively determines the optimal cluster number according to the predicted trajectory, that is, the number of selected generators, and the specific steps include:

[0042] Based on the calculation of the similarity of the predicted trajectory, a similarity matrix K is obtained;

[0043] The similarity matrix is normalized to obtain a matrix L;

[0044] The spectral information of the matrix L is used to determine the range of the cluster number;

[0045] Based on the information obtained from the characteristic decomposition of L, the distortion score under each cluster number is calculated;

[0046] The cluster number with the best distortion score is selected as the final selected generator number.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] 1) The present application designs a structured space-time graph, which recombines the pedestrian trajectory and the background image through coding to form a structured space-time graph, which can represent the complex social changes between the physical scene and the pedestrians, and assist in modeling the long-term dependence relationship in the pedestrian trajectory. Through the space-time encoder, the transient changes of the structured space-time graph over time can be captured, which can effectively improve the accuracy of trajectory prediction and improve the ability of the model to predict complex pedestrian behavior.

[0049] 2) In the process of prior learning of multiple generators, the spectral clustering is used for the predicted trajectory, the upper limit of the number of generators is limited, the probability of each generator is selected, and the necessary number of generators is inferred, so as to filter out redundant generators. The scheme of the present application can reduce model redundancy, and can flexibly adapt to different trajectory prediction scenes compared with the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A schematic diagram for a pedestrian trajectory prediction scene;

[0051] Figure 2 A schematic diagram of the Social-DMGAN model structure of the present application;

[0052] Figure 3 A schematic diagram of the establishment of a structured spatiotemporal graph;

[0053] Figure 4 A schematic diagram of the trajectory prediction result of the Social-DMGAN of the present application; the top behavior is the scene of pedestrians passing through a T-shaped intersection, and the future trajectory contains two unconnected modes; the bottom behavior is the scene of pedestrians passing through a cross intersection, and the future trajectory contains three unconnected modes; Fig. (a) is a real trajectory containing multiple unconnected modes after processing the SDD dataset, Fig. (b) is the result of the predicted clustering trajectory after the spectral clustering model, and Fig. (c) is the determination result of whether the predicted trajectory is an OOD sample, and shows the prior distribution of each generator learned by the generator selector. DETAILED DESCRIPTION

[0054] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical scheme of the present application, and gives a detailed implementation manner and specific operation process, but the protection scope of the present application is not limited to the following examples.

[0055] Example 1

[0056] 1.1 Problem definition

[0057] Pedestrian trajectory prediction is a problem of predicting the future directed movement of pedestrians in a given environment, which involves analyzing the current position, speed, direction and behavior of pedestrians, the layout of the environment, and the existence of obstacles and other pedestrians. The goal of the present application is to make full use of spatiotemporal information to learn a multi-modal trajectory distribution and generate reasonable future trajectories for each pedestrian. For a given prediction scene, the input of the model contains two parts: (1) observed trajectory coordinate data of pedestrian i in the scene: (2) a scene image I containing visual information, which is flipped, rotated and scaled at different scales. The model finally needs to generate K different predicted trajectories

[0058] 1.2 Model Overview

[0059] As Figure 2 shown in the figure is the structure of the proposed Social-DMGAN model. The model uses a physical scene encoder to extract visual features, and a social encoder to extract features of observed pedestrian trajectories. The attention module contains two parts: physical attention and social attention, where the social attention is also used to build a structured graph sequence based on attention labels, which uses a spatiotemporal LSTM (ST-LSTM) encoder to extract the transient changes of physical background and pedestrian movement. The overall model adopts a multi-generator architecture, where each generator receives the concatenation of social attention, physical scene attention, and spatiotemporal encoding as input, and is specifically used to learn one of the multiple disconnected modes of trajectory distribution. To improve the quality of prediction, a generator selector is proposed to learn the prior of multiple generators, and the generator with a larger prior value can be used to generate the future trajectory. During training, the spectral clustering module can update the required number of generators every period of time. Since the number of preset generators is usually larger than the actual number of required generators, the spectral clustering module can effectively achieve model redundancy reduction.

[0060] 1.3 Trajectory and Scene Encoding

[0061] In this invention, the scene image and the observed trajectory sequence are respectively encoded by a physical encoder and a social encoder to obtain a high-dimensional feature representation in the space. The physical scene encoder uses a convolutional neural network (CNN) to extract visual features V p from the scene image I, where I contains information at multiple different scales and after flipping and rotating. Then the original data is augmented. According to the absolute coordinate calculation speed, the acceleration is represented by , and finally the input trajectory of pedestrian i at time step t is The social encoder takes the sequence of as input to learn the feature representation V s (i) of the observed trajectory. The steps are as follows: 1. The sequence of through a Multi-layer Perceptron (MLP) to get high-dimensional embedding; 2. The embedding sequence is input through LSTM to get the feature of pedestrian i at time step t The encoding process of the scene image and the trajectory sequence is represented as follows:

[0062] V p = CNN(I; W cnn )

[0063] h en (i) = LSTM en (MLP embed (S i , W embed ), h en (i); W en )

[0064] The attention mechanism allows the model to selectively focus on certain parts of the trajectories in the scene, just like pedestrians selectively focus on certain stimuli in their interaction with obstacles and other pedestrians (e.g. to avoid collisions). Attention is applied to the extracted features to allow the model to focus on relevant information in the trajectory prediction task. For physical attention, soft attention is applied to obtain the context vector C ph (i), where the input is the visual feature V p extracted from the scene image and the trajectory encoding hidden state obs from the LSTM encoder at the last observation time step t The network is parameterized by W ph For social attention, first the attention score between any two neighboring pedestrians is calculated based on their distance and azimuth angle, then these scores are used to weight the encoder LSTM’s embedding V en (i) to obtain the social feature V so (i). V so (i) is then fed into the social attention network ATT s In addition, the hidden state of the decoder LSTM used to generate the predicted trajectory is also used as input. According to the multi-generator structure of the present method, the generator prior is obtained by a generator selector, and the hidden state of pedestrian i at time step t obs of each generator is weighted to obtain the average hidden state from the prior, which contains information about the predicted pedestrian’s future path. The specific content of the generator selector will be detailed in the later sections. The attention network is parameterized by W so and outputs the social attention vector Cso (i):

[0065] C ph (i) = ATT p (V p , V en (i); W ph )

[0066]

[0067] Social attention feature C so (i) will be used to construct the spatio-temporal graph described in detail in the next section.

[0068] 1.4 Structured Spatio-Temporal Graph

[0069] Due to the existence of missing spatio-temporal data constraints in existing trajectory datasets, and inspired by graph attention networks and spatio-temporal sequence prediction tasks, a structured spatio-temporal graph is proposed, as shown in Figure 3 The construction process of this graph structure is as follows:

[0070] 1. Crop a local image of size H x W centered on the target pedestrian from the global scene image I. In order to provide clear semantics to the graph, the global image can be a semantic image or a manually annotated obstacle occupation map;

[0071] 2. Construct a graph with the same length and width as the cropped image, with an additional channel dimension, and divide it into a two-dimensional grid. Represent the distance between each real trajectory coordinate X t The nearest grid cell represents the position of the pedestrian at time step t;

[0072] 3. Concatenate the observation encoding V en (i) and the social attention vector C so (i) to obtain the annotation vector of pedestrian i at time step t. In all grid cells containing pedestrians, the annotation vectors of the target pedestrian and other pedestrians within the local region are filled in the channel dimension, i.e. attention annotation.

[0073] It should be noted that the existing mature model is used to encode the spatio-temporal graph, and in the invented model this encoding module is called ST-LSTM, and the spatio-temporal encoding is denoted as C st(i). The focus of the method of the present application is on structured graph sequences incorporating spatio-temporal information for trajectory prediction. The spatio-temporal graph sequences can be used to assist in modeling long-term dependencies in pedestrian trajectories, enabling the model to capture the scene-dependent positional changes of pedestrians moving over time. This structure can effectively improve the accuracy of trajectory prediction and enhance the model's ability to predict complex pedestrian behaviors, such as situations involving crossing the street or interacting with other pedestrians.

[0074] 1.5 Multi-generator architecture

[0075] Generator: In the model of the present application, n G generators are constructed, all of which share the same network structure but do not share weights. Each generator consists of an LSTM decoder that accepts pedestrian trajectory encoding V en (i), physical scene attention C ph (i), social attention C so (i), spatio-temporal encoding C st (i), and a random noise vector z sampled from a normal distribution. These features are concatenated to obtain C gen (i) = [V en (i), C ph (i), C so (i), C st (i)]. Therefore, the future trajectory of pedestrian i can be obtained as follows:

[0076]

[0077] where g e [1, n G ] represents the index of the generator.

[0078] Discriminator: Unlike C gen (i) in the generator, which only processes the trajectory during the observation period, the discriminator needs to additionally process the trajectory in the future time period, i.e., the ground truth Y and the generated fake trajectory Pedestrian trajectory features, physical scene attention features, social attention features, and spatio-temporal encoding are obtained from the encoding of the complete sequence containing the observation sequence and the prediction sequence. C gen (i) is concatenated in a similar manner to C dis (i), which is used as input to the discriminator, where the prediction sequence can be the real trajectory Y or the generated future time period trajectory The score of the discriminator is calculated using a multi-layer perceptron (MLP):

[0079]

[0080] where, is a binary classification score indicating whether the sample is real or not. However, the discriminator itself can cause mode dropping, i.e. several disconnected modes of the actual trajectory data are not fully covered by the generated trajectory distribution.

[0081] Classifier: The classifier plays an important role in the multi-generator model, which encourages the generators to cover as many disconnected mixed-mode trajectories as possible. The invention also introduces a classifier that focuses on identifying which generator generated the trajectory. Since each generator receives the same features and latent inputs sampled from the same distribution, all generators can learn to generate similar samples if this additional classifier is not used. In the model of the invention, the classifier receives the same input C dis (i) as the discriminator and also computes a classification score using an MLP:

[0082]

[0083] where, is an n G category classification score indicating the probability that the generated trajectory belongs to each generator.

[0084] 1.6 Generator selector

[0085] In the real world, there can be differences in the likelihood of pedestrians walking in different directions, or in other words, in different disconnected modes. It is desirable to model which direction, or in other words, which generator is more likely to generate a pedestrian. A flexible generator selector is constructed using an MLP:

[0086]

[0087] Here, g e [1, n G ] represents the index of the generator, represents the prior probability assigned to each generator. The model output represents the probability that each generator generates the future trajectory.

[0088] Although the prior probability of the generator can be learned by the prior art solution, the number of generators is fixed and cannot be flexibly adjusted according to a specific scenario. Suppose there are n R disconnected manifolds in the actual trajectory, if n R < n G , then blindly setting the number of generators n G may result in a large number of OOD samples, because some generators must be forced to cover multiple disconnected modes. Conversely, if n R > n GIf some generators must share a pattern, only a subset of disconnected patterns will be learned. The goal of a multi-generator architecture is for each generator to learn only one of several disconnected patterns, which can be understood from the perspective of interpretability in real-world scenarios as multiple clusters of trajectories. Trajectory clustering methods are used to help explore the number of generators in a multi-generator architecture. This method primarily addresses two cases: 1) Traditional trajectory prediction datasets (such as ETH, UCY, SDD) represent actual trajectories. For each pedestrian, the future trajectory, i.e., the label, is uniquely determined. In this case, the generator selector needs to cluster and update the number of generators from the trajectories generated by multiple generators. 2) Processed datasets. In the real world, there is no prior knowledge of the number and distribution of disconnected trajectory patterns. Therefore, simulation processing is performed on the SDD dataset with special scenarios (intersections, roundabouts, etc.) to make the future trajectory of each pedestrian diverse, not just a single trajectory, but multiple dispersed trajectories, so that the future trajectory distribution can be viewed as a mixture of multiple disconnected patterns. In this case, the number of selectable actual generators n sel This can be inferred from the maximum number of disconnected patterns in each pedestrian's future trajectory, i.e., nse. l =max{r i |1≤i≤N}, where r i This represents the number of disconnected patterns in pedestrian i's future trajectory. The goal is for the pattern distribution from trajectory clustering to match the future trajectory distribution obtained through simulation.

[0089] This invention uses spectral clustering to adaptively determine the optimal number of clusters, i.e., the number of selected generators, based on the predicted trajectory. Trajectory similarity is measured using a robust Modified Hausdorff Distance (MHD). The input to the spectral clustering module is a similarity matrix K, where k... ij This represents the similarity between trajectories i and j. Using L=W -1 / 2 KW -1 / 2 Normalize the similarity matrix, where the i-th element of the diagonal matrix W is... th Each element is defined as w i =∑ 1≤j≤n k ij The appropriate range for the number of clusters is determined using the spectral information of L. Next, the distortion score for each cluster number is calculated based on the information obtained from the eigenvalue decomposition of L. The cluster number with the optimal distortion score is selected as the final number of generators to be chosen. In summary, the algorithm primarily relies on the priors of each generator and adds a spectral trajectory clustering module to limit the upper limit of the number of generators based on the number of clusters, filtering out redundant generators and flexibly adapting to various prediction scenarios.

[0090] 1.7 Model training

[0091] The model is trained using a method similar to the EM algorithm. The optimal priors p(g) of the generators are derived by maximizing the log-likelihood of the data given the encoding features C gen (i) and the predicted trajectories Using q(g|Y) to denote the probability of the future trajectory Y belonging to each generator G g and approximating the true posterior p(g|Y) with q(g|Y). The EM algorithm is used to learn the priors: the expectation of q(g|Y) over the true data distribution gives an approximation of p(g) (E-step: generator selector training) which can be used to train the model of the multi-generator architecture of the invention (M-step: GAN training). The GAN training is similar to a standard GAN, for the multi-generator structure the goal of each generator is to maximize the error of the common discriminator. The multi-generator needs to generate realistic trajectories, the discriminator needs to distinguish between real trajectories Y and generated trajectories the classifier needs to let the generators cover as many trajectory patterns as possible. The GAN training loss function contains three parts:

[0092] 1. Adversarial loss The original adversarial loss is used. Considering the distribution of n G generators, each generator needs to be weighted for the calculation: where, is the generator weight obtained by softmax activation.

[0093] 2. Trajectory diversity loss The best-of-many loss calculation method is used to encourage each generator to generate diverse trajectories.

[0094] 3. Classification loss The cross-entropy is used to calculate the classification loss, i.e. the loss between the generator index output by the classifier and the true generator index of the generated trajectory. In general, the training objective can be summarized as follows:

[0095]

[0096] where λ variety and λ cls are hyperparameters.

[0097] Generator selector training: Assuming the optimal prior probability of each generator is p(g), the generator selector learns the generator prior probability s(g; ζ). The purpose of training is to learn the prior probability s(g; ζ) of each generator to approximate the optimal prior p(g). As mentioned before, the learning of the classifier can be summarized as learning the distribution q(g|Y) representing the likelihood of the future trajectory Y belonging to each generator to approximate the real posterior p(g|Y). Using the results of the learning of the classifier, the generator selector is trained with the following cross-entropy loss:

[0098]

[0099] where H(p(g|Y), s(g)) is the cross-entropy between the model distribution s(g) and the real posterior p(g|Y), approximated by q(g|Y). On the basis of , the model distribution entropy H(s(g)) is added. The training objective is as follows:

[0100]

[0101] In the initial stage of model training, the model has not yet been able to generate more realistic and diverse trajectories, and a decay rate a t that changes with training time is used to ensure smoothness. After sufficient training iterations, there can still be a large number of generators in the generator selector that are activated in the multi-generator architecture due to being greater than a given probability threshold. The trajectory spectrum clustering module is used to add an additional upper limit to the number of generators, and more than the upper limit of the generators will be discarded and continue to be trained.

[0102] Experimental verification

[0103] 2.1 Experimental setup

[0104] Experiments are conducted on three publicly available real-world datasets ETH, UCY and SDD, and one simulated dataset FPD (Forking Paths Dataset). The ETH and UCY datasets together contain five scenes: ETH, HOTEL, UNIV, ZARA1 and ZARA2. In addition, multiple future trajectories of each pedestrian are simulated on the SDD dataset to verify the model's ability to learn multiple unrelated manifolds, as well as the flexibility of the number of generators and the prior. ADE and FDE are used as evaluation metrics, and the strategy of selecting the minimum value among K trajectories is used for evaluation, where K is set to 20. The method in is also used to evaluate the ability to reduce OOD samples in trajectory prediction using precision and recall. Precision measures the proportion of generated trajectories that are predicted to be non-OOD samples, while recall measures the comprehensiveness of covering the range of ground truth.

[0105] 2.2 Quantitative Experiments

[0106] The method of the present application is compared with some state-of-the-art methods on three datasets (ETH, UCY, and SDD). Given the first 8 time steps, each of which is 0.4 seconds, the future 12 time steps are predicted. The method of the present application is compared with baseline models in terms of ADE / FDE evaluation metrics, where the generative model is evaluated using K = 20 samples, and the results are shown in Table 1. Social-DMGAN achieves good results on all three datasets. On the ETH and HOTEL scenes, Social-DMGAN achieves state-of-the-art performance. Compared with four single-generator methods (SocialGAN, SoPhie, SocialWays, and Social-BiGAT), Social-DMGAN achieves the best results. Compared with the multi-generator architecture MG-GAN, Social-DMGAN adopts a more complex network structure, adds structured spatio-temporal graph encoding, and flexible generator selector, and achieves better performance. To evaluate the ability of the model to reduce OOD samples, precision and recall are used as evaluation metrics, and experiments are conducted on the Forking Paths dataset. The dataset provides multiple artificially annotated high-fidelity future trajectories for each pedestrian. From Table 2, it can be seen that the ability of Social-DMGAN to reduce OOD samples is similar to the state-of-the-art MG-GAN. Although Social-DMGAN does not achieve the best performance in terms of ADE / FDE, its ability to reduce OOD samples exceeds the model that achieves the best performance in terms of ADE / FDE. The present application believes that the reduction of these L2 distance-based indicators is at the cost of increasing the prediction variance and generating more OOD samples.

[0107] Table 1. Comparison results of the method of the present application and prior art methods

[0108]

[0109]

[0110] Table 2. Evaluation results of the Forking Paths Dataset

[0111] Precision↑ Recall↑ F1↑ Trajectron++ 0.38 0.96 0.54 MG-GAN 0.71 0.89 0.79 SocialVAE 0.50 0.96 0.66 Social-DMGAN(Ours) 0.70 0.92 0.80

[0112] 2.3 Qualitative Experiments

[0113] The trajectory prediction results of Social-DMGAN are visualized on the processed SDD dataset as follows: Figure 4 As shown, two trajectory prediction scenarios were selected. The future trajectories of different people exhibit different disconnected patterns in different scenarios. The trajectory spectrum clustering module can effectively constrain the number of generators based on the derived number of patterns. After constraining the number of generators, the model discards redundant generators to avoid parameter redundancy. The Social-DMGAN generator selector also learns a reasonable distribution for each generator.

[0114] 2.4 Ablation Experiment

[0115] The key components of the final model of this invention were reduced on the processed SDD dataset: multi-generator architecture, structured spatiotemporal graph, classifier, and generator selector. The results are shown in Table 3. The classifier encourages each generator to learn a specific manifold, and the structured spatiotemporal graph allows the model to explore more spatiotemporal information and better represent complex interactions. The generator selector can learn the distribution of generators, improving accuracy from 37% to 68%, while the number of generators is limited by the spectral clustering module. Using these four components combines all the advantages, resulting in improved performance on all metrics. The construction of the proposed spatiotemporal graph structure was investigated, and four types of graphs were constructed: (1) using images from the original dataset with only target pedestrians labeled; (2) using semantic images obtained from the original dataset by Deeplab v3 with only target pedestrians labeled; (3) using images from the original dataset with target pedestrians and other pedestrians labeled, where other pedestrians are labeled with the same vector; (4) using images from the original dataset with attention vectors computed for each pedestrian for annotation. These four structures are represented as STG1 to STG4, respectively. Experiments were conducted on the Zara2 scene of UCY, and the results are shown in Table 4. Due to the poor semantic meaning of the images in the original dataset, ADE / FDE performance was poor. STG4 uses different attention weights for annotation, making it easier for the spatiotemporal encoder to capture the spatiotemporal variation information of the graph structure, thus achieving the best performance.

[0116] Table 3. Ablation Experiment Results

[0117] M STG C S ADE FDE Precision Recall 1.24 2.10 0.36 0.41 √ 0.81 1.39 0.37 0.68 √ √ 0.80 1.36 0.72 0.90 √ √ 0.63 0.99 0.68 0.91 √ √ 0.75 1.13 0.35 0.70 √ √ √ 0.58 0.78 0.77 0.95 √ √ √ √ 0.54 0.73 0.78 0.95

[0118] In the table: (M) Multi-generator, (STG) Structured Spatiotemporal Graph, (C) Classifier, and (S) Generator Selector

[0119] Table 4. ADE / FDE results for four types of structured spatiotemporal graphs

[0120] STG1 STG2 STG3 STG4 ADE 0.31 0.33 0.29 0.26 FDE 0.63 0.69 0.61 0.53

[0121] The future trajectory of a pedestrian can be seen as a mixture of multiple disconnected modes, and the present invention learns to predict pedestrian trajectories based on the study of disconnected modes. Existing GAN models in the trajectory prediction task are mainly single generators, which leads to a large number of OOD samples, and existing multi-generator methods lack effective generator number restrictions. In order to overcome these problems, the present invention proposes Social-DMGAN, which uses a multi-generator architecture, where each generator learns one of the multiple disconnected modes in the trajectory distribution. The present invention introduces a generator selector to learn the prior distribution of multiple generators. By applying a spectral clustering module on the predicted trajectory, the number of generators is limited to improve the quality of prediction. In addition, the present invention proposes a structured spatiotemporal graph for trajectory prediction, which can provide additional spatiotemporal information for existing datasets. Experimental results show that Social-DMGAN has the ability to flexibly adjust the number of multiple generators and the prior, as well as the ability to prevent the generation of OOD samples, compared with the existing state-of-the-art methods.

[0122] The foregoing detailed description of the application has been presented for purposes of illustration and description. It is understood that numerous modifications and changes can be made by those skilled in the art without departing from the underlying inventive concepts disclosed herein. It is intended that the scope of the application be limited only by the appended claims.

Claims

1. A pedestrian trajectory prediction method based on multi-disconnected pattern learning, characterized in that, The method generates predicted trajectories using a socially disconnected pattern generative adversarial network model, where the model's input includes observed pedestrians in the scene. Trajectory coordinate data and scene images containing visual information I The specific steps for the model to generate the predicted trajectory include: Visual features are extracted using a physical scene encoder. The specific steps include: extracting visual features from scene images. I Extracting visual features Augment the original data, calculate velocity and express acceleration based on absolute coordinates; obtain pedestrian data. i At time step t pedestrian trajectory Representation of absolute coordinates, velocity, and acceleration; The social encoder is used to extract features from observed pedestrian trajectories. The specific steps include: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] The sequence is processed by a multilayer perceptron to obtain a high-dimensional embedding, and the embedded sequence is then input through an LSTM to obtain the pedestrian data. i At time step t pedestrian trajectory characteristics ; Based on the outputs of the physical scene encoder and the social encoder, physical scene attention features and social attention features are obtained through physical scene attention and social attention, respectively. The physical scene attention input is visual features extracted from the scene image. and the last observation time step of the social encoder LSTM Trajectory encoding hidden state The physical scene attention uses soft attention to obtain the context vector. ; The social attention is calculated based on the distance and azimuth between pedestrians, resulting in an attention score between any two adjacent pedestrians. This attention score is then used to evaluate the pedestrian trajectory features embedded by the LSTM social encoder. Weighting to obtain social characteristics Social characteristics as well as The pedestrian is obtained from the LSTM decoder based on the prior weights of the generator. Average hidden state As input, it is sent to the social attention network. In the middle, output the social attention vector ; A structured graph sequence based on attention annotation is established using social attention features, and the structured graph sequence is encoded using a spatiotemporal encoder to extract the transient changes of physical background and pedestrian movement; The concatenated features of physical scene attention, social attention, and spatiotemporal encoder output are input into a multi-generator architecture to output a predicted future trajectory of a pedestrian; in the multi-generator architecture, each generator receives the concatenated input and learns one of multiple unconnected patterns of trajectory distribution independently; A generator selector is used to learn the priors of multiple generators, and the generator with the larger prior value is selected to generate the future trajectory. During the prior learning process, the upper limit of the number of generators is updated using a spectral trajectory clustering module.

2. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 1, characterized in that, The specific steps for constructing the structured spatiotemporal graph include: From global scene image I Crops a local image centered on the target pedestrian; A new image with the same dimensions as the cropped image is constructed, but with an additional channel dimension. This additional channel dimension is then divided into a two-dimensional grid. The pedestrian in the image is at a specific time step. The position at that time is measured by the distance from each actual trajectory coordinate. Recent grid cell representation; pedestrian trajectory characteristics and social attention vector By piecing together the images, we can obtain the pedestrians. i At time step The labeled vector; In all grid cells containing pedestrians, the annotation vectors of the target pedestrian and other pedestrians within the local area are filled into the extra channel dimension.

3. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 1, characterized in that, The multi-generator architecture includes: There are 16 generators, all sharing the same network structure but using different weights; each generator includes an LSTM decoder, and the generator takes pedestrian trajectory features. Physical scene attention characteristics Social attention characteristics Spatiotemporal coding and random noise vector z The input to the generator is obtained by concatenating the received features. The generator generates pedestrians. i future trajectory ; The discriminator is obtained by concatenating pedestrian trajectory features, physical scene attention features, social attention features, and spatiotemporal codes from complete sequence encoding containing the observation and prediction sequences, resulting in the discriminator input. The predicted sequence can be the actual trajectory. Or generate trajectories for future time periods The discriminant score is calculated using a multilayer perceptron. The discriminator score The binary classification score is used to indicate whether a sample is real. The classifier receives the same input as the discriminator. Multilayer perceptron is used to calculate classification scores. The It is The classification score for each category is used to represent the probability that the generated trajectory belongs to each generator.

4. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 1, characterized in that, The model uses the EM algorithm to learn priors, and the specific training steps include: Step E: Generator selector training, learning the prior probabilities of each generator. To approximate the highest priority test ; M-step: Training a multi-classifier architecture GAN to learn future trajectories. Y Belongs to each generator probability Using approximate true posterior probability .

5. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 4, characterized in that, The training of the multi-classifier architecture GAN is as follows: For a multi-generator architecture, the goal of each generator is to maximize the error of the common discriminator, where the loss function for GAN training includes: Combating losses Using the original adversarial loss, consider The distribution of each generator is calculated by weighting each generator; Trajectory diversity loss The best multiple loss calculation method is used to encourage each generator to generate diverse trajectories; Classification loss Cross-entropy is used to calculate the loss between the generator index output by the classifier and the true generator index of the generated trajectory.

6. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 4, characterized in that, The generator selector training is specifically as follows: Calculate the model distribution using the learning results of the classifier. and true posterior The cross-entropy between the generator and the selector is used to train the generator selector, where the cross-entropy between the generator and the selector is used to train the generator selector. Come to approximate; Lower cross-entropy loss Based on this, add model distribution entropy And set a decay rate that varies with training time. To ensure smoothness; During training, the trajectory spectrum clustering module is used to add an additional upper limit to the number of generators. Generators exceeding the upper limit will be discarded and training will continue.

7. The pedestrian trajectory prediction method based on multi-disconnected pattern learning according to claim 6, characterized in that, The trajectory spectrum clustering module adaptively determines the optimal number of clusters, i.e., the number of selected generators, based on the predicted trajectory. Specific steps include: A similarity matrix is ​​obtained based on the calculated similarity of the predicted trajectories. K ; Normalize the similarity matrix to obtain the matrix ; Using matrices L Spectral information determines the range of cluster numbers; Based on L The information obtained from the eigenvalue decomposition is used to calculate the distortion score for each cluster number; The number of clusters with the best distortion score is selected as the final number of generators to choose from.

Images