A multi-modal crowd trajectory prediction method based on a generative adversarial network

Abstract

The application is a kind of multi-modal crowd trajectory prediction method based on generative adversarial network, and belongs to the field of multi-modal trajectory prediction.It is characterized by: obtaining crowd historical trajectory points, calculating the Euclidean distance between crowds, and dividing the crowd historical trajectory into three interaction levels.The processed crowd historical trajectory is input into the generator based on the generative adversarial network model to encode the time and sociality of crowd movement, and the identity attention is used to maintain the pedestrian identity information, and then the crowd future trajectory is generated.The generated future trajectory is discriminated by the discriminator combined with the real future trajectory, and the final output is the closest real prediction trajectory.The application can avoid losing the time and sociality of crowd movement, and can maintain the crowd identity information during the interaction process, thereby improving the accuracy of the trajectory prediction problem.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multi-modal trajectory prediction, and particularly relates to a crowd trajectory prediction network combining a Transformer and a GAN, a method for encoding the temporality and sociality of crowd movement and maintaining crowd identity information to perform multi-modal crowd trajectory prediction. BACKGROUND

[0002] Crowds may unconsciously go to restricted areas or even regulated areas due to their autonomous mobility. For example, they may go to indoor areas such as supermarkets and shopping malls whose carrying capacity is saturated, or go to scenic spots with high crowd flow during holidays, and so on. The inside of the area is already saturated, and the outside crowd is not aware of the internal environment, so if it is not timely dredged and managed, it may cause dangerous events such as stampedes.

[0003] Currently, observation and evaluation of crowds going to the above-mentioned areas are mostly based on manual observation. Schools, shopping malls, administrative agencies, communities, museums, and other places mostly use the method of obtaining real-time monitoring videos outside the restricted area through monitoring equipment, and security personnel observe whether the crowd is close to the edge of the restricted area to issue a warning, whether the crowd has entered the restricted area, and timely dispatch security forces to prevent. This method consumes a large amount of human resources, and is limited by the energy of security personnel, and it is impossible to maintain 24-hour dead-angle-free monitoring.

[0004] Using an intelligent monitoring system and a crowd trajectory prediction method can automatically issue a warning to the crowd approaching the restricted area in the predicted trajectory in advance, so that they can stay away. When a person who does not heed the warning enters the regulated area, the intelligent monitoring system can directly alarm and dispatch security forces for management. This method can free up human resources and more accurately control the crowd to avoid omissions caused by human resources. The intelligent monitoring system needs a crowd trajectory prediction method to produce accurate prediction results. SUMMARY

[0005] The purpose of the application is to provide a multi-modal crowd trajectory prediction method based on a generative adversarial network and combining a Transformer structure, which can solve the problem of poor time modeling of existing trajectory prediction models and inaccurate multi-modal trajectory prediction caused by crowd interaction.

[0006] Technical scheme: A multi-modal crowd trajectory prediction method based on a generative adversarial network, characterized in that the method is performed according to the following steps:

[0007] S1: obtaining crowd historical trajectory points from a video;

[0008] S2: calculating the Euclidean distance between different pedestrians in the crowd, and dividing the crowd trajectory into three interaction levels;

[0009] S3: Use a generative adversarial network-based model to predict crowd trajectories, use a time-social encoder to encode the temporality and sociality of crowd movement, and use identity attention to encode the identity information of the crowd;

[0010] S4: The predicted trajectories at the three levels are weighted and averaged to obtain the final prediction result. This result is then visualized as a frame image and converted back into a video output.

[0011] In step S2, the historical trajectory of the crowd is input into the interactive discrimination module to determine the social distance between pedestrians. This is divided into three levels, determined using the following Euclidean distance calculation formula:

[0012]

[0013] The parameter D in the formula ij This represents the Euclidean distance between pedestrian i and pedestrian j. and Let i and j represent the position coordinates of pedestrians i and j at the current time t, where t∈(0,t). obs ), t obs The observation time step is defined by the formula, which calculates the average Euclidean distance between pedestrians within that time step and serves as the classification standard.

[0014] In step S3, the processed historical trajectory of the crowd is input into the generator of the generative adversarial network. The Transformer encoder-decoder is used to generate the predicted trajectory. The following sine and cosine formulas are introduced to encode the temporality and sociality of the crowd movement using the same calculation method:

[0015]

[0016] In the formula, This represents the time / social code, where n is the number of pedestrians and N is the total number of people, n∈N. express The feature of the k-th dimension, t∈(0,t) obs ) represents the current time step, t obs For the observation time step, d input This indicates the dimension of the input data. Assuming the historical trajectory of the population is X, the result after time-social encoding is... Then for each element in X Its result of time and social coding The calculation formula is:

[0017]

[0018] w1, w2 are weights, which encode the temporality and sociality of crowd movement, and obtain crowd history trajectory data coded by time and sociality.

[0019] In step S3, the crowd identity information is maintained using identity attention, the crowd identity is encoded, and the following formula is introduced:

[0020]

[0021] In identity attention, the input query matrix Q, key matrix K and value matrix V are represented using crowd trajectory sequence. Wherein Q represents the crowd future trajectory Y, K, V represents the crowd history trajectory X.d. k The dimension of the key matrix is represented by D, and the element D ij The attention weight matrix between the i-th query and the j-th key is represented by, which is calculated as follows:

[0022]

[0023] Wherein Element-wise multiplication is represented by, The Q, K matrix multiplication of the current pedestrian is represented by, The Q, K matrix multiplication of other pedestrians is represented by, T∈(t obs ,t pred ). M is an attention mask, M=1 when the i-th query and the j-th key belong to the same pedestrian, otherwise 0. Using this way will not lose the original pedestrian identity information of the model.

[0024] Beneficial effects: The application discloses a multi-modal crowd trajectory prediction method based on a generative adversarial network, which obtains crowd history trajectory points, calculates the Euclidean distance between crowds, and divides the crowd history trajectory into three interaction levels. The processed crowd history trajectory is input into the generator based on the generative adversarial network model to encode the temporality and sociality of crowd movement, identity attention is used to maintain pedestrian identity information, and then the crowd future trajectory is generated. Using the generated future trajectory, the discriminator is used to combine the real future trajectory to determine its authenticity, and finally the prediction trajectory closest to the real one is output. The application can avoid losing the temporality and sociality of crowd movement, and can maintain the crowd identity information in the interaction process, thereby improving the accuracy of the trajectory prediction problem. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The crowd trajectory prediction flowchart of the application;

[0026] Figure 2 The crowd interaction discrimination principle diagram of the application;

[0027] Figure 3The model overall structure diagram of the application;

[0028] Figure 4 The time and social coding principle diagram of the application;

[0029] Figure 5 The Transformer encoder and decoder structure diagram of the application; DETAILED DESCRIPTION

[0030] The application will be described in detail below with the specific embodiments combined with the accompanying drawings.

[0031] As Figure 1 The flow chart of a multi-modal crowd trajectory prediction method based on a generative adversarial network provided by the application is shown, and the method is characterized by the following steps:

[0032] S1: obtaining crowd historical trajectory points from a video;

[0033] S2: calculating the Euclidean distance between different pedestrians in the crowd to divide the crowd trajectory into three interaction levels;

[0034] S3: using a model based on a generative adversarial network to predict the crowd trajectory, using a time-social encoder to encode the time and sociality of the crowd movement, and using identity attention to encode the identity information of the crowd;

[0035] S4: weighting and averaging the predicted three levels of trajectory to obtain the final prediction result, visualizing it to a frame image and restoring it to a video output.

[0036] The trajectory is formed from the crowd trajectory coordinates in the video, the crowd historical trajectory is input into the interaction discrimination module, the social distance between pedestrians is determined, and the crowd interaction discrimination principle diagram is shown in Figure 2 The following formula is used for determination:

[0037]

[0038] Assuming that the solid circles in the figure are crowds, the calculated Euclidean distance between pedestrians is first determined according to threshold 1 whether it has a similar motion trajectory, and after calculating all pedestrians, the pedestrian trajectories with similar motion trajectories are regarded as a pedestrian pair (such as the pedestrian in the dashed box in the figure), forming a new crowd historical trajectory set. The new historical trajectory set is input to determine again whether the pedestrian pair has a similar motion trend according to threshold 2, and it is regarded as a pedestrian group (such as the pedestrian pair in the dashed circle in the figure), forming a new crowd historical trajectory set.

[0039] The overall structure of the model based on a generative adversarial network is shown in Figure 3The three different crowd historical trajectories are input into the model, embedded through the fully connected layer, input into the generator through the time-social encoder, then through the encoder and decoder of the Transformer structure, and finally output through the prediction module; the generated trajectory and the real trajectory are input into the discriminator, and after passing through the discriminator decoder, the authenticity is judged through the prediction module. The prediction module is composed of two fully connected layers.

[0040] For the crowd trajectory prediction task using spatio-temporal sequence data, the data has the following properties:

[0041] (1) The historical trajectories of the same pedestrian at different times have different influences on the future trajectory. The closer the historical trajectory to the current time step, the greater the influence, and the farther the historical trajectory from the current time step, the smaller the influence. Therefore, it is necessary to encode the time dimension of the crowd movement historical trajectory and give different weights to the trajectory information at different times.

[0042] (2) For the crowd trajectory prediction task, each pedestrian cannot be considered alone, but the interaction between the crowd at the same time should be considered, so it is necessary to encode the social dimension of the crowd movement historical trajectory.

[0043] Since encoding the time and social interaction alone will lead to information loss, a time-social encoder is designed to encode the crowd historical trajectory considering both the time and social interaction dimensions. The time dimension is used to model the influence of the past movement information of the pedestrian on the future trajectory, and the social interaction dimension is used to model the complex interaction between the crowd. The crowd trajectory is encoded as a sequence, considering the historical trajectory of each person as the time dimension to model the influence of the past trajectory on the future, and considering the position information of different pedestrians interacting at each time step as the social interaction dimension to model the influence of the interaction between the pedestrians.

[0044] A time-social modeling example is shown in FIG. 1. Figure 4 In the figure, number 1 represents the time dimension modeling, and number 2 represents the social interaction dimension modeling at time step t. Assuming that the three curves in the figure represent the historical observed trajectories of three pedestrians, and there is interaction between the three people. First, the trajectory is encoded into an embedded representation to be input into the network, and each block represents the position coordinates of the pedestrian at a time step. Next, from the perspective of a single pedestrian, all its observed trajectories are input into the network as a sequence. From the social interaction dimension, the trajectory coordinate embedding representation of the three pedestrians at time step t is input into the network as a sequence.

[0045] Assuming that the crowd historical trajectory at time step t is

[0046]

[0047] where N is the total number of people. For all historical trajectories of people, we can represent as:

[0048]

[0049] t obs is the observation time step. Similarly, we can know that at time step t, for the future predicted trajectory, we can represent as:

[0050]

[0051] For all future trajectories, we can represent as

[0052]

[0053] Specifically, we can represent all historical trajectories as:

[0054]

[0055] Then for each element in , its temporal social encoder The calculation process can be represented as the following formula:

[0056]

[0057] where represents the kth dimension feature in . d input represents the dimension of the input. The output of the temporal encoder Each element in can be represented as:

[0058]

[0059] w1, w2 are weight matrices, thus obtaining the crowd historical trajectory data encoded by time and social.

[0060] Figure 5 is the structure diagram of the Transformer encoder and decoder. The Transformer encoder contains two layers: identity attention layer and feedforward network layer; the decoder contains three layers: masked multi-head attention, identity attention and feedforward network layer. Each layer is followed by residual connection and layer normalization.

[0061] The original Transformer in the crowd trajectory prediction process, all the trajectory information of people is regarded as the same, without distinguishing each trajectory coordinate belongs to the pedestrian, that is, assuming that the observed trajectory of people i and j at time step t is and , they have the same meaning for the network model, and all pedestrians are considered uniformly, which is not conducive to the extraction of the interaction characteristics between pedestrians. And when there is a trajectory interaction, it will not be able to distinguish the pedestrian to which the trajectory after the interaction belongs. Therefore, according to the principle of multi-head attention mechanism, identity attention is designed to encode the identity information between different pedestrians.

[0062] In the identity attention layer, the input query matrix Q, key matrix K and value matrix V are represented using the crowd trajectory sequence table. Wherein Q represents the future trajectory Y of the crowd, K and V represent the historical trajectory X of the crowd. Then the calculation formula of identity attention is:

[0063]

[0064] Wherein d k represents the dimension of the key matrix, and the elements D ij in D obs represent the attention weight matrix between the i-th query and the j-th key, and the calculation formula is:

[0065]

[0066] Wherein represents element-wise multiplication, represents the multiplication of the Q and K matrices of the current pedestrian, represents the multiplication of the Q and K matrices of other pedestrians, T∈(t obs ,t pred ). M is an attention mask, when the i-th query and the j-th key belong to the same pedestrian, M = 1, otherwise 0.

[0067] After identity attention encoding, the order of the Transformer network structure in Figure 5 is executed in sequence, and the generator finally generates the predicted trajectory, which is input to the discriminator to determine the authenticity, and the output is the trajectory predicted by the model based on the generative adversarial network. In the training process, the following evaluation indexes are used to determine the advantages and disadvantages: (1) average displacement error ADE: the average Euclidean distance difference between the coordinate sequence of the real trajectory and the predicted trajectory of each pedestrian at each time step, which can represent the average prediction performance, the smaller the better; (2) final displacement error FDE: the average Euclidean distance difference between the coordinate sequence of the real trajectory and the predicted trajectory of each pedestrian at the terminal position, only considering the final prediction accuracy, the smaller the better.

[0068] The three interaction level crowd future prediction trajectories are weighted and fused to generate a final trajectory prediction result, and converted into coordinate point form. The prediction trajectory is visualized through the matplotlib tool, all images are converted back to video, and output for use. The visualization results of the prediction results of the present application and the SoPhie model on the data set ETH are compared, and the prediction effect of the present application is closer to the real trajectory than the SoPhie model.

[0069] The above method of obtaining crowd trajectory points can be selected according to actual needs, such as selecting a faster single-stage crowd detection method for pursuing prediction speed, or selecting a two-stage crowd detection method for pursuing accuracy. The above examples show that the method of the present application for crowd multi-modal trajectory prediction based on a generative adversarial network model can more accurately predict the crowd trajectory, thereby providing strong support for the security work of public places and private places with restricted areas, liberating human resources, and developing towards an intelligent future society, and has a broad application prospect.

Claims

1. A multimodal crowd trajectory prediction method based on generative adversarial networks, characterized in that: This method is performed according to the following steps: S1: Obtain historical trajectory points of the crowd from the video; S2: Calculate the Euclidean distance between different pedestrians in the crowd and divide the crowd trajectory into three interaction levels; S3: Use a generative adversarial network-based model to predict crowd trajectories, use a time-social encoder to encode the temporality and sociality of crowd movement, and use identity attention to encode the identity information of the crowd; S4: The predicted trajectories at the three levels are weighted and averaged to obtain the final prediction result. This result is then visualized as a frame image and restored to video output. In step S3, the processed historical trajectory of the crowd is input into the generator of the generative adversarial network. The Transformer encoder-decoder is used to generate the predicted trajectory. The following sine and cosine formulas are introduced to encode the temporality and sociality of the crowd movement using the same calculation method: ； In the formula, Indicates time / social encoding, n For the number of pedestrians, N The total number of people, ; express Middle k Features in each dimension Indicates the current time step. For the observation time step, This indicates the dimension of the input data; assuming the historical trajectory of the population is... X The result of time-social coding is Then for X Each element in The result of time and social coding The calculation formula is: ； , Using these as weights, the temporal and social aspects of crowd movement are encoded, thus obtaining historical trajectory data of the crowd after time and social encoding; In step S3, identity attention is used to maintain the identity information of the crowd, and the crowd identity is encoded by introducing the following formula: ； In identity attention, the input query matrix Q Key matrix K and value matrix V， Represented using crowd trajectory sequences; where Q Representative population's future trajectory Y , K , V Historical trajectory of representative groups X ; This represents the dimension of the key matrix. D elements in Indicates the first i The first query and the first j The attention weight matrix between the keys is calculated using the following formula: ； in This indicates element-wise multiplication. Indicates the current pedestrian Q , K Matrix multiplication, Indicates other pedestrians Q , K Matrix multiplication, ; M For attention masking, when the first i The first query and the first j When all keys belong to the same pedestrian. M =1, otherwise 0; Using this method will prevent the model from losing the original pedestrian identity information.

2. The multimodal crowd trajectory prediction method based on generative adversarial networks according to claim 1, characterized in that: In step S2, the historical trajectory of the crowd is input into the interactive discrimination module to determine the social distance between pedestrians. This is divided into three levels, determined using the following calculation formula: ； The parameters in the formula, pedestrian i and pedestrians j The Euclidean distance between them and They represent pedestrians. i and j At the present moment t Location coordinates, , The observation time step is defined by the formula, which calculates the average Euclidean distance between pedestrians within the observation time step and serves as the classification standard.

Images