Pedestrian trajectory prediction method based on agent interaction and environment context interaction

Abstract

The invention provides a pedestrian trajectory prediction method based on intelligent agent interaction and environment context interaction, and the method comprises the steps: obtaining a pedestrian trajectory data set, constructing a pedestrian trajectory prediction model (M2Tames) which comprises an encoder, an improved interaction module (IM) and a decoder, and training a pedestrian trajectory prediction model based on the pedestrian trajectory data set, and performing pedestrian trajectory prediction based on the trained pedestrian trajectory prediction model. According to the method, the historical trajectory features of pedestrians and the potential interaction relationship between the agents are accurately captured, the prediction accuracy is excellent, the adaptability is high, the method can effectively operate in a dynamically changing environment, and the method is expected to be widely applied to the fields of intelligent traffic management, public safety monitoring, automatic driving and the like, and has a wide application prospect in the fields of intelligent traffic management, public safety monitoring, automatic driving and the like. And reliable support is provided for real-time decision making.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method based on intelligent agent interaction and environmental context interaction. Background Art

[0002] The field of pedestrian trajectory prediction is a key task in the field of computer vision and autonomous driving, which is significantly challenging. The core of the research is not only to understand the motion characteristics of pedestrians themselves, but also to explore the complex interactions between pedestrians and between pedestrians and the environment. This interaction is not only multimodal, but also highly dynamic, because human behavior is influenced by factors such as self-awareness and social behavioral norms. Therefore, when predicting the motion trajectory of multi-agents, it is necessary to model / consider the influence of two key factors: time factors and social factors.

[0003] From a temporal perspective, the historical state of each agent will affect its future state. Therefore, it is necessary to model the temporal dependencies between the past states of each agent to predict its future direction. From a social perspective, the states of the surrounding agents and the surrounding environment information will also affect the decision of the central agent in real time. Therefore, it is also necessary to model the interactions between each agent and the complex changes in the surrounding environment to capture complex social information.

[0004] Early crowd trajectory prediction models mainly rely on manually designed energy functions to model the interactions between individuals. This approach requires extensive feature engineering and has limited effectiveness in capturing pedestrian interactions in busy scenes.

[0005] With the development of deep neural network technology, especially the application of recurrent neural network (RNN) in pedestrian trajectory prediction, new ideas are provided to solve this challenge. RNN-based methods can capture the movement trajectory of pedestrians by analyzing their hidden states and simulate the interaction between individuals by integrating the hidden states of surrounding pedestrians. This method improves the understanding of the dynamics of social behavior to some extent. As a supplement to the RNN method, social pooling technology considers the spatial relationship between pedestrians by treating the states of neighboring pedestrians equally and integrating them using pooling technology. In contrast, attention mechanism, as an advanced method, differentiates the importance of neighboring pedestrians through learned weight functions, providing a more flexible trajectory prediction method. Nevertheless, the existing attention mechanism still shows its limitations in fully simulating the complex interactions between pedestrians and between pedestrians and the environment.

[0006] Traditional attention mechanisms have difficulty handling complex interactions between multiple pedestrians. In scenes such as intersections, there may be multiple interaction modes between pedestrians, such as avoidance and intersection. Traditional attention mechanisms are difficult to flexibly adapt to these changes, and therefore perform poorly in capturing the relationship between multiple agents. In addition, traditional single-head attention mechanisms have difficulty processing multi-scale information. In pedestrian trajectory prediction, pedestrian movements may involve different spatiotemporal scales, and the single-head structure of the traditional mechanism cannot simultaneously pay attention to and adapt to the features of these different scales, thereby limiting the model's ability to process multi-scale information.

[0007] On the other hand, in many traditional models, the environment is regarded as a static background or simple obstacles, without in-depth consideration of the complex semantic information of the environment, such as road signs, traffic signals, etc. This information is crucial for understanding and predicting pedestrian behavior patterns in specific environments. For example, in urban street scenes, pedestrians are often influenced by the presence or absence of crosswalks when choosing where to cross the road. They choose to cross safely at locations with traffic signals rather than risk crossing streets with heavy traffic. How to effectively combine spatial information and pedestrian interaction information remains an open problem.

[0008] Therefore, it is necessary to design a pedestrian trajectory prediction method based on the interaction between intelligent agents and environmental context. Summary of the invention

[0009] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a pedestrian trajectory prediction method based on agent interaction and environmental context interaction.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] The present invention provides a pedestrian trajectory prediction method based on agent interaction and environmental context interaction, comprising:

[0012] Get the pedestrian trajectory dataset;

[0013] Constructing a pedestrian trajectory prediction model, wherein the pedestrian trajectory prediction model includes an encoder, an improved interaction module and a decoder;

[0014] Train the pedestrian trajectory prediction model based on the pedestrian trajectory dataset;

[0015] Pedestrian trajectory prediction is performed based on the trained pedestrian trajectory prediction model.

[0016] Preferably, the pedestrian trajectory dataset is a public real dataset provided by SDD and ETH / UCY.

[0017] Preferably, the encoder is a U-Net encoder, and the decoder is a U-Net decoder. The U-Net encoder is used to process pedestrian trajectory data, extract features, and input the features into an improved interaction module. The improved interaction module generates an interactive feature map and inputs it into the last layer of the U-Net decoder to generate fused features. The U-Net decoder is used to process the fused features and reconstruct the predicted pedestrian trajectory.

[0018] Preferably, the improved interaction module includes a feature initialization layer, a temporal encoder, a multi-head mask attention mechanism module M 2 Tea, Interactive Reasoning Module I 2 and aggregation modules;

[0019] The feature initialization layer is used to obtain initial features;

[0020] The time encoder is used to embed time-related information into M 2 In each input of Tea, make M 2 Tea can distinguish between past and future time points;

[0021] The multi-head mask attention mechanism module M 2 Tea is used to capture and learn the behavior patterns and trajectory characteristics possessed by the subject;

[0022] The interactive reasoning module I 2 Used to infer potential interaction categories between subjects;

[0023] The aggregation module is used to obtain the information from the interactive reasoning module I 2 The aggregate weights are learned among the interaction categories.

[0024] Preferably, after obtaining the pedestrian trajectory data set, the method further includes:

[0025] The pedestrian trajectory data set is preprocessed to obtain a preprocessed pedestrian trajectory data set.

[0026] Preferably, preprocessing the pedestrian trajectory dataset to obtain a preprocessed pedestrian trajectory dataset includes:

[0027] Performing data cleaning on the pedestrian trajectory data set to obtain a cleaned data set;

[0028] Performing behavior annotation on the cleaned data set to obtain an annotated data set with annotation information; the annotation information includes environmental features of walking, staying, turning and trajectory data; the environmental features include cities, shopping malls and campuses;

[0029] Performing scene classification on the labeled data set to obtain a classified data set;

[0030] Performing timestamp standardization on the classified data set to obtain a standardized data set;

[0031] Data enhancement is performed on the standardized data set to obtain a preprocessed pedestrian trajectory data set.

[0032] Preferably, the pedestrian trajectory dataset is cleaned to obtain a cleaned dataset, including:

[0033] The pedestrian trajectory data set is grouped according to a preset collection period to obtain multiple data groups;

[0034] Calculate the difference coefficient between the current data group and the previous data group in sequence;

[0035] Determine whether the value of the coefficient of difference is within a preset range;

[0036] If the value of the difference coefficient is not within the preset range, the corresponding data group is removed;

[0037] If the value of the difference coefficient is within the preset range, the corresponding data group will be retained until all data groups are traversed to obtain the cleaned data set.

[0038] Preferably, the difference coefficient calculation formula is:

[0039]

[0040] Among them, p X,Y is the coefficient of difference, cov(X,Y) represents the covariance between the current data set X and the previous data set Y, α X represents the mean of the current data set X, β Y Represents the mean of the previous data set Y.

[0041] Preferably, the data enhancement methods include: rotation, translation and scaling.

[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] The present invention provides a pedestrian trajectory prediction method based on agent interaction and environmental context interaction, the method comprising: obtaining a pedestrian trajectory data set, constructing a pedestrian trajectory prediction model (M 2 Tames), wherein the pedestrian trajectory prediction model includes an encoder, an improved interaction module (IM) and a decoder, the pedestrian trajectory prediction model is trained based on the pedestrian trajectory data set, and the pedestrian trajectory prediction is performed based on the trained pedestrian trajectory prediction model. The present invention has the following advantages:

[0044] 1. The pedestrian trajectory prediction model provided by the present invention comprehensively considers the motion characteristics of the subject and the interaction between the subject itself or with the environment, thereby improving the accuracy and adaptability of trajectory prediction;

[0045] 2. The multi-head masked attention mechanism module M2Tea used in this paper deeply characterizes the time and interaction features and extracts the dynamic interaction features of pedestrians to improve the model's ability to learn and simulate real scenes. 2 It is also designed to infer specific interactions between agents. 2 The goal is to infer the interaction relationships between agents based on the input motion data;

[0046] 3. This paper uses the public real data sets provided by SDD and ETH / UCY to test the proposed M 2 Tames conducted extensive experiments and found that M 2 Tames achieves high accuracy in predicting pedestrian trajectories in both long-term and short-term settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of multi-head mask attention;

[0050] Figure 3 Schematic diagram of the pedestrian trajectory prediction model structure. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The purpose of the present invention is to provide a pedestrian trajectory prediction method based on agent interaction and environmental context interaction. By accurately capturing the historical trajectory characteristics of pedestrians and the potential interaction relationship between agents, it not only performs well in prediction accuracy, but also has high adaptability and can operate effectively in a dynamically changing environment. It is expected to be widely used in the fields of intelligent traffic management, public safety monitoring, and autonomous driving, providing reliable support for real-time decision-making.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a pedestrian trajectory prediction method based on agent interaction and environmental context interaction, comprising:

[0055] Step 100: Obtain a pedestrian trajectory dataset;

[0056] Step 200: Construct a pedestrian trajectory prediction model (M 2 Tames), wherein the pedestrian trajectory prediction model includes an encoder, an improved interaction module (IM) and a decoder;

[0057] Step 300: training a pedestrian trajectory prediction model based on a pedestrian trajectory dataset;

[0058] Step 400: performing pedestrian trajectory prediction based on the trained pedestrian trajectory prediction model.

[0059] The pedestrian trajectory dataset is a public real dataset provided by SDD and ETH / UCY.

[0060] like Figure 2 As shown, the encoder is a U-Net encoder, and the decoder is a U-Net decoder. The functions of the U-Net encoder, the U-Net decoder, and the improved interaction module (IM) are introduced:

[0061] The U-Net encoder is used to process pedestrian trajectory data and extract features, which converts the trajectory data into a high-dimensional feature representation;

[0062] The features are input into the improved interaction module, IM adaptively aggregates useful neighbor features, generates a more accurate interactive feature map, and inputs it into the last layer of the U-Net encoder to fuse with the output of the encoder to generate fused features. The improved interaction module is specially designed to capture the interaction between pedestrians and the influence of environmental factors, so that the model can not only consider the historical motion information of individuals, but also dynamically understand and adapt to the behavior of surrounding pedestrians and changes in the environment, thereby improving the accuracy and applicability of trajectory prediction;

[0063] The U-Net decoder is used to process the fused features and reconstruct the predicted pedestrian trajectory;

[0064] It should be noted that the encoder and decoder adopt the U-Net structure, so that M 2 Tames can learn and interpret scene semantic information, enhancing its understanding of the spatial relationship between pedestrians and their surroundings.

[0065] like Figure 3 As shown, the improved interaction module includes a feature initialization layer, a time encoder, and a multi-head mask attention mechanism module M 2 Tea, Interactive Reasoning Module I 2 And aggregation module, where the multi-head mask attention mechanism module M 2 Tea thoroughly describes historical trajectories and potential interactions, while the interactive reasoning module I 2 The precise type of interaction is determined;

[0066] The feature initialization layer is used to obtain initial features, specifically:

[0067] The goal of the feature initialization layer is to obtain the initial features, which are recorded as The feature of the i-th agent is composed of D feature coordinates. There are N agents in total. Feature initialization mainly includes calculating the speed, direction, acceleration, position, relative position and other information of the agent, summarizing this information as the initial feature and initializing it.

[0068] The time encoder is used to embed time-related information into M 2 In each input of Tea, make M 2 Tea can distinguish between past and future time points. In time series problems such as trajectory prediction, the traditional attention mechanism with position information is not enough to reflect the temporal relationship between elements. Therefore, position encoding with timestamp is introduced. The temporal encoder includes embedding time-related codes into M 2 In each input of Tea, make M 2 Tea is able to distinguish between past and future time points, a feature that allows considering time series when processing temporal data;

[0069] The multi-head mask attention mechanism module M 2 Tea is used to capture and learn the behavior patterns and trajectory characteristics of the subject. In the past, models that used attention mechanisms to predict pedestrian trajectories often ignored the problem of loss of social identity information of the subject and its surroundings. In order to solve this problem, M 2 Tea, its principle is Figure 2 where the mask is assigned to 0 or 1 if the i-th query q i and the jth key k j If the agents belong to the same type, the mask is equal to 1, otherwise it is 0. This distinction makes M 2 Tea is able to focus on the motion characteristics of a single agent when analyzing its interactions with other agents or the environment. The masking mechanism ensures that M 2 Tea can accurately capture and learn the behavioral patterns and trajectory characteristics of the subjects, while understanding how the subjects interact with each other within the group. This method improves the comprehensiveness and accuracy of trajectory prediction.

[0070] The interactive reasoning module I 2 Used to infer the potential interaction categories between subjects. In complex pedestrian trajectory prediction scenarios, the interaction relationship between subjects is usually implicit and difficult to grasp directly. In order to effectively solve this problem, an innovative interaction reasoning module I is proposed. 2 ,The goal is to infer the potential interaction categories between subjects, various categories of social behavior patterns including avoidance, leadership, following, etc.,I 2 is designed to infer that agent i and The interaction relationship between them generates the interaction weight C ij ∈[0, 1]K, where K is the number of interaction categories, I 2 Calculate and take the relative position, velocity, acceleration, distance and direction of the agent to the past trajectory as the input of Φc, which is a multi-layer perception with softmax activation function;

[0071] The aggregation module is used to obtain the information from the interactive reasoning module I 2 Aggregation weights are learned from the obtained interaction categories, which are used to aggregate the interaction features of neighboring agents so that M 2 Tames is able to generate more accurate interaction features, which are crucial for predicting the future movement of the agent. le , k is used to simulate the interactive mode.

[0072] Preferably, after obtaining the pedestrian trajectory data set, the method further includes:

[0073] The pedestrian trajectory data set is preprocessed to obtain a preprocessed pedestrian trajectory data set.

[0074] Preferably, preprocessing the pedestrian trajectory dataset to obtain a preprocessed pedestrian trajectory dataset includes:

[0075] Performing data cleaning on the pedestrian trajectory data set to obtain a cleaned data set;

[0076] Performing behavior annotation on the cleaned data set to obtain an annotated data set with annotation information; the annotation information includes environmental features of walking, staying, turning and trajectory data; the environmental features include cities, shopping malls and campuses;

[0077] Performing scene classification on the labeled data set to obtain a classified data set;

[0078] Performing timestamp standardization on the classified data set to obtain a standardized data set;

[0079] Data enhancement is performed on the standardized data set to obtain a preprocessed pedestrian trajectory data set.

[0080] Preferably, the pedestrian trajectory dataset is cleaned to obtain a cleaned dataset, including:

[0081] The pedestrian trajectory data set is grouped according to a preset collection period to obtain multiple data groups;

[0082] Calculate the difference coefficient between the current data group and the previous data group in sequence;

[0083] Determine whether the value of the coefficient of difference is within a preset range;

[0084] If the value of the difference coefficient is not within the preset range, the corresponding data group is removed;

[0085] If the value of the difference coefficient is within the preset range, the corresponding data group will be retained until all data groups are traversed to obtain the cleaned data set.

[0086] Preferably, the difference coefficient calculation formula is:

[0087]

[0088] Among them, p X,Y is the coefficient of difference, cov(X,Y) represents the covariance between the current data set X and the previous data set Y, α X represents the mean of the current data set X, β Y Represents the mean of the previous data set Y.

[0089] Specifically, after obtaining the pedestrian trajectory data set, data cleaning is first performed. This process includes grouping the pedestrian trajectory data according to the preset collection period to form multiple data groups. Then, the difference coefficient between the current data group and the previous data group is calculated in turn to evaluate the stability of the data. Represent the mean of the current and previous data groups respectively. If the value of the difference coefficient exceeds the preset range, the data group is regarded as abnormal and removed; otherwise, the data group is retained, and finally a cleaned data set is obtained.

[0090] Furthermore, after data cleaning is completed, the cleaned data set is annotated. The annotation process involves identifying the main behavioral patterns in pedestrian trajectories, such as walking, staying, and turning, and adding corresponding annotation information to each trajectory point. In addition, the environmental features of the trajectory data are also extracted and annotated, such as scene information such as cities, shopping malls, and campuses. This process can be achieved through a combination of machine learning or manual annotation to ensure the accuracy and comprehensiveness of the annotation, thereby providing rich contextual information for subsequent analysis.

[0091] Furthermore, the labeled dataset is scene-classified to group the trajectory data in different environments, so as to facilitate the subsequent model training and prediction in specific scenarios. Scene classification can be clustered based on environmental characteristics to ensure the diversity and representativeness of the dataset. Subsequently, the classified dataset is timestamp-normalized to ensure that the time format of all trajectory data is consistent, which is convenient for subsequent time series analysis. The timestamp normalization process usually includes converting the time to a unified format (such as UTC time) and sorting it according to time intervals to ensure the temporal order of the data.

[0092] Preferably, the data enhancement methods include: rotation, translation and scaling.

[0093] Finally, this embodiment performs data enhancement on the standardized data set to increase the diversity of the data and the robustness of the model. Data enhancement methods include geometric transformations such as rotation, translation, and scaling, which are intended to simulate changes in pedestrian trajectories under different conditions. These transformations can be achieved by performing matrix operations or coordinate transformations on the trajectory data. For example, rotation can be achieved through a rotation matrix, while translation and scaling can be accomplished through simple addition and multiplication operations. Through these data enhancement techniques, the generated preprocessed pedestrian trajectory data set will be richer, able to better train the model, and improve the accuracy and reliability of pedestrian trajectory prediction.

[0094] The present invention provides an embodiment, respectively, on the ETH / UCY and SDD data sets M 2 Tames conducted a short-term evaluation and the results showed that M 2Tames outperforms the state-of-the-art model MSRL by 2.49% (ADE) and 8.77% (FDE) in the short term, and outperforms the optimal Y-Net by 6.89% (ADE) and 1.12% (FDE) in the long term. It also shows good performance on the ETH / UCY dataset. This proves that the method adopted by the present invention improves the accuracy and adaptability of predicting pedestrian trajectories.

[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0096] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A pedestrian trajectory prediction method based on agent interaction and environmental context interaction, characterized in that: include: Get the pedestrian trajectory dataset; Constructing a pedestrian trajectory prediction model, wherein the pedestrian trajectory prediction model includes an encoder, an improved interaction module and a decoder; Train the pedestrian trajectory prediction model based on the pedestrian trajectory dataset; Pedestrian trajectory prediction is performed based on the trained pedestrian trajectory prediction model.

2. The method according to claim 1, characterized in that The pedestrian trajectory dataset is a public real dataset provided by SDD and ETH / UCY.

3. The method according to claim 1, characterized in that The encoder is a U-Net encoder, and the decoder is a U-Net decoder. The U-Net encoder is used to process pedestrian trajectory data, extract features, and input the features into an improved interaction module. The improved interaction module generates an interactive feature map and inputs it into the last layer of the U-Net decoder to generate fused features. The U-Net decoder is used to process the fused features and reconstruct the predicted pedestrian trajectory.

4. The method according to claim 1, characterized in that The improved interaction module includes a feature initialization layer, a temporal encoder, and a multi-head mask attention mechanism module M 2 Tea, Interactive Reasoning Module I 2 and aggregation modules; The feature initialization layer is used to obtain initial features; The time encoder is used to embed time-related information into M 2 In each input of Tea, make M 2 Tea can distinguish between past and future time points; The multi-head mask attention mechanism module M 2 Tea is used to capture and learn the behavior patterns and trajectory characteristics possessed by the subject; The interactive reasoning module I 2 Used to infer potential interaction categories between subjects; The aggregation module is used to obtain the information from the interactive reasoning module I 2 The aggregate weights are learned among the interaction categories.

5. The method according to claim 1, characterized in that After obtaining the pedestrian trajectory dataset, it also includes: The pedestrian trajectory data set is preprocessed to obtain a preprocessed pedestrian trajectory data set.

6. The method according to claim 5, characterized in that Preprocessing the pedestrian trajectory dataset to obtain a preprocessed pedestrian trajectory dataset includes: Performing data cleaning on the pedestrian trajectory data set to obtain a cleaned data set; Performing behavior annotation on the cleaned data set to obtain an annotated data set with annotation information; the annotation information includes environmental features of walking, staying, turning and trajectory data; the environmental features include cities, shopping malls and campuses; Performing scene classification on the labeled data set to obtain a classified data set; Performing timestamp standardization on the classified data set to obtain a standardized data set; Data enhancement is performed on the standardized data set to obtain a preprocessed pedestrian trajectory data set.

7. The method according to claim 6, characterized in that The pedestrian trajectory dataset is cleaned to obtain a cleaned dataset, including: The pedestrian trajectory data set is grouped according to a preset collection period to obtain multiple data groups; Calculate the difference coefficient between the current data group and the previous data group in sequence; Determine whether the value of the coefficient of difference is within a preset range; If the value of the difference coefficient is not within the preset range, the corresponding data group is removed; If the value of the difference coefficient is within the preset range, the corresponding data group will be retained until all data groups are traversed to obtain the cleaned data set.

8. The method according to claim 7, characterized in that The coefficient of difference calculation formula is: Among them, p X,Y is the coefficient of difference, cov(X,Y) represents the covariance between the current data set X and the previous data set Y, α X represents the mean of the current data set X, β Y Represents the mean of the previous data set Y.

9. The method according to claim 6, characterized in that The data enhancement methods include: rotation, translation and scaling.

Images