A traffic participant trajectory prediction method in a complex heterogeneous environment

Abstract

The application relates to a traffic participant trajectory prediction method in a complex heterogeneous environment, comprising the following steps: acquiring traffic participant information in a complex heterogeneous environment; arranging and numbering traffic participant categories based on category information; respectively constructing a position graph, a speed graph, an acceleration graph and a category graph introducing expert experience; respectively capturing topological structure relationship and time dependence relationship of the position graph, the speed graph, the acceleration graph and the category graph to obtain position hidden states, speed hidden states, acceleration hidden states and category hidden states; based on the numbering, using an equation to classify the position hidden states, the speed hidden states, the acceleration hidden states and the category hidden states to obtain hidden state sets of different traffic participants; respectively decoding the hidden states of different traffic participants based on corresponding decoders to obtain future trajectory prediction of the traffic participants. Compared with the prior art, the application has the advantages of high trajectory prediction accuracy in a complex heterogeneous environment.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction, and in particular to a method for predicting the trajectories of traffic participants in complex and heterogeneous environments. Background Technology

[0002] With the development of intelligent vehicles and people's desire for convenient and safe travel, intelligent vehicles represent the main development direction of transportation. Therefore, it is inevitable that intelligent vehicles will operate in complex and heterogeneous environments comprised of pedestrians, bicycles, and human-driven cars. Accurate traffic participation trajectory prediction is crucial to ensuring the safe and reliable trajectory planning of intelligent vehicles. Therefore, researching a trajectory prediction method applicable to complex and heterogeneous environments is urgently needed.

[0003] Existing methods based on Long Short-Term Neural Networks (LSTNs) or Graph Convolutional Neural Networks (GCNNs), and their improvements, all study the impact of social interactions on future trajectories. They have achieved good results on pedestrian and vehicle datasets by incorporating obstacle avoidance based on field of view, an empirical approach that addresses the nonlinearity and asymmetry of interactions among similar traffic participants. However, they perform poorly in complex, heterogeneous environments involving pedestrians, bicycles, and human-driven cars. This is because they ignore the differences between different participant categories. For example: First, in the same traffic environment, different participants are subject to different social constraints, leading to different traffic information they focus on. Second, the different dynamic characteristics of different participants result in varying mobility and inconsistent motion state dependencies between time intervals. Finally, different participants place different weights on their own state information and traffic information. For instance, pedestrians are more concerned with traffic information because they are agile enough to stop immediately, unlike bicycles and cars.

[0004] However, existing technologies lack trajectory prediction methods that can individually consider the dynamic characteristics of different traffic participants, their different social attributes, and the differences in how they process environmental information. This makes it impossible to further improve the prediction accuracy of intelligent robots in complex and heterogeneous environments. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting the trajectory of traffic participants in complex and heterogeneous environments. This method considers the dynamic characteristics of different traffic participants, their different social attributes, and the differences in how they process environmental information, thereby improving the accuracy of trajectory prediction in complex and heterogeneous environments.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A method for predicting the trajectories of traffic participants in complex and heterogeneous environments includes the following steps:

[0008] Information on traffic participants in complex and heterogeneous environments is obtained, including location information, speed information, acceleration information, and category information.

[0009] Traffic participants are categorized and numbered based on category information;

[0010] Based on traffic participant information, location maps, velocity maps, acceleration maps, and category maps incorporating expert experience are constructed respectively;

[0011] Based on four parallel spatial relationship capture networks and temporal dependency capture networks, topological structure relationships and temporal dependencies are captured in the position graph, velocity graph, acceleration graph and category graph, respectively, to obtain the hidden state of position, hidden state of velocity, hidden state of acceleration and hidden state of category.

[0012] Based on the number, the hidden states of position, velocity, acceleration and category are classified by using the equation judgment to obtain the set of hidden states of different traffic participants.

[0013] A multi-mode decoder based on traffic participant categories is used to decode the hidden states of different traffic participants to obtain predictions of their future trajectories.

[0014] The categories of traffic participants include pedestrians, bicycles, and cars.

[0015] The specific steps for categorizing and numbering traffic participants based on category information are as follows: The category information is arranged sequentially into a category sequence, denoted as [C]. 1 C 2 …C i …C N ], C i C represents the category of the i-th traffic participant. i =1 represents the i-th traffic participant being a pedestrian, C i =2 represents that the i-th traffic participant is a bicycle, C i =3 represents that the i-th traffic participant is a car, and N represents the number of traffic participants; the category sequence corresponds to the number [1,2,…i…N].

[0016] The category graph incorporating expert experience uses the traffic participant risk coefficient as nodes and the risk level as edges. Expert experience correction coefficients related to traffic agent categories are introduced into the edges. The traffic participant risk coefficient depends on the traffic participant category, and the risk level is related to the relative distance between traffic participants and the angle between their speed vectors.

[0017] The expert experience correction factor related to the transportation agency category is:

[0018]

[0019] in, This represents the expert experience correction factor related to the category of transportation agency. It is the velocity vector of traffic participant i. This represents the relative position vector between traffic participant i and traffic participant j.

[0020] The method for constructing the category diagram is as follows:

[0021] G C =(U C E C )

[0022]

[0023]

[0024] Among them, G C Representative category diagram; U C For each node in the category graph, the risk coefficient of a traffic participant is represented, where c∈[1,2,3] corresponds to pedestrians, bicycles, and cars, respectively; E C The edges of the category graph represent the degree of danger. It is the velocity vector of traffic participant j.

[0025] The classification of position hidden states, velocity hidden states, acceleration hidden states, and category hidden states based on numbering and using equality judgment is specifically as follows: According to the arrangement of position hidden states, velocity hidden states, acceleration hidden states, and category hidden states output by the spatial relationship capture network and the temporal dependency capture network, the features of the arrangement are matched based on the numbering. When the numbers in different arrangements are the same, that is, when the equality judgment is satisfied, the features with the same number are classified. Based on the category hidden state information, the hidden state sets of pedestrians, bicycles, and cars are divided into sets.

[0026] The spatial relationship capture network employs a graph convolutional neural network.

[0027] The time dependency capture network is a temporal convolutional network, a long short-term neural network, a recurrent neural network, or a gated neural network.

[0028] The multi-mode decoder is constructed using long short-term neural networks, convolutional neural networks, temporal convolutional networks, recurrent neural networks, or gated neural networks.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) This invention uses a multi-layer directed graph convolutional neural network to encode the nonlinear relationship between the position, speed, acceleration and surrounding neighbor categories of traffic participants in social interaction. It can describe social interaction in a complex and heterogeneous environment composed of multiple categories of traffic participants in more detail. In particular, the constructed category graph is the key to expressing the social attributes of different neighbors, i.e. the degree of danger, and can provide necessary information for accurately predicting the future trajectory of participants to avoid different categories of traffic participants.

[0031] (2) The decoder used in this invention is a multi-mode decoder based on traffic participant categories, which is the key to expressing the characteristics of different traffic participants' differentiated processing of environmental information. Under the same social interaction and environmental information, since different traffic participants pay different attention to multiple environmental information and multiple social interactions, the multi-mode decoder proposed in this invention can effectively express the differentiated processing of information by different traffic participants and improve the trajectory prediction accuracy in complex heterogeneous environments composed of multiple categories of traffic participants. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a model architecture diagram of the method of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the working principle of the classifier of the present invention;

[0035] Figure 4 This is a pedestrian decoder architecture diagram in one embodiment of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0037] Accurate trajectory prediction is challenging in complex and heterogeneous traffic environments due to the time-varying and nonlinear social interactions among traffic participants, as well as the differences in how various participants process environmental information. This embodiment proposes a trajectory prediction model based on multi-layer directed graph convolutional neural networks and multimodal analysis to address these challenges. Figure 1As shown, the model consists of an encoder and a decoder. The encoder is built based on a multi-layer graph convolutional neural network. This embodiment innovatively introduces a category graph to consider the differentiated social attributes of different traffic participants. In the decoding part, this embodiment innovatively constructs a multi-modal decoder based on traffic participant categories, which endows the proposed model with the ability to express the characteristics of different traffic participants' differentiated processing of environmental information.

[0038] Based on the above model architecture, this embodiment specifically provides a method for predicting the trajectories of traffic participants in complex and heterogeneous environments, such as... Figure 2 As shown, it includes the following steps:

[0039] 1) Obtain traffic participant information in complex and heterogeneous environments. In this embodiment, traffic participant information includes location information, speed information, acceleration information, and category information. The categories of traffic participants are described in detail using three typical examples: pedestrians, bicycles, and cars.

[0040] 2) Arrange and number traffic participants according to category information.

[0041] Arrange the category information in order into a category sequence, denoted as [C]. 1 C 2 …C i …C N ], C i C represents the category of the i-th traffic participant. i =1 represents the i-th traffic participant being a pedestrian, C i =2 represents that the i-th traffic participant is a bicycle, C i =3 represents that the i-th traffic participant is a car, and N represents the number of traffic participants; the category sequence corresponds to the number [1,2,…i…N].

[0042] After the above processing, the numbers can be mapped one-to-one with the traffic participant categories, which can be used in subsequent steps to classify the characteristics of different traffic participant categories.

[0043] 3) Based on traffic participant information, construct location maps, velocity maps, acceleration maps, and category maps that incorporate expert experience.

[0044] 31) Location map G P

[0045] G P =(U P E P )

[0046]

[0047]

[0048]

[0049] In the formula: U P For nodes in the location map, the location is represented by the position of the traffic participant; E P The edges of the location graph represent the interaction relationships influenced by their relative positions. This represents the relative position vector between traffic participant i and traffic participant j; It is the expert-experienced correction coefficient for the relative positional relationship introduced; It is the velocity vector of traffic participant i, representing the direction of the center line of his field of view.

[0050] Obstacles outside the agent's field of view will not affect the future trajectory of traffic participants. Based on this, assuming the traffic participants' field of view is within 180°, and considering experience with traffic participants' avoidance, the closer their relative positions, the more pronounced their interaction and antagonism. Therefore, using the traffic participant's speed as the center line of the field of view, when the angle between the traffic participant's speed and their relative position is less than or equal to ±90°, it will affect the prediction of the traffic participant's trajectory, and the magnitude of this impact depends on both the speed and the relative position. Based on this, an expert-experienced correction coefficient for the aforementioned relative position relationship is determined.

[0051] 32) Velocity diagram G V

[0052] G V =(U V E V )

[0053]

[0054]

[0055]

[0056] In the formula: U V For nodes on the speed map, the speed is represented by the speed of traffic participants; E V The edges of the velocity graph represent the interaction relationships influenced by relative velocity. It is an expert-empirical correction coefficient for the relative velocity relationship.

[0057] Within the visible field of view, the interaction between traffic participants depends not only on their relative positions but also on their relative speeds and accelerations. For example, even if two participants are close together, the conflict between them is not intense because their relative speeds and accelerations are parallel. Conversely, if they are far apart but their relative speeds and accelerations are perpendicular, their interaction exhibits stronger conflict. Therefore, the impact of interaction between traffic participants gradually increases as their relative speeds and accelerations increase and their angle increases to perpendicularity. This allows us to determine the expert-empirical correction coefficients for the aforementioned relative speed relationship and the following relative acceleration relationship.

[0058] 33) Acceleration diagram G A

[0059] G A =(U A E A )

[0060]

[0061]

[0062]

[0063] In the formula: U A For nodes in the acceleration graph, the acceleration of traffic participants is used; E A The edges of the acceleration graph represent the interaction relationships under the influence of relative acceleration. It is the expert-empirical correction coefficient for the relative acceleration relationship introduced; It is the acceleration vector of traffic participant i; It is the acceleration vector of traffic participant j.

[0064] 34) Category diagram

[0065] In complex and heterogeneous environments, pedestrians, bicycles, and cars share roads. However, due to the inconsistencies in the size, mobility, and materials of these traffic participants, the risk levels differ among the various agents. This is the fundamental reason for the different social attributes among different traffic participants.

[0066] This embodiment introduces a category graph based on expert experience, using the traffic participant's risk coefficient as nodes and risk level as edges. Expert experience correction coefficients related to the traffic agent category are introduced into the edges. The traffic participant's risk coefficient depends on the traffic participant category, with nodes 1, 2, and 3 defined for pedestrians, bicycles, and cars, respectively. When there are identical traffic participants ahead, the smaller the relative distance, the higher the risk. Similarly, a perpendicular velocity vector angle indicates a higher risk than a parallel velocity vector angle. Therefore, the risk level is related to the relative distance between traffic participants and the angle between their velocity vectors.

[0067] Thus, the constructed category graph G is obtained. C for:

[0068] G C =(U C E C )

[0069]

[0070]

[0071]

[0072] Among them, G C Representative category diagram; U C For each node in the category graph, the risk coefficient of a traffic participant is represented, where c∈[1,2,3] corresponds to pedestrians, bicycles, and cars, respectively; E C The edges of the category graph represent the degree of danger. This represents the expert experience correction factor related to the category of transportation agency; It is the velocity vector of traffic participant i. It is the velocity vector of traffic participant j; This represents the relative position vector between traffic participant i and traffic participant j.

[0073] 4) Based on four parallel spatial relationship capture networks and temporal dependency capture networks, topological structure relationships and temporal dependencies are captured for the position graph, velocity graph, acceleration graph and category graph, respectively, to obtain the hidden states of position, velocity, acceleration and category.

[0074] The above process constructs a location graph, velocity graph, acceleration graph, and category graph; however, the topological relationships and temporal dependencies between nodes in the graphs are not obtained. This embodiment employs four parallel graph convolutional neural networks and a temporal convolutional network to capture the topological relationships and temporal dependencies between the four types of graph nodes, respectively. Considering the excellent topological learning ability of graph convolutional neural networks, other networks are not recommended. For capturing temporal dependencies, other sequential neural networks, such as long short-term neural networks, recurrent neural networks, or gated neural networks, can be used.

[0075] 5) Based on the number, use the equation to classify the hidden states of position, velocity, acceleration and category, and obtain the hidden state set of different traffic participants.

[0076] Specifically, such as Figure 3 As shown, the hidden state arrangements of the spatial relationship capture network, the temporal dependency capture network output, the hidden state arrangements of the position, velocity, acceleration and category are captured. The features of the arrangement are matched based on the number. When the numbers in different arrangements are the same, that is, when the equality judgment is satisfied, the features with the same number are classified. The hidden state sets of pedestrians, bicycles and cars are divided according to the category hidden state information.

[0077] 6) A multi-mode decoder based on traffic participant categories is used to decode the hidden states of different traffic participants to obtain the future trajectory predictions of traffic participants.

[0078] This embodiment employs a multi-modal decoder based on traffic participant categories, which is crucial for expressing the differentiated processing characteristics of environmental information by different traffic participants. Specifically, three parallel-computing long short-term neural networks (LSNs) are used to decode the latent states of pedestrians, bicycles, and cars, respectively. These latent states are calculated and classified by the encoding part. The three parallel-computing LSNs output the future trajectories of pedestrians, bicycles, and cars, respectively. Finally, the states are concatenated and plotted into a single graph to obtain the future trajectories of traffic participants around the agent. Other intelligent algorithms can also be used for the decoder, such as convolutional neural networks, temporal convolutional networks, recurrent neural networks, or gated neural networks. That is, this embodiment does not limit the specific model used in the decoder but focuses on constructing corresponding multi-modal decoders for the latent states of different traffic participants to identify their differentiated processing capabilities of environmental information.

[0079] This embodiment uses the simplest convolutional neural network as an example to illustrate the construction principle of a pedestrian decoder. The bicycle decoder and the car decoder have the same architecture, only the input features differ. Figure 4As shown, the input to the pedestrian decoder is the pedestrian hidden state, which includes the hidden states of position, velocity, acceleration, and category, obtained by the classifier. i The hidden layer, whose number of units can be increased depending on the computing platform's capabilities, contains learned weighted matrix coefficients and activation function coefficients. The output of the convolutional neural network is the pedestrian's future trajectory, including horizontal and vertical coordinates.

[0080] Based on the above method, this embodiment uses the complex heterogeneous traffic environment dataset provided by Baidu Apollo for performance testing. This dataset consists of 16,300 pedestrians, 5,500 bicycles, and 60,100 cars. This embodiment compares the performance of the present invention with two well-known baseline models (RNN-ED, Social LSTM) and three popular models in the field (Social Attention, TrafficPredict, VDRGCN) in complex heterogeneous environments.

[0081] During training, the time steps for observing and predicting trajectories were uniformly defined as 4 (2s) and 6 (3s), respectively. The batch size for each scene was set to 128. The model was trained using stochastic gradient descent (SGD) with an initial learning rate of 0.01 over 300 epochs. The learning rate was multiplied by 0.1 every 100 epochs to accelerate convergence. In this embodiment, the average error and terminal error are used to represent the model's performance, calculated as follows:

[0082]

[0083] In the formula: ADE represents the average error; This represents the predicted position of the i-th traffic participant at time t; Let N represent the actual position of the i-th traffic participant at time t; N represents the number of traffic participants; and P is the number of prediction steps.

[0084]

[0085] In the formula: PDE represents the terminal error; This represents the predicted position of the last step for the i-th traffic participant; This represents the actual position of the i-th traffic participant in the last step.

[0086] The test results obtained in this embodiment are shown in Table 1. The numbers in the table represent the average error / terminal error. The smaller the error, the higher the prediction accuracy. It is easy to observe from the test results that this invention performs excellently in complex heterogeneous environments, ranking first in all indicators. The average error and terminal error are reduced by 32% and 40% respectively compared to the second-place invention.

[0087] Table 1 Comparison of prediction results between the method of the present invention and existing technologies.

[0088]

[0089] In summary, the present invention performs well on complex and heterogeneous environmental datasets, demonstrating that the proposed solution has the ability to consider the dynamic characteristics of different traffic participants, their different social attributes, and the differences in how they process environmental information.

[0090] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for predicting the trajectories of traffic participants in complex and heterogeneous environments, characterized in that, Includes the following steps: Information on traffic participants in complex and heterogeneous environments is obtained, including location information, speed information, acceleration information, and category information. Traffic participants are categorized and numbered based on category information; Based on traffic participant information, location maps, velocity maps, acceleration maps, and category maps incorporating expert experience are constructed respectively; Based on four parallel spatial relationship capture networks and temporal dependency capture networks, topological structure relationships and temporal dependencies are captured in the position graph, velocity graph, acceleration graph and category graph, respectively, to obtain the hidden state of position, hidden state of velocity, hidden state of acceleration and hidden state of category. Based on the number, the hidden states of position, velocity, acceleration and category are classified by using the equation judgment to obtain the set of hidden states of different traffic participants. A multi-mode decoder based on traffic participant categories is used to decode the hidden states of different traffic participants to obtain predictions of their future trajectories. The method for constructing the location map is as follows: in: G p This is a location map. The nodes in the location map are represented by the locations of traffic participants; The edges of the location graph represent the interaction relationships influenced by their relative positions. Indicates traffic participants With traffic participants The relative position vectors between them; It is the expert-experienced correction coefficient for the relative positional relationship introduced; Traffic participants The velocity vector represents the direction of the center line of his field of view. N Indicates the number of traffic participants; The method for constructing the velocity map is as follows: In the formula: For velocity graphs, The nodes on the speed map are represented by the speed of traffic participants; The edges of the velocity graph represent the interaction relationships influenced by relative velocity. It is an expert-empirical correction coefficient for the relative velocity relationship. Traffic participants The velocity vector; The method for constructing the acceleration map is as follows: In the formula: For acceleration diagrams, For nodes in the acceleration graph, the acceleration of traffic participants is used as the representation. The edges of the acceleration graph represent the interaction relationships under the influence of relative acceleration. It is the expert-empirical correction coefficient for the relative acceleration relationship introduced; Traffic participants The acceleration vector; Traffic participants The acceleration vector; The method for constructing the category diagram is as follows: in, Representative category diagram; The nodes in the category graph represent the risk coefficient of traffic participants. These correspond to pedestrians, bicycles, and cars, respectively. The edges of the category graph represent the degree of danger. This represents the expert experience correction coefficient related to the category of transportation agency.

2. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 1, characterized in that, The categories of traffic participants include pedestrians, bicycles, and cars.

3. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 2, characterized in that, The specific steps for categorizing and numbering traffic participants based on category information are as follows: The category information is arranged sequentially into a category sequence, denoted as... , Representing the i Categories of traffic participants Representing the i The traffic participant is a pedestrian. Representing the i The traffic participant was a bicycle. Representing the i The traffic participants are cars, and the corresponding category sequence number is: .

4. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 2, characterized in that, The category graph incorporating expert experience uses the traffic participant risk coefficient as nodes and the risk level as edges. Expert experience correction coefficients related to traffic agent categories are introduced into the edges. The traffic participant risk coefficient depends on the traffic participant category, and the risk level is related to the relative distance between traffic participants and the angle between their speed vectors.

5. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 4, characterized in that, The expert experience correction factor related to the transportation agency category is: in, Traffic participants The velocity vector, Indicates traffic participants With traffic participants The relative position vectors between them.

6. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 1, characterized in that, The classification of position hidden states, velocity hidden states, acceleration hidden states, and category hidden states based on numbering and using equality judgment is specifically as follows: According to the arrangement of position hidden states, velocity hidden states, acceleration hidden states, and category hidden states output by the spatial relationship capture network and the temporal dependency capture network, the features of the arrangement are matched based on the numbering. When the numbers in different arrangements are the same, that is, when the equality judgment is satisfied, the features with the same number are classified. Based on the category hidden state information, the hidden state sets of pedestrians, bicycles, and cars are divided into sets.

7. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 1, characterized in that, The spatial relationship capture network employs a graph convolutional neural network.

8. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 1, characterized in that, The time dependency capture network is a temporal convolutional network, a long short-term neural network, a recurrent neural network, or a gated neural network.

9. The method for predicting the trajectory of traffic participants in a complex and heterogeneous environment according to claim 1, characterized in that, The multi-mode decoder is constructed using long short-term neural networks, convolutional neural networks, temporal convolutional networks, recurrent neural networks, or gated neural networks.

Images