An interpretable pedestrian prediction method introducing expert experience in a complex interactive environment

Abstract

The application relates to an interpretable pedestrian prediction method introducing expert experience in a complex interactive environment, comprising the following steps: acquiring pedestrian information in an environment, wherein the pedestrian information comprises relative position, speed and acceleration information; determining expert experience coefficients of relative position relationships, relative speed relationships and relative acceleration relationships based on the pedestrian information; constructing interpretable position graphs, speed graphs and acceleration graphs based on the expert experience coefficients and the pedestrian information; performing topological structure coding on the position graphs, the speed graphs and the acceleration graphs based on a multilayer graph convolutional neural network to obtain hidden states containing spatial topological information; and performing space-time coding on the hidden states containing the spatial topological information, decoding by using a decoder to obtain predicted pedestrian trajectories. Compared with the prior art, the application has the advantages of fine consideration of interactions between pedestrians, accurate pedestrian trajectory prediction in a complex interactive environment and strong interpretability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pedestrian trajectory prediction, in particular to an interpretable pedestrian prediction method introducing expert experience in complex interaction environment. BACKGROUND

[0002] With the development of intelligent agents and the good prospect of bringing convenience to life, popularizing intelligent agents is the aspect of future development. In this process, accurate prediction of the position of pedestrians in the environment is a necessary condition for intelligent agents to plan to avoid collision. However, the interaction between pedestrians is highly nonlinear and time-varying, so the prediction of pedestrian trajectories in complex interaction environment is a research hotspot at this stage.

[0003] With the development of intelligent algorithms, compared with the initial way of predicting trajectories by pedestrian state, there is higher accuracy because the interaction between pedestrians is considered. Among them, the typical one is the social long short-term neural network constructed based on long short-term neural network and pooling layer. Although the latter research expands the network to differentiate the processing of environmental information by increasing the self-attention mechanism or convolutional neural network to enhance the nonlinear expression of social interaction of the network. However, they all capture social interaction through heuristic aggregation mechanism, which makes fine social interaction be missed. In addition, the aggregation of feature states is neither intuitive nor direct when simulating the interaction between agents, and its physical meaning is difficult to explain.

[0004] Recently, a trajectory prediction method based on graph convolutional neural network has been proposed, which has good spatial topology learning ability, which makes it improve the expression of social interaction compared with the previous work. Although some researchers have captured the influence of relative position, speed and acceleration of pedestrians in the environment on interaction through multi-layer graph convolutional neural network. However, they weaken the interaction between multiple pedestrians within the field of view by introducing whether within the field of view angle, which weakens the interaction between multiple pedestrians within the field of view. And, the crowded environment of pedestrians in complex interaction ring is inevitable. Therefore, a method that can individualize and describe the influence of the interaction between multiple pedestrians within the field of view of intelligent agents on their future trajectories is needed, which is a guarantee for intelligent agents to complete the task well in the crowded environment of pedestrians in complex interaction ring. But the prior art does not take this into account. SUMMARY

[0005] The purpose of the present application is to provide an interpretable pedestrian prediction method introducing expert experience in complex interaction environment, which individualizes and describes the influence of the interaction between multiple pedestrians within the field of view of intelligent agents on their future trajectories, and improves the accuracy of intelligent agents in predicting the future trajectories of pedestrians in the crowded environment of pedestrians in complex interaction ring.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] An interpretable pedestrian prediction method introducing expert experience in a complex interactive environment, comprising the following steps:

[0008] Obtaining pedestrian information in the environment, the pedestrian information including relative position, speed and acceleration information;

[0009] Determining expert experience coefficients of relative position relationship, relative speed relationship and relative acceleration relationship based on the pedestrian information respectively;

[0010] Based on the expert experience coefficients and the pedestrian information, constructing interpretable position graph, speed graph and acceleration graph respectively;

[0011] Based on the multi-layer graph convolutional neural network, topological structure coding is performed on the position graph, speed graph and acceleration graph to obtain hidden states containing spatial topological information;

[0012] After spatiotemporal coding of the hidden states containing spatial topological information, a decoder is used for decoding to obtain the predicted pedestrian trajectory.

[0013] The expert experience coefficient of the relative position relationship is determined according to the field of view angle of the pedestrian and the relative position of the pedestrian:

[0014]

[0015] Wherein, is the expert experience coefficient of the relative position relationship, is the speed vector of pedestrian i, represents the relative position vector between pedestrian i and pedestrian j.

[0016] The construction method of the interpretable position graph is:

[0017] G P =(U P ,E P )

[0018]

[0019]

[0020] Wherein, G P is the position graph; U P is the node of the position graph, represented by the position (x, y) of the pedestrian; E P is the edge of the position graph, representing the interaction relationship under the influence of the relative position; N is the number of pedestrians.

[0021] The expert experience coefficient of the relative speed relationship is determined according to the relative speed size and direction between pedestrians:

[0022]

[0023] wherein, is the expert experience coefficient of relative velocity relationship, respectively represent the velocity vector of pedestrians i, j.

[0024] The construction method of the interpretable velocity graph is:

[0025] G V = (U V , E V )

[0026]

[0027]

[0028] wherein, G V is the velocity graph; U V is the node of the velocity graph, represented by the velocity (v x , v y ) of the pedestrian; E V is the edge of the velocity graph, representing the interaction relationship under the influence of relative velocity; represents the relative position vector between pedestrians i and j; N is the number of pedestrians.

[0029] The expert experience coefficient of the relative acceleration relationship is determined according to the size and direction of the relative acceleration between pedestrians:

[0030]

[0031] wherein, is the expert experience coefficient of relative acceleration relationship, respectively represent the acceleration vector of pedestrians i, j.

[0032] The construction method of the interpretable acceleration graph is:

[0033]

[0034]

[0035] wherein, G A is the acceleration graph; U A is the node of the acceleration graph, represented by the acceleration (a x , a y ) of the pedestrian; E A is the edge of the acceleration graph, representing the interaction relationship under the influence of relative acceleration; represents the relative position vector between pedestrians i and j; N is the number of pedestrians.

[0036] The multi-layer graph convolutional neural network encodes the topological structure of the nonlinear relationship between nodes in the position graph, the speed graph and the acceleration graph, corrects the coefficients of the edges, obtains the hidden state containing the spatial topological information, and learns the influence of the interaction between multiple pedestrians on the future trajectory of the predicted pedestrian.

[0037] The spatio-temporal coding of the hidden state containing the spatial topological information is specifically: a time convolution network, a long short-term neural network, a recurrent neural network or a gated neural network is used to capture the time dependence between spatial states.

[0038] The decoder is a long short-term neural network, a convolutional neural network, a time convolution network, a recurrent neural network or a gated neural network.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] The expert's obstacle avoidance experience is introduced in the form of a vector product, so that the interference features can be selected from multiple information, and the interfering pedestrians can be selected from multiple interacting pedestrians, the interaction relationship between pedestrians is fully and finely considered, the accuracy of pedestrian trajectory prediction in a complex crowded pedestrian environment can be improved, and good interpretability is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The present application is a method flowchart. DETAILED DESCRIPTION

[0042] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and detailed implementation and specific operation processes are given, but the protection scope of the present application is not limited to the following examples.

[0043] In order to improve the accuracy of pedestrian trajectory prediction in a complex interactive environment and expand the use range of intelligent agents in a complex interactive pedestrian crowded environment, the present embodiment proposes an interpretable pedestrian prediction method introducing expert experience in a complex interactive environment, which is constructed based on an encoding-decoding architecture, as shown in Figure 1The detailed encoding of social interaction is the key to improving the accuracy of trajectory prediction. The encoding of social interaction can be divided into two parts: topological structure encoding and spatio-temporal encoding. In topological structure encoding, the relative position, velocity, and acceleration between pedestrians have different degrees of influence on future trajectories. Therefore, this embodiment adopts a multi-layer graph convolutional neural network that introduces expert experience to encode the topological structure in social interaction, including three parts of relative position, velocity, and acceleration graphs. The calculation method of the edges of each graph is to introduce the expert's obstacle avoidance experience through the vector product, which enables the present application to have the ability to select interference features from multiple information and select interference pedestrians from multiple interacting pedestrians. The graph convolutional neural network containing expert experience constructed in this way improves the prediction accuracy of the prediction model in crowded pedestrian environments and the interpretability of the model. The decoding part uses a decoder such as a long short-term neural network to decode to obtain the predicted pedestrian trajectory.

[0044] Specifically, the present embodiment provides an interpretable pedestrian prediction method with expert experience in a complex interactive environment, including the following steps:

[0045] An interpretable pedestrian prediction method with expert experience in a complex interactive environment includes the following steps:

[0046] 1) Obtain pedestrian information in the environment, specifically including relative position, velocity, and acceleration information of the pedestrians.

[0047] 2) Determine the expert experience coefficients of the relative position relationship, the relative velocity relationship, and the relative acceleration relationship based on the pedestrian information.

[0048] 21) Determine the expert experience coefficient of the relative position relationship

[0049] Obstacles outside the agent's perspective will not affect the future trajectory of the pedestrian. Based on this, it is assumed that the pedestrian's perspective range is within 180°, and through the pedestrian's avoidance experience, it is known that the closer the relative position between pedestrians, the more obvious the interactive confrontation between pedestrians. Therefore, taking the pedestrian speed as the center line of the field of view angle, when the angle between the pedestrian speed and the relative position of the pedestrian is less than or equal to ±90°, it will affect the pedestrian trajectory prediction, and the size of the influence depends on the size of the pedestrian speed and the relative position.

[0050] Therefore, the expert experience coefficient of the relative position relationship can be determined according to the pedestrian field of view angle and the relative position of the pedestrian:

[0051]

[0052] wherein, is the expert experience coefficient of the relative position relationship, is the velocity vector of pedestrian i, represents the relative position vector between pedestrian i and pedestrian j.

[0053] 22) Expert experience coefficient of relative velocity relationship

[0054] In the visible view, the interaction between pedestrians is related to not only the relative position, but also the relative velocity and relative acceleration between them. For example, although the distance between two pedestrians is close, the conflict between them is not intense because the relative velocity and relative acceleration are parallel. On the contrary, if the distance between two pedestrians is far, but the relative velocity and relative acceleration are perpendicular, the interaction between them shows a strong conflict. Therefore, the interaction between pedestrians gradually increases with the increase of the relative velocity and acceleration and the increase of the angle between them to the vertical.

[0055] Therefore, the expert experience coefficient of relative velocity relationship is determined according to the size and direction of the relative velocity between pedestrians:

[0056]

[0057] wherein, is the expert experience coefficient of relative velocity relationship, respectively represents the velocity vector of pedestrian i and j.

[0058] 23) Expert experience coefficient of relative acceleration relationship

[0059] Similar to the analysis in 22), the expert experience coefficient of relative acceleration relationship is determined according to the size and direction of the relative acceleration between pedestrians:

[0060]

[0061] wherein, is the expert experience coefficient of relative acceleration relationship, respectively represents the acceleration vector of pedestrian i and j.

[0062] 3) Based on the expert experience coefficient and the pedestrian information, the interpretable position graph, velocity graph and acceleration graph are respectively constructed.

[0063] 31) Construction of interpretable position graph G P

[0064] G P = (U P , E P )

[0065]

[0066]

[0067] wherein, UP is the node of the position graph, represented by the position (x, y) of the pedestrian; E P is the edge of the position graph, representing the interaction relationship under the influence of relative position; N is the number of pedestrians.

[0068] According to the analysis in 22), the interaction relationship between pedestrians has a great relationship with the relative position, therefore, in the construction of the edge of the graph, not only the expert experience coefficient is considered, but also the relative position is considered, and since the closer the relative position is, the greater the influence on the future trajectory is, therefore, the influence of the relative position on the future trajectory is described more accurately in the form of sub-items. The same factor is also considered in the construction of the speed graph and the acceleration graph in the following 32) and 33).

[0069] 32) Construction of an interpretable speed graph G V

[0070] G V = (U V ,E V )

[0071]

[0072]

[0073] wherein, U V is the node of the speed graph, represented by the speed (v x , v y ) of the pedestrian; E v is the edge of the speed graph, representing the interaction relationship under the influence of relative speed.

[0074] 33) Construction of an interpretable acceleration graph G A

[0075]

[0076]

[0077] wherein, U A is the node of the acceleration graph, represented by the acceleration (a x , a y ) of the pedestrian; E A is the edge of the acceleration graph, representing the interaction relationship under the influence of relative acceleration.

[0078] 4) Topological structure coding of the position graph, the speed graph and the acceleration graph based on the multi-layer graph convolutional neural network, to obtain the hidden state containing the spatial topological information.

[0079] ​For the nonlinear relationship between the nodes in the constructed position graph, velocity graph, and acceleration graph, the embodiment adopts a multi-layer graph convolutional neural network to further correct the coefficients of the edges. This correction process uses the good topological structure learning ability of the graph convolutional neural network to obtain the influence of the interaction between multiple pedestrians on the predicted pedestrian's future trajectory as much as possible. This expands the ability of the node (representing a pedestrian) to view the global node (representing all pedestrians within the visual angle), which helps to improve the prediction accuracy of the model. The input of the multi-layer graph convolutional neural network is the position graph, velocity graph, and acceleration graph containing expert experience constructed in step 3). Since the graph convolutional neural network is a relatively mature solution, to avoid obscuring the purpose of the present application, no further description is provided here.

[0080] 5) After spatio-temporal encoding of the hidden state containing spatial topological information, a decoder is used to decode to obtain the predicted pedestrian trajectory.

[0081] The planning of the future trajectory of a pedestrian is related not only to the spatial state but also to the temporal state. After all, the state of a pedestrian cannot change abruptly. The embodiment preferentially uses a temporal convolutional network to capture the temporal dependence between spatial states because it has the advantages of small volume and fast calculation speed. The input of the temporal convolutional network is the hidden state containing spatial topological information output in step 4). Since the temporal convolutional network is a relatively mature solution, no further description is provided here. In other embodiments, other sequential neural networks can also be used, such as long short-term neural networks, recurrent neural networks, or gated neural networks.

[0082] The decoding part of the embodiment uses a long short-term neural network to decode to obtain the predicted pedestrian trajectory because the long short-term neural network has the ability to inherit and discard the expression of state information during transmission in the time direction, which helps to improve the prediction ability of the model. In other embodiments, other intelligent algorithms can also be used, such as convolutional neural networks, temporal convolutional networks, recurrent neural networks, or gated neural networks.

[0083] According to the above method, the embodiment selects two widely used pedestrian data sets, ETH and UCY, for verification. These two data sets contain 5 scenes, ETH, HOTEL, UNIV, ZARA1, and ZARA2, which are complex social data sets composed of 1536 pedestrians. The embodiment compares the performance of the present application with two well-known baseline models (Social-LSTM, SS-LSTM) and 8 popular models in the field (Social Attention, SoPhie, SR-LSTM, STSGN, STGAT, Social-STGCNN, GTPPO, VDRGCN).

[0084] In the training process, the time steps of the observed trajectory and the predicted trajectory are uniformly defined as 8 (3.2s) and 12 (4.8s). The batch size of each scene is set to 128. The model is trained using a stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01 for 300 epochs. The learning rate is multiplied by 0.1 every 100 epochs to accelerate the convergence of the loss. The average error and the terminal error are used to represent the performance of the model, and the calculation formulas are as follows:

[0085]

[0086] In the formula, ADE represents the average error; represents the predicted position of the ith pedestrian at time t; represents the actual position of the ith pedestrian at time t; N represents the number of pedestrians; and P is the prediction step.

[0087]

[0088] In the formula, FDE represents the terminal error; represents the predicted position of the ith pedestrian at the last step; represents the actual position of the ith pedestrian at the last step; and N represents the number of pedestrians.

[0089] The test results obtained by the embodiment are shown in Table 1, where the numbers represent the average error / terminal error, and the smaller the error, the higher the prediction accuracy.

[0090] Table 1 Comparison of prediction results of the method of the present application and the prior art

[0091]

[0092] From the test results, it can be easily observed that the performance score of the present application in the enumerated data set has only one result ranked second (i.e., the average error result in the ETH scene), and the others are all first. In addition, the average performance score of the present application is improved by 43% in ADE and 44% in FDE compared to the second place. Therefore, it is proved that the present application can realize high-precision pedestrian trajectory prediction in a complex interactive environment.

[0093] The above describes the preferred embodiments of the present application in detail. It should be understood that those skilled in the art can make many modifications and changes without creative labor according to the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning, or limited experiments by those skilled in the art on the basis of the prior art according to the concept of the present application shall be within the scope of protection determined by the claims.

Claims

1. An interpretable pedestrian prediction method that incorporates expert experience in complex interactive environments, characterized in that, Includes the following steps: Acquire pedestrian information in the environment, including relative position, velocity, and acceleration information; Expert experience coefficients for determining relative position, relative velocity, and relative acceleration relationships based on pedestrian information; Interpretable location maps, velocity maps, and acceleration maps are constructed based on expert experience coefficients and pedestrian information, respectively. Topological structure encoding of position, velocity and acceleration maps is performed using a multi-layer graph convolutional neural network to obtain implicit states containing spatial topological information. After spatiotemporally encoding the implicit state containing spatial topological information, a decoder is used to decode it to obtain the predicted pedestrian trajectory. The expert experience coefficients for the relative positional relationships are determined based on the pedestrian's field of view and relative position. in, This represents the expert experience coefficient for the relative positional relationship. pedestrian The velocity vector, pedestrian with pedestrians The relative position vectors between them; The expert-empirical coefficients for the relative velocity relationship are determined based on the magnitude and direction of the relative velocities between pedestrians: in, The expert-empirical coefficients for the relative velocity relationship. , They represent pedestrians. , j The velocity vector; The expert-empirical coefficients for the relative acceleration relationship are determined based on the magnitude and direction of the relative acceleration between pedestrians: in, For the expert empirical coefficients of the relative acceleration relationship, , They represent pedestrians. , j The acceleration vector.

2. The interpretable pedestrian prediction method incorporating expert experience in a complex interactive environment according to claim 1, characterized in that, The method for constructing the interpretable location map is as follows: in, Location map; For the nodes in the location map, use the pedestrian's location ( x , y )express; The edges of the location graph represent the interaction relationships influenced by their relative positions. N This represents the number of pedestrians.

3. The interpretable pedestrian prediction method incorporating expert experience in a complex interactive environment according to claim 1, characterized in that, The method for constructing the interpretable velocity map is as follows: in, This is a velocity graph; For the nodes of the velocity map, use the speed of pedestrians ( v x , v y )express; The edges of the velocity graph represent the interaction relationships influenced by relative velocity. pedestrian with pedestrians The relative position vectors between them; N This represents the number of pedestrians.

4. The interpretable pedestrian prediction method incorporating expert experience in complex interactive environments according to claim 1, characterized in that, The method for constructing the interpretable acceleration map is as follows: in, This is an acceleration graph; For the nodes of the acceleration graph, use the pedestrian's acceleration ( a x , a y )express; The edges of the acceleration graph represent the interaction relationships caused by the influence of relative acceleration. pedestrian with pedestrians The relative position vectors between them; N This represents the number of pedestrians.

5. The interpretable pedestrian prediction method incorporating expert experience in a complex interactive environment according to claim 1, characterized in that, The multi-layer graph convolutional neural network performs topological structure encoding on the nonlinear relationships of mutual influence between nodes in the position graph, velocity graph, and acceleration graph, corrects the coefficients of the edges, obtains the implicit state containing spatial topological information, and learns the influence of the interaction between multiple pedestrians on the future trajectory of the predicted pedestrian.

6. The interpretable pedestrian prediction method incorporating expert experience in a complex interactive environment according to claim 1, characterized in that, The spatiotemporal encoding of the implicit states containing spatial topological information specifically involves using temporal convolutional networks, long short-term neural networks, recurrent neural networks, or gated neural networks to capture the temporal dependencies between spatial states.

7. The interpretable pedestrian prediction method incorporating expert experience in a complex interactive environment according to claim 1, characterized in that, The decoder is a long short-term neural network, a convolutional neural network, a temporal convolutional network, a recurrent neural network, or a gated neural network.

Images