Pedestrian trajectory prediction method and automatic driving method of vehicle

Abstract

The application provides a pedestrian trajectory prediction method and an automatic driving method of a vehicle. First, starting from N pedestrian graph structure trajectories, a grouping distribution module is used to estimate grouping information, then irrelevant nodes are shielded and a pedestrian group pool is generated to generate intra-group or inter-group interaction graphs. The weight sharing trajectory prediction model adopts three types of graphs and captures group-aware social interaction. Then, a group pool operator is applied to encode the agent features from the group features, and then input the group integration module to estimate the probability distribution of future trajectory prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method and an automatic driving method of a vehicle. BACKGROUND

[0002] As the main participant in the traffic scene, the reasonable inference and prediction of the future motion trajectory of a pedestrian is of great importance to automatic driving and road safety. In the traffic scene, the motion trajectory of a pedestrian will be influenced not only by its own intention but also by surrounding pedestrians and vehicles. Pedestrian trajectory prediction is also crucial to the computer vision and robotics industry. Pedestrian trajectory prediction is challenging because: (1) the interaction between people is multi-modal and difficult to capture, for example, strangers will try to avoid close contact with others, while companions tend to walk in groups; (2) complex temporal prediction and spatial interaction between people.

[0003] The pioneering work of the prior art pedestrian trajectory prediction simulates group motion by assigning additional hand-crafted terms as energy potentials. The current mainstream work models social interaction through convolutional neural networks (CNN) and graph neural networks (GNN). However, pedestrian trajectory prediction is still a challenging problem due to the complexity of simultaneously learning individual and group behaviors implicitly. Recent studies multiply the sum of the hidden states of LSTM and other agent states by a binary tuple indicator function to show encoding group-consistent behavior, but there is a key problem in capturing group interaction. Although this method can conceptually capture group motion behavior, it is difficult for deep learning-based methods to represent because the number of edges of individual interaction is too large. Moreover, this problem becomes increasingly difficult to implement in crowded environments. SUMMARY

[0004] In order to overcome the above technical defects, the purpose of the present application is to provide a pedestrian trajectory prediction method for predicting the motion of pedestrians in a complex and crowded environment, so that the vehicle performing automatic driving can plan a reasonable route to avoid pedestrians.

[0005] The present application discloses a pedestrian trajectory prediction method, comprising the following steps: acquiring a pedestrian graph, the pedestrian graph g ped = (v ped , ε ped ) indicates a set of pedestrian nodes v ped = {X n | n∈[1,...N]} and its pairwise social interaction g ped = {e i,jEdges on |i,j∈[1,...,N]}; Based on the pedestrian graph, each pedestrian is assigned to the most probable behavior group by group index to obtain pedestrian grouping information; The pedestrian graph is converted into intra-group and inter-group interaction graphs based on the pedestrian grouping information through a direct-group backpropagation process; Hierarchical representation is enforced from the input pedestrian graph and separated interactions by weight sharing with the baseline trajectory predictor; Agent features are encoded from group features through a U-NET network structure with pooling layers, and the number of nodes is reduced through the pooling layers in U-NET to obtain higher-level grouping features; The agent features are recovered through depooling operations to obtain a group integration module, and a probability graph of socially acceptable future trajectories is obtained through the group integration module; The agent includes pedestrians, and the agent features include pedestrian trajectories.

[0006] Preferably, the group index includes estimating the group information to which a pedestrian belongs through the pedestrian's social interactions.

[0007] Preferably, estimating the group information of pedestrians through their social interactions includes: obtaining the historical trajectory of each pedestrian; measuring the feature similarity between all pedestrian pairs based on the distance of the historical trajectories; defining the two pedestrians with the highest similarity as a pair of pedestrians; and then a pedestrian and several pairs of that pedestrian are considered to be in the same group of pedestrians.

[0008] Preferably, the distance matrix D of the historical trajectories of the paired pedestrians is defined as: D i,j =||F φ (X i )-F φ (X j )|fori,j∈[1,...,N]; γ={pair(i,j)|i,j∈[1,...,N],i≠j,D i,j ≤π};where, F φ It is a learnable convolutional layer, and π is a learnable threshold parameter.

[0009] Preferably, the group index set G is: G = G k |G k =U (i,j)∈γ {i,j},G a ∩G b =φfora≠b; where, G k Let k denote the k-th group, which is the union of every pair (i,j).

[0010] Preferably, the step of converting the pedestrian graph into an intra-group and inter-group interaction graph based on the pedestrian grouping information through the back propagation process of the direct group further includes: performing group pooling and unpooling operations by shielding the edges of the pedestrian graph as unrelated group members, and representing a group with multiple members as a graph node.

[0011] Preferably, the historical trajectory of the pedestrian n [1,..., N] can be represented as X n , where is the two-dimensional coordinate of the pedestrian n at a specific time t.

[0012] The application also discloses an automatic driving method of a vehicle, which adopts the pedestrian trajectory prediction method to predict the pedestrian trajectory, thereby planning the driving route by avoiding pedestrians.

[0013] After adopting the above technical scheme, compared with the prior art, the following beneficial effects are achieved:

[0014] 1. The application reduces the complexity of trajectory prediction caused by different social behaviors of individuals by modeling group interaction; alleviates inherent scenario bias by considering a large number of unseen pedestrian graph nodes between training and testing environments; provides a graph-enhanced effect by grouping pedestrian nodes; and the group feature represents group members in the input scene as a single node, making the pedestrian graph simpler.

[0015] 2. The application achieves consistent performance improvement on all baseline models, in addition, the group-aware prediction also reduces the collision rate between Agent interaction graphs, and shows similar motion patterns to reality by well capturing the motion behavior of the group. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the pedestrian trajectory prediction method provided by the application is shown in the figure.

[0017] Figure 2 The learning group representation of the GP-pedestrian trajectory prediction provided by the application is shown in the figure. DETAILED DESCRIPTION

[0018] The advantages of the application will be further described below in combination with the specific embodiments and the accompanying drawings.

[0019] The exemplary embodiments will be described in detail herein below with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0020] The terminology used in the disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in the description of the disclosure and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0021] It is to be understood that the terms first, second, third, etc. can be adopted herein only to describe various information and should not be limited to the terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the disclosure. Depending on the context, the word “if” as used herein can be interpreted as “when” or “upon” or “in response to determining”.

[0022] In the description of the present application, it should be understood that the terms “longitudinal”, “transverse”, “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0023] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms “mounting”, “connection”, “connection” should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between two elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0024] In the subsequent description, the suffix such as “module”, “component” or “unit” used to represent elements is only for the convenience of the description of the present application, and has no specific meaning in itself. Therefore, “module” and “component” can be used interchangeably.

[0025] Referring to the accompanying Figure 1 The present application discloses a pedestrian trajectory prediction method, which can be defined as a sequential reasoning task for observing all agents in a scene. Assuming that N is the number of pedestrians in a scene, the historical trajectory of each pedestrian n∈[1,...,N] can be represented as X n , where is the two-dimensional coordinate of pedestrian n at time t. Similarly, the ground truth future trajectory of pedestrian n can be defined as Social interactions are modeled based on the past trajectories of other pedestrians. Generally, the graph of pedestrians g ped = (v ped , e ped ) denotes a set of pedestrian nodes v ped = {X n | n e [1,...,N]} and edges e ped = {e i,j | i,j e [1,...,N]} on pairs of them. The pedestrian trajectory prediction process predicts the future sequence of pedestrians based on their past trajectories and social interactions: Y = F0(X, g ped ). Where Y = {Y n | n e [1,...,N]} represents the estimated future trajectories of all pedestrians in the scene, and F0 represents the trajectory generation network.

[0026] The key idea of the present application is to model individual relationships and group relationships as a graph representation, in order to solve the problem of pedestrian trajectory prediction in complex, group environments, a new prediction algorithm GP-Graph is proposed based on Agent interaction graph.

[0027] GP-Graph captures intra-group and inter-group interactions by unrolling the input pedestrian graph. In the process of unrolling the input pedestrian graph, first of all, it is necessary to learn to assign each pedestrian to the most likely behavior group, that is, to assign each pedestrian to the most likely behavior group through group index, so as to obtain pedestrian grouping information. The group index of each pedestrian is generated using the pair distance matrix. The pair distance matrix D and a set of colleague indexes are defined as: D i,j = ||F φ (X i )-F φ (X j )| for i,j e [1,...,N]; y = {pair(i,j)| i,j e [1,...,N], i≠j, D i,j ≤ p}; where F φ is a learnable convolutional layer, and p is a learnable threshold parameter. The group index set G is: G = G k | G k = U (i,j)∈γ {i,j}, G a ∩ G b = f for a≠b; where G k represents the kth group, which is the union of each pair set (i,j).

[0028] Next, to make the indexing process end-to-end trainable, a pass-through group backpropagation trick is introduced, which then converts the input pedestrian graph into intra-group and inter-group interaction graphs. During this process, group pooling and unpooling operations are proposed for unassociated group members by masking the edges of the input pedestrian graph, and a group with multiple members is represented as a graph node.

[0029] Next, by weight sharing with the baseline trajectory predictor, weight sharing means that the parameters of each filter (i.e., the values of the parameters of the filter) are fixed when the filter traverses the entire image. A hierarchical representation is enforced from the input pedestrian graph and the separated interactions. This representation is used to infer a socially acceptable probability map of future trajectories after passing through the group integration module of the present application.

[0030] Also, a U-NET architecture with a pooling layer is used to encode the features on the graph, and the U-Net network can use the effective labeled data more effectively from the few training images by relying on data augmentation. By reducing the number of nodes through the pooling layer in the U-NET, higher-level grouping features can be obtained, and then the agent features are recovered through the unpooling operation. Unlike traditional pooling and unpooling operators that handle grid structure data, it is not the most appropriate to apply them to the graph structure data of the present application. For this reason, the present application proposes a pedestrian graph-oriented group pooling and unpooling method, that is, the grouping information of the pedestrians is estimated through the social interaction of the pedestrians: (1) Estimate the grouping information of the pedestrians using the grouping assignment module. Based on the distance between the historical trajectories, the feature similarity between all pairs of pedestrians is measured using the historical trajectories of each pedestrian. By this pairwise distance, all pedestrians who are likely to be in the same group can be selected.

[0031] Referring to the accompanying drawings Figure 2 The present application also introduces a group-level latent vector sampling to ensure collective inference on a set of plausible future trajectories.

[0032] The present application groups the pedestrian features and group indices from the input trajectories and estimated grouping assignment information, and calculates a new pedestrian trajectory feature X' through the aggregation of inter-group features. A more powerful group intelligence feature is encoded in addition to the existing agent social interaction aggregation model to achieve highly accurate human trajectory prediction.

[0033] The application further provides an automatic driving method of a vehicle, which adopts the pedestrian trajectory prediction method to predict the pedestrian trajectory, thereby planning a driving route by avoiding pedestrians. Based on the characteristics of urban crowds, the automatic driving method of the vehicle provided by the application can specify corresponding urban planning measures under complex crowds by trajectory prediction, solve large-scale and irregular crowd trajectory prediction, estimate grouping information by using a grouping distribution module, and then generate intra-group or inter-group interaction graphs by shielding irrelevant nodes and performing pedestrian group pool generation. The weight sharing trajectory prediction model adopts three types of graphs, captures group-aware social interactions, and then applies a group pool operator to encode agent features from group features, and then inputs a group integration module to estimate a probability distribution of future trajectory prediction, and finally obtains pedestrian trajectory features.

[0034] It should be noted that the embodiments of the present application have better implementation, and do not limit the present application in any form, and any skilled person in the art can change or modify the above disclosed technical content into equivalent effective embodiments, as long as it does not deviate from the technical solution of the present application, and any modification or equivalent change and modification of the above embodiments according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A method for pedestrian trajectory prediction, the method comprising: The method comprises the following steps: acquiring a graph of pedestrians, the graph of pedestrians refers to a set of pedestrian nodes and edges on it representing their pairwise social interactions ; allocating each pedestrian to the most likely behavior group by group indexing according to the pedestrian graph to obtain pedestrian grouping information; converting the pedestrian graph into intra-group and inter-group interaction graphs according to the pedestrian grouping information through a straight-through group backpropagation process; implementing a hierarchical representation from an input pedestrian graph and separated interactions through weight sharing with a baseline trajectory predictor; encoding agent features from group features through a U-NET network structure with a pooling layer, reducing the number of nodes through the pooling layer in the U-NET to obtain higher-level group features; and recovering the agent features through a de-pooling operation to obtain a group integration module, and obtaining a probability map of a socially acceptable future trajectory through the group integration module; the agent includes a pedestrian, and the agent feature includes a pedestrian trajectory; the converting the pedestrian graph into intra-group and inter-group interaction graphs according to the pedestrian grouping information through the straight-through group backpropagation process further comprises: performing group pooling and de-pooling operations by masking edges of the pedestrian graph for unrelated group members, and representing a group with multiple members as a graph node.

2. The pedestrian trajectory prediction method of claim 1, wherein, The group indexing comprises estimating grouping information to which a pedestrian belongs through social interactions of the pedestrian.

3. The pedestrian trajectory prediction method of claim 2, wherein, The estimating grouping information to which a pedestrian belongs through social interactions of the pedestrian comprises: obtaining a historical trajectory of each pedestrian; measuring feature similarity between all pairs of pedestrians based on distances of the historical trajectories; and defining two pedestrians with the highest similarity as a pair of pedestrians; and the pedestrian and a plurality of pair-of-pedestrians of the pedestrian are in the same group of pedestrians.

4. The pedestrian trajectory prediction method of claim 3, wherein, The distance matrix D of the historical trajectories of the pair of pedestrians is defined as: ; ； wherein, is a learnable convolutional layer, is a learnable threshold parameter.

5. The pedestrian trajectory prediction method of claim 4, wherein, The set index set G is: ; wherein, represents the kth group, is the union of each pair set of the kth group.

6. The pedestrian trajectory prediction method of claim 1, wherein, a history of trajectories of the pedestrian is represented as wherein is a two-dimensional coordinate of the pedestrian n at a specific time t.

7. An automatic driving method of a vehicle, characterized by, The pedestrian trajectory prediction method according to any one of the preceding claims is used to predict pedestrian trajectories, thereby planning a driving route by avoiding pedestrians.

Images