Pedestrian trajectory prediction method and system based on group graph attention network

Abstract

The invention discloses a pedestrian trajectory prediction method and system based on a group graph attention network, and the method comprises the steps: obtaining a to-be-predicted video; extracting pedestrian information from the video to be predicted; based on the pedestrian information, performing group division on the pedestrians, and dividing the pedestrians into different groups; extracting features in the group to obtain a consistency value in the group; splicing the intra-group consistency value, the group speed and the group density to obtain an intra-group feature vector; extracting features among the groups to obtain a conflict value matrix among the groups; performing feature fusion on the intra-group feature vectors and the inter-group conflict value matrix to obtain group aggregation features; splicing the group aggregation features, the current pedestrian coordinates, the surrounding pedestrian features and the obstacle features to obtain multi-feature information representation; and inputting the multi-feature information representation into the trained multi-feature time convolution network to obtain a prediction trajectory of the current pedestrian.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian trajectory prediction, and in particular to a pedestrian trajectory prediction method and system based on a group graph attention network. Background Art

[0002] Pedestrian trajectory prediction has been widely used in many fields, including autonomous driving, mobile robots, and intelligent surveillance, and has become a current research hotspot. However, trajectory prediction remains a challenging problem due to implicit group features and pedestrian interactions. More accurate future pedestrian trajectory prediction is closely related to group features. For example, when walking, more than 70% of individuals will form groups with people around them. In reality, pedestrians in a group have similar distances, similar speeds, and the same destinations, and the constraints of the group on pedestrians in the group make the group show consistency characteristics. On the contrary, pedestrians between groups have different speeds and destinations, as well as the impact of the environment on the distance between pedestrians. When pedestrians between groups surpass or approach in opposite directions, they will show closer distances and larger speed differences, resulting in strong conflicting characteristics between groups. Therefore, deeply capturing rich group features is the key and challenge in pedestrian trajectory prediction.

[0003] Traditional rule-based methods simulate pairwise interactions of pedestrians, but they are computationally expensive and difficult to apply to complex environments. With the continuous application of deep learning methods in trajectory prediction, a large number of works simply model pedestrian interactions by learning feature representations of pedestrian historical trajectories. In order to be more consistent with real interactions in society, Kosaraju et al. increased the interaction between pedestrians and scenes and combined it with the interaction between pedestrians. Unfortunately, humans exhibit characteristic properties that contribute to group behavior, and only considering pedestrian interactions may ignore these group effects, resulting in inaccurate modeling of real-world scenarios.

[0004] At present, some studies have begun to focus on the impact of groups on trajectory prediction. Group-LSTM clusters pedestrian trajectories through coherent filtering, and uses the obtained clustering information to derive the neighborhood hidden state tensor. RSBG uses group annotations as supervision, calculates the pedestrian relationship matrix by performing inner product operations on historical trajectory features, and iteratively obtains pedestrian combination features. GP-Graph uses pedestrian historical trajectories to represent pedestrian features in a group, and simply aggregates and averages the features of surrounding pedestrians to form group features. However, this method lacks dynamic quantification of interactions within and between groups, which also limits its ability to capture rich group features. In addition, in real scenes, the factors that affect pedestrian interactions are complex and diverse, and it is necessary to consider the multiple effects of groups, obstacles in the scene, and surrounding pedestrians on pedestrian interactions. Summary of the invention

[0005] In order to solve the shortcomings of the prior art, the present invention provides a pedestrian trajectory prediction method and system based on a group graph attention network; first, the present invention proposes a group feature aggregation method (Group-GAT) based on a group graph attention network to deeply capture rich group feature information. The method enhances the modeling of pedestrian interactions within and between groups in the scene based on the proposed intra-group consistency and inter-group conflict features. By splicing intra-group consistency, group speed and group density, the representation ability of group features is enhanced. At the same time, the introduction of the inter-group conflict matrix not only helps the network to measure the correlation between group feature nodes when calculating the attention coefficient, but also considers the influence of inter-group conflict, thereby enhancing the aggregation of group feature information. Secondly, since the interaction in the real world is very complex, the present invention proposes a group multi-feature information representation model. Multi-feature information representation is obtained by splicing group aggregation features, pedestrian coordinates, surrounding pedestrian features and obstacle features to represent the interaction information in the real world. Finally, the present invention proposes a multi-feature temporal convolutional network to measure the feature output of multi-feature information affecting pedestrian interaction, and then operates the feature output in the time dimension to generate a series of predicted trajectory coordinates in the future time.

[0006] On the one hand, a pedestrian trajectory prediction method based on a group graph attention network is provided, including:

[0007] Obtain the video to be predicted; extract pedestrian information from the video to be predicted;

[0008] Based on the pedestrian information, the pedestrians are divided into groups, and the pedestrians are divided into different groups; the features within the group are extracted to obtain the consistency value within the group; the consistency value within the group, the group speed and the group density are spliced ​​to obtain the feature vector within the group; the features between the groups are extracted to obtain the conflict value matrix between the groups;

[0009] The feature vectors within the group and the conflict value matrix between groups are fused to obtain the group aggregation features; the group aggregation features, the current pedestrian coordinates, the surrounding pedestrian features and the obstacle features are spliced ​​to obtain a multi-feature information representation; the multi-feature information representation is input into the trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

[0010] On the other hand, a pedestrian trajectory prediction system based on group graph attention network is provided, including:

[0011] The acquisition module is configured to: acquire the video to be predicted; extract pedestrian information from the video to be predicted;

[0012] The feature extraction module is configured to: divide the pedestrians into groups based on the pedestrian information, and divide the pedestrians into different groups; extract the features within the group to obtain the consistency value within the group; splice the consistency value within the group, the group speed and the group density to obtain the feature vector within the group; extract the features between the groups to obtain the conflict value matrix between the groups;

[0013] The trajectory prediction module is configured as follows: feature fusion of intra-group feature vectors and inter-group conflict value matrices to obtain group aggregate features; concatenation of group aggregate features, current pedestrian coordinates, surrounding pedestrian features, and obstacle features to obtain multi-feature information representation; and inputting the multi-feature information representation into a trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

[0014] On the other hand, there is also provided an electronic device, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing the computer readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] (1) The present invention proposes to group pedestrians based on the pedestrian information, and divide the pedestrians into different groups; extract the features within the group to obtain the consistency value within the group; splice the consistency value within the group, the group speed and the group density to obtain the feature vector within the group; extract the features between groups to obtain the conflict value matrix between groups, so as to capture rich group features. This method effectively captures group features and interactions by quantifying and aggregating the consistency within the group and the conflict between groups shown by the groups.

[0022] (2) The present invention proposes to splice group aggregation features, current pedestrian coordinates, surrounding pedestrian features and obstacle features to obtain a multi-feature information representation.

[0023] (3) The present invention proposes a multi-feature temporal convolutional network that can quantify the feature outputs of multi-feature information that affect pedestrian interaction and operate on the temporal dimension of these feature outputs to obtain future trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 is a flow chart of the method of embodiment 1;

[0026] FIG. 2( a ) and FIG. 2( b ) are schematic diagrams of consistent interaction within a group according to Embodiment 1;

[0027] FIG3(a) and FIG3(b) are schematic diagrams of conflicting interactions between pedestrian groups in Example 1;

[0028] Figure 4 A schematic diagram of a two-layer group graph attention network of Example 1;

[0029] Figure 5 Schematic diagram of a multi-feature temporal convolutional network of Example 1. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] Embodiment 1

[0032] This embodiment provides a pedestrian trajectory prediction method based on a group graph attention network;

[0033] like Figure 1 As shown in FIG. 1 , a pedestrian trajectory prediction method based on a group graph attention network includes:

[0034] S101: Obtain a video to be predicted; extract pedestrian information from the video to be predicted;

[0035] S102: based on the pedestrian information, divide the pedestrians into groups, and divide the pedestrians into different groups; extract the features within the group to obtain the consistency value within the group; splice the consistency value within the group, the group speed and the group density to obtain the feature vector within the group; extract the features between the groups to obtain the conflict value matrix between the groups;

[0036] S103: Fusing the intra-group feature vectors and the inter-group conflict value matrix to obtain group aggregate features; concatenating the group aggregate features, the current pedestrian coordinates, the surrounding pedestrian features, and the obstacle features to obtain a multi-feature information representation; inputting the multi-feature information representation into the trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

[0037] Furthermore, the S101: obtaining the video to be predicted is collected through a camera or a webcam.

[0038] Furthermore, the S101: extracting pedestrian information from the video to be predicted, the pedestrian information including pedestrian speed, pedestrian position and pedestrian historical trajectory.

[0039] Assume that there are N pedestrians in a video scene, and extract the time [1,T his ] the historical trajectory coordinates of each pedestrian i∈[1,...,N] in is a pair of spatial coordinates of pedestrian i at time t. Similarly, at time [T his+1 ,T pred ] The actual trajectory coordinates of each pedestrian i are expressed as The scene image obtained by taking a bird's-eye view at time step t is represented by I t express.

[0040] Group interaction diagram In each time step t∈{1,2,...,T his}, The set of edges of The initial value of is the group characteristics consisting of intra-group consistency, group speed and group density.

[0041] The set of edges Represents interactions between groups. The initial value of is the weight in the inter-group conflict matrix.

[0042] Next, the intra-group features and inter-group conflict matrix are input into the graph attention network to obtain group aggregation features. Finally, the concatenated multi-feature information representation is input into MF-TCN to measure the impact of multi-feature information on pedestrian interaction and obtain feature output. Then, multi-layer convolution operations are performed on the feature output to obtain the time [T his +1,T pred ] to generate the predicted coordinate sequence

[0043] Furthermore, the step S102: dividing pedestrians into groups based on the pedestrian information, and dividing the pedestrians into different groups, includes:

[0044] The distance between pedestrians, the similarity of the speed directions between pedestrians and the similarity of the speeds between pedestrians are calculated; the pedestrian similarity is determined based on the distance between pedestrians, the similarity of the speed directions between pedestrians and the similarity of the speeds between pedestrians; and the pedestrians are divided into different groups according to the pedestrian similarity.

[0045] Furthermore, the step S102: dividing pedestrians into groups based on the pedestrian information, and dividing the pedestrians into different groups, specifically includes:

[0046] First, the Euclidean distance is used to calculate the distance between pedestrian i and pedestrian j at time step t

[0047]

[0048] in, represents the spatial coordinates of pedestrian i at time step t, represents the spatial coordinates of pedestrian j at time step t, The smaller it is, the closer the distance between pedestrians will be.

[0049] Secondly, when two pedestrians are back to back, although they are close to each other, they are not in the same group because they are moving in different directions. Therefore, the similarity of the speed directions between the pedestrians should be considered.

[0050]

[0051] in, represents the velocity vector of pedestrian i at time t represents the inner product of vectors, The larger it is, the more similar the movement directions of pedestrians are.

[0052] Finally, the similarity of speed between pedestrians

[0053]

[0054] Among them, |·| represents the absolute value operation, The smaller it is, the closer the speeds of pedestrians are.

[0055] Because the distance, speed direction and speed similarity between pedestrians in a group are similar, based on the group characteristics, the pedestrian similarity is obtained by integrating formulas (1), (2) and (3):

[0056]

[0057] Among them, 0<γ 1 ,γ 2 ,γ 3 <1 and γ 1 +γ 2 +γ 3 =1, each parameter represents a different weight, pedestrian similarity The larger it is, the more likely it is that pedestrians i and j belong to the same group at time step t;

[0058] When the group value is greater than δ, the two pedestrians belong to the same group;

[0059] After continuous calculation, all pedestrians are finally assigned to each group. Among them, represents the Kth group at time step t.

[0060] Then calculate the maximum similarity value max{group} between each pedestrian in the group, and obtain the coordinates of the pedestrian with the maximum value as the center coordinates of the group

[0061] The group division method divides pedestrians that meet social relationships into the same group for the next step of group feature analysis. The δ value can be fine-tuned according to different scenarios.

[0062] It should be understood that people have complex social relationships in society. In trajectory prediction, such social relationships will cause individuals to form groups. Pedestrians in the same group often have the characteristics of close distance, similar speed direction and similar speed. Therefore, the present invention proposes the Group-K-means algorithm.

[0063] Furthermore, the S102: extracting features within the group to obtain a consistency value within the group includes:

[0064] Measure the consistency between pedestrians in a group based on distance, speed, and direction:

[0065]

[0066] Among them, at time step t, and Represent the direction vectors of pedestrian i and pedestrian j respectively and and They represent the pedestrian’s velocity vectors i and j respectively. represents the Euclidean distance between pedestrians i and j. By calculating formula (5), the consistency values ​​between pedestrians are normalized to the same range.

[0067] Then, the consistency values ​​between each pair of pedestrians in a group are summed and divided by the number of pedestrian pairs to obtain the consistency value of each group. in represents the number of pedestrians in the kth group.

[0068] It should be understood that the distance between pedestrians in a group is close, the speed and direction are relatively consistent, and pedestrians in a group tend to think in a group way when making the next path selection, which makes pedestrians in the group show consistency characteristics. In addition, the speed differences between pedestrians in different groups due to different destinations, and the distance between groups caused by loose or crowded environments, thus causing different degrees of conflict between groups. It is hoped that the model will pay more attention to group movement in order to accurately and reasonably predict the future trajectory of pedestrians.

[0069] In Figure 2(a), when the direction angle between pedestrians i and j is small, the speed is similar and the distance is close, the consistency within the group is relatively strong. In Figure 2(b), when the direction angle between pedestrians i and j increases, and the speed difference and distance increase, the consistency within the group is relatively weakened. The consistency within the group indicates the degree of consistency of the mutual cooperation of individuals. The closer the distance between pedestrians in the group, the closer the speed, and the more the direction is the same, the stronger the consistency within the group.

[0070] Furthermore, the S102: concatenating the intra-group consistency value, the group speed and the group density to obtain the intra-group feature vector includes:

[0071] The group center coordinates The corresponding speed is taken as the group center speed

[0072] To obtain the density of each group, first, at each time step t, the Euclidean distance between each pedestrian in the group is calculated. Get the distance matrix.

[0073]

[0074] Secondly, take the average of all distances in the distance matrix to represent the average distance between pedestrians in the group

[0075]

[0076] Finally, the group density is obtained by taking the inverse of the average distance

[0077]

[0078] Among them, at time step t, represents the number of pedestrians in the kth group. The higher it is, the smaller the average distance between pedestrians is, and the denser the distribution of pedestrians in the group is.

[0079] The group feature vector is obtained by concatenating the intra-group consistency value, group speed and group density.

[0080] It should be understood that in order to better measure the characteristics within a group, the present invention adds considerations to group speed and group density.

[0081] Furthermore, the S102: extracting the features between groups to obtain the conflict value matrix between groups includes:

[0082] Obtain the inter-group conflict value matrix through the inter-group conflict function

[0083]

[0084] Among them, at time step t, and Respectively represent group g k and group g m The velocity vector of μ. t and θ t Respectively represent group g k and g m The angle between the velocity direction and the position. Indicates group g k and g m The Euclidean distance between .

[0085] In order to calculate the weight of the edge between the group feature nodes in the group graph The inter-group conflict value matrix is ​​normalized by the SoftMax function:

[0086]

[0087] It should be understood that Figure 3(a) and Figure 3(b) show that pedestrians of different colors represent different groups. Figure 3(a) shows two groups walking in opposite directions. Figure 3(b) shows two situations: when group g m The speed is greater than group g k When the speed of group g m Overtake the group from one side k When group g m The speed is less than group g k When the speed of group g m Follow in groupg k after.

[0088] Inter-group conflict characterizes the interaction or friction between groups when they approach each other. When pedestrians walk, they inevitably overtake others and meet others. In Figure 3(a), the two groups that meet in opposite directions affect each other, and there is a possibility of collision. As the distance is closer and the speed difference is greater, the conflict value is higher. In Figure 3(b), when the rear group wants to overtake the front group from one side, the obstruction of the front group has a one-way impact on the rear group, resulting in a medium inter-group conflict value. When the rear group always follows the front group, the inter-group conflict value is relatively low. By measuring the conflict values ​​between pedestrians in different groups, we can better understand why people often prefer to keep a certain distance from others, thereby predicting more accurate trajectories.

[0089] Furthermore, the S103: performing feature fusion on the intra-group feature vector and the inter-group conflict value matrix to obtain group aggregate features includes:

[0090] By combining the characteristics within the group Intergroup Conflict Matrix Input into the graph attention layer to more accurately capture the importance between group feature nodes, and output the attention coefficient of the group feature node

[0091]

[0092] Among them, LeakyReLU is a nonlinear activation function, α is the weight vector of the feedforward neural network, Τ represents the transpose, and W gat represents the learned weight matrix, || represents the concatenation operation, ⊙ represents the multiplication of the elements at the corresponding positions, N(g k ) is group g k The first-order neighborhood set of . It is a group feature node that combines the intra-group consistency, group density, and group speed.

[0093] In order to improve the generalization performance and aggregate the deep information of group feature nodes, a two-layer graph attention network is adopted, such as Figure 4 The first graph attention layer uses a multi-head attention mechanism. At time step t, the group feature node g k The output of the first graph attention layer is:

[0094]

[0095] Where N(g k ) is group g kis the first-order neighborhood set of , σ(·) is the nonlinear activation function, and H represents the number of attention heads. The output of the first layer is processed by the nonlinear activation function ELU and regularization technology as the input of the next layer to increase the nonlinearity of the graph attention layer and reduce the risk of overfitting.

[0096] The second layer uses a single-head attention mechanism. At time step t, the group feature node The output of the second graph attention layer is:

[0097]

[0098] Finally, we get the group aggregation features in

[0099] It should be understood that unlike most works that use pedestrian relative coordinates as feature nodes, It is a group feature node that combines the consistency, density and speed of the group. By providing each group node with richer group information, the representation ability of the group node can be improved, which helps to better capture the relationship and dependency between group nodes.

[0100] in, Figure 4 The first layer in the network adopts a multi-head attention mechanism, and the second layer adopts a single-head attention mechanism.

[0101] Furthermore, the S103: combining the group aggregation features, the current pedestrian coordinates, the surrounding pedestrian features and the obstacle features to obtain a multi-feature information representation, including:

[0102] First, extract the features of surrounding pedestrians and obtain the distance set by calculating the Euclidean distance between pedestrian i and all pedestrians; if the distance between a pedestrian and pedestrian i is less than D, then this pedestrian belongs to the neighborhood of pedestrian i, and the neighborhood set N(i) of pedestrian i is obtained; splice the coordinates of surrounding pedestrians and speed The splicing result is used as the feature information of surrounding pedestrians

[0103]

[0104] Second, use the pre-trained VGGNet-19 to extract the scene image I t Extract obstacle features from

[0105]

[0106] Among them, W VGGnet-19 is the weight in the fully convolutional neural network scene segmentation task.

[0107] Finally, the current pedestrian coordinates Group Aggregation Features Surrounding pedestrian characteristics and obstacle characteristics Input the fully connected layer respectively, the current pedestrian coordinates The linear transformation result of for:

[0108]

[0109] Among them, FC(·) is a linear transformation of one layer, W l is the weight of the linear transformation.

[0110] Similarly, through formula (14), we can also get X i ' t , and Features can be mapped to a higher dimensional representation space.

[0111]

[0112] Then, by concatenating X i ' t , and Get multi-feature information representation

[0113]

[0114] Among them, g k represents the group that pedestrian i belongs to, and Concat(·) represents the concatenation operation.

[0115] It should be understood that in order to obtain a rich multi-feature information representation, the surrounding pedestrian features and obstacle features are spliced ​​and fused with the pedestrian coordinates and group aggregation features. The multi-feature information representation is obtained by splicing, thereby fusing richer feature information and learning the complex relationship between them.

[0116] Furthermore, the S103: inputting the multi-feature information representation into the trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian includes:

[0117] First, the multi-feature information representation is input into the multi-layer perceptron MLP to obtain the intermediate representation f of the features that affect the current pedestrian i i t :

[0118]

[0119] where γ(·) is a multi-layer perceptron (MLP) with a nonlinear activation function.

[0120] Secondly, f i t The relative weight of the influence of multiple feature information on pedestrians is calculated by Softmax, and the weight is combined with f i t Perform element-wise multiplication and then embed the product into the pedestrian coordinates:

[0121]

[0122] in, is an element-by-element multiplication operation, W ee is the embedding weight. φ(·) is the embedding layer. Specifically, the pedestrian coordinates and the weight product are concatenated in the feature dimension, and then the concatenated three-dimensional tensor is flattened into a two-dimensional tensor, and the feature output is obtained through linear transformation. The feature output F = {F 1 ,F 2 ,...F i ,...,F N},in

[0123] Among them, F N is the Nth pedestrian feature output, is at time step T obs The feature output vector of the i-th pedestrian.

[0124] Next, the feature output F is input into seven convolutional layers connected in sequence to perform time dimension operations to predict the future [T his +1,T pred ]Trajectory in time;

[0125] Finally, after multiple layers of convolution operations, the two-dimensional Gaussian distribution parameters [σ, μ, ρ] required for predicting coordinates are obtained; the distribution parameters are used to generate the corresponding two-dimensional Gaussian distribution.

[0126]

[0127] Then sampling is applied to the 2D Gaussian distribution to generate a specific predicted 2D coordinate sequence for the future time step t

[0128]

[0129] Among them, ρ t is the correlation coefficient of the coordinates x and y, obtained by applying the hyperbolic tangent function, and is the standard deviation, and is the mean.

[0130] Formula (18) can represent that a two-dimensional Gaussian distribution is generated according to the two-dimensional Gaussian distribution parameters, and the predicted two-dimensional coordinate sequence obeys the two-dimensional Gaussian distribution.

[0131] like Figure 5 As shown, the seven convolutional layers connected in sequence, wherein the first convolutional layer includes a two-dimensional convolutional layer and an activation function layer PReLU connected in sequence; the middle five layers have one more residual connection than the first layer; the structures of the middle five layers are consistent, wherein the second convolutional layer includes a two-dimensional convolutional layer, an activation function layer and an adder connected in sequence, and the input end of the adder is connected to the input end of the two-dimensional convolutional layer of the second convolutional layer; the seventh convolutional layer has only convolution operations.

[0132] Furthermore, the training process of the trained multi-feature temporal convolutional network includes:

[0133] Constructing a training set, wherein the training set is a pedestrian feature representation of a known predicted trajectory;

[0134] The training set is input into the multi-feature time convolutional network, and the multi-feature time convolutional network is trained. When the loss function value of the multi-feature time convolutional network no longer decreases, the training is stopped to obtain the trained multi-feature time convolutional network.

[0135] Furthermore, the loss function of the multi-feature temporal convolutional network is expressed as:

[0136]

[0137] In order to make the loss function better guide model training, minimize the true two-dimensional coordinates of pedestrian i at time step t With the prediction parameters of the Gaussian distribution Specifically, during the predicted time period, the negative log-likelihood value of the predicted coordinates that obey the two-dimensional Gaussian distribution is calculated to obtain the negative log-loss. Then the negative log-loss of all predicted coordinates is averaged to obtain the final loss value for model optimization.

[0138] It should be understood that after merging multiple feature information into a high-dimensional space to obtain a multi-feature information representation, the present invention needs to measure their interactive impact on the current pedestrian, and then predict the real trajectory in society through operations in the time dimension, such as Figure 5 As shown. Among them, Figure 5 The 7 cubes in represent 7 convolutional layers. The two numbers in each block represent the number of input channels and the number of output channels, respectively.

[0139] Group characteristics have a significant impact on pedestrian movement and are a hot topic in the field of pedestrian trajectory prediction. In reality, pedestrians in a group show significant consistency due to their compact locations, similar destinations, and coordination within the group. Due to the dispersed destinations and lack of coordination between groups, there are large differences in speed and direction between groups, showing strong conflicts. However, existing pedestrian trajectory prediction based on group characteristics lacks sufficient consideration of intra-group and inter-group characteristics.

[0140] In order to solve this problem, the present invention proposes a pedestrian trajectory prediction method and system based on a group graph attention network. Specifically, first, the present invention proposes a group feature aggregation method (Group-GAT) based on a group graph attention network. By quantifying and aggregating the intra-group and inter-group features exhibited by the group, the method of the present invention can well capture the intra-group and inter-group features and interactions. Secondly, the present invention proposes a group multi-feature information representation model, which fuses and splices the captured group aggregation features, pedestrian coordinates, surrounding pedestrian features, and obstacle features. Finally, the present invention proposes a multi-feature temporal convolutional network (MF-TCNs), which embeds the influence weights of multi-feature information on pedestrians into pedestrian coordinates to obtain feature outputs, and operates the feature outputs in the time dimension to predict future trajectories.

[0141] First, video data is obtained from imaging devices, such as surveillance, cameras, etc., and then pedestrian information (speed, position, etc.) in the video is extracted. Then, the Group-K-means algorithm proposed in the present invention is used to calculate group features (i.e., consistency within the group and conflict between groups), and then the group features are combined with pedestrian features and obstacle features to construct a group multi-feature representation model. Then, a multi-feature temporal convolutional network is constructed to fuse multiple features. Finally, the pedestrian trajectory prediction problem is defined as fitting Gaussian distribution parameters, and the future trajectory of the pedestrian is obtained based on distribution sampling.

[0142] Embodiment 2

[0143] This embodiment provides a pedestrian trajectory prediction system based on a group graph attention network, including:

[0144] The acquisition module is configured to: acquire the video to be predicted; extract pedestrian information from the video to be predicted;

[0145] The feature extraction module is configured to: divide the pedestrians into groups based on the pedestrian information, and divide the pedestrians into different groups; extract the features within the group to obtain the consistency value within the group; splice the consistency value within the group, the group speed and the group density to obtain the feature vector within the group; extract the features between the groups to obtain the conflict value matrix between the groups;

[0146] The trajectory prediction module is configured as follows: feature fusion of intra-group feature vectors and inter-group conflict value matrices to obtain group aggregate features; concatenation of group aggregate features, current pedestrian coordinates, surrounding pedestrian features, and obstacle features to obtain multi-feature information representation; and inputting the multi-feature information representation into a trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

[0147] It should be noted that the acquisition module, feature extraction module and trajectory prediction module correspond to steps S101 to S103 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0148] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0150] Embodiment 3

[0151] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0152] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0153] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0154] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0155] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] Embodiment 4

[0158] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pedestrian trajectory prediction method based on a group graph attention network, characterized by: include: Get the video to be predicted; Extract pedestrian information from the video to be predicted; Based on the pedestrian information, the pedestrians are divided into groups, and the pedestrians are divided into different groups; the features within the group are extracted to obtain the consistency value within the group; The consistency value, group speed and group density within the group are concatenated to obtain the feature vector within the group; the features between groups are extracted to obtain the conflict value matrix between groups; The feature vectors within the group and the conflict value matrix between groups are fused to obtain the group aggregation features; the group aggregation features, the current pedestrian coordinates, the surrounding pedestrian features and the obstacle features are spliced ​​to obtain a multi-feature information representation; the multi-feature information representation is input into the trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

2. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: Extracting pedestrian information from the video to be predicted, the pedestrian information including: pedestrian speed, pedestrian position and pedestrian historical trajectory; Assume that there are N pedestrians in a video scene, and extract the time [1,T his ] the historical trajectory coordinates of each pedestrian i∈[1,...,N] in is a pair of spatial coordinates of pedestrian i at time t; similarly, at time [T his+1 ,T pred ] The actual trajectory coordinates of each pedestrian i are expressed as The scene image obtained by taking a bird's-eye view at time step t is represented by I t express; Group interaction diagram Represented; at each time step t∈{1,2,...,T his }, The set of edges of The initial value of is the group characteristics composed of intra-group consistency, group speed and group density; The set of edges Represents interactions between groups; The initial value of is the weight in the inter-group conflict matrix; Next, the intra-group features and inter-group conflict matrix are input into the graph attention network to obtain group aggregation features. Finally, the concatenated multi-feature information representation is input into MF-TCN to measure the impact of multi-feature information on pedestrian interaction and obtain feature output. Then, multi-layer convolution operations are performed on the feature output to obtain the time [T his +1,T pred ] to generate the predicted coordinate sequence 3. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: The dividing the pedestrians into groups based on the pedestrian information into different groups specifically includes: First, the Euclidean distance is used to calculate the distance between pedestrian i and pedestrian j at time step t in, represents the spatial coordinates of pedestrian i at time step t, represents the spatial coordinates of pedestrian j at time step t, The smaller it is, the closer the distance between pedestrians will be; Secondly, when two pedestrians are back to back, although they are close to each other, they are not in the same group because they are moving in different directions. Therefore, the similarity of the speed directions between the pedestrians should be considered. in, represents the velocity vector of pedestrian i at time t represents the inner product of vectors, The larger it is, the more the movement directions of pedestrians are the same; Finally, the similarity of speed between pedestrians Among them, |·| represents the absolute value operation, The smaller it is, the closer the speeds of pedestrians are; Therefore, based on the group characteristics, the pedestrian similarity is obtained by integrating formulas (1), (2), and (3): Among them, 0<γ1,γ2,γ3<1 and γ1+γ2+γ3=1, each parameter represents a different weight, pedestrian similarity The larger it is, the more likely it is that pedestrians i and j belong to the same group at time step t; When the group value is greater than δ, the two pedestrians belong to the same group; After continuous calculation, all pedestrians are finally assigned to each group. Among them, represents the Kth group at time step t; Then calculate the maximum similarity value max{group} between each pedestrian in the group, and obtain the coordinates of the pedestrian with the maximum value as the center coordinates of the group 4. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: The extracting of features within the group to obtain a consistency value within the group includes: Measure the consistency between pedestrians in a group based on distance, speed, and direction: Among them, at time step t, and Represent the direction vectors of pedestrian i and pedestrian j respectively and and They represent the pedestrian’s velocity vectors i and j respectively; represents the Euclidean distance between pedestrians i and j. Through the calculation of formula (5), the consistency values ​​between pedestrians are normalized to the same range. Then, the consistency values ​​between each pair of pedestrians in a group are summed and divided by the number of pedestrian pairs to obtain the consistency value of each group. in represents the number of pedestrians in the kth group; The intra-group consistency value, group speed and group density are concatenated to obtain the intra-group feature vector, including: The group center coordinates The corresponding speed is taken as the group center speed To obtain the density of each group, first, at each time step t, the Euclidean distance between each pedestrian in the group is calculated. Get the distance matrix; Secondly, take the average of all distances in the distance matrix to represent the average distance between pedestrians in the group Finally, the group density is obtained by taking the inverse of the average distance Among them, at time step t, represents the number of pedestrians in the kth group; The higher the value, the smaller the average distance between pedestrians, and the denser the distribution of pedestrians in the group; the group feature vector is obtained by concatenating the consistency value, group speed and group density within the group.

5. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: The inter-group features are extracted to obtain an inter-group conflict value matrix, including: Obtain the inter-group conflict value matrix through the inter-group conflict function Among them, at time step t, and Respectively represent group g k and group g m The velocity vector of t and θ t Respectively represent group g k and g m The angle between the velocity direction and the position line; Indicates group g k and g m The Euclidean distance between In order to calculate the weight of the edge between the group feature nodes in the group graph The inter-group conflict value matrix is ​​normalized by the SoftMax function:

6. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: The feature fusion of the intra-group feature vector and the inter-group conflict value matrix to obtain the group aggregation feature includes: By combining the characteristics within the group Intergroup Conflict Matrix Input into the graph attention layer, and output the attention coefficient of the group feature node Where t=1,...,T his , LeakyReLU is a nonlinear activation function, α is the weight vector of the feedforward neural network, Τ represents the transpose, W gat represents the learned weight matrix, || represents the concatenation operation, ⊙ represents the multiplication of the elements at the corresponding positions, N(g k ) is group g k The first-order neighborhood set of ; It is a group feature node that combines the consistency within the group, the group density, and the group speed; The first graph attention layer uses a multi-head attention mechanism; at time step t, the group feature node g k The output of the first graph attention layer is: Where N(g k ) is group g k is the first-order neighborhood set of , σ(·) is the nonlinear activation function, and H represents the number of attention heads; The second layer uses a single-head attention mechanism. At time step t, the group feature node The output of the second graph attention layer is: Finally, we get the group aggregation features or, The group aggregation features, the current pedestrian coordinates, the surrounding pedestrian features and the obstacle features are spliced ​​to obtain a multi-feature information representation, including: First, extract the features of surrounding pedestrians and obtain the distance set by calculating the Euclidean distance between pedestrian i and all pedestrians; if the distance between a pedestrian and pedestrian i is less than D, then this pedestrian belongs to the neighborhood of pedestrian i, and the neighborhood set N(i) of pedestrian i is obtained; splice the coordinates of surrounding pedestrians and speed The splicing result is used as the feature information of surrounding pedestrians Second, use the pre-trained VGGNet-19 to extract the scene image I t Extract obstacle features from Among them, W VGGnet-19 is the weight in the fully convolutional neural network scene segmentation task; Finally, the current pedestrian coordinates Group Aggregation Features Surrounding pedestrian characteristics and obstacle characteristics Input the fully connected layer respectively, the current pedestrian coordinates The linear transformation result of for: Among them, FC(·) is a linear transformation of one layer, W l is the weight of the linear transformation; we get and Then, by concatenating and Get multi-feature information representation Among them, g k represents the group that pedestrian i belongs to, and Concat(·) represents the concatenation operation.

7. The pedestrian trajectory prediction method based on group graph attention network as claimed in claim 1, characterized in that: The multi-feature information representation is input into the trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian, including: First, the multi-feature information representation is input into the multi-layer perceptron MLP to obtain the intermediate representation of the features that affect the current pedestrian i where γ(·) is a multi-layer perceptron MLP with a nonlinear activation function; Secondly, The relative weight of the influence of multiple feature information on pedestrians is calculated by Softmax, and the weight is combined with Perform element-wise multiplication and then embed the product into the pedestrian coordinates: in, is an element-by-element multiplication operation, W ee is the embedding weight; φ(·) is the embedding layer. Specifically, the pedestrian coordinates and the weight product are concatenated in the feature dimension, and then the concatenated three-dimensional tensor is flattened into a two-dimensional tensor, and the feature output is obtained by linear transformation, and the feature output F={F1,F2,...F i ,...,F N },in Among them, F N is the Nth pedestrian feature output, is at time step T obs The feature output vector of the i-th pedestrian; Next, the feature output F is input into seven convolutional layers connected in sequence to perform time dimension operations to predict the future [T his +1,T pred ]Trajectory in time; Finally, after multiple layers of convolution operations, the two-dimensional Gaussian distribution parameters [σ, μ, ρ] required for predicting coordinates are obtained; the distribution parameters are used to generate the corresponding two-dimensional Gaussian distribution. Then sampling is applied to the 2D Gaussian distribution to generate a specific predicted 2D coordinate sequence for the future time step t Among them, ρ t is the correlation coefficient of the coordinates x and y, obtained by applying the hyperbolic tangent function, and is the standard deviation, and is the mean; The training process of the trained multi-feature temporal convolutional network includes: Constructing a training set, wherein the training set is a pedestrian feature representation of a known predicted trajectory; Input the training set into the multi-feature temporal convolutional network, train the multi-feature temporal convolutional network, and stop training when the loss function value of the multi-feature temporal convolutional network no longer decreases, thereby obtaining the trained multi-feature temporal convolutional network; The loss function of the multi-feature temporal convolutional network is expressed as: In order to make the loss function better guide model training, minimize the true two-dimensional coordinates of pedestrian i at time step t With the prediction parameters of the Gaussian distribution The negative log-likelihood loss.

8. A pedestrian trajectory prediction system based on a group graph attention network, characterized by: include: An acquisition module is configured to: acquire a video to be predicted; Extract pedestrian information from the video to be predicted; A feature extraction module is configured to: divide the pedestrians into groups based on the pedestrian information, and divide the pedestrians into different groups; extract features within the group to obtain a consistency value within the group; The consistency value, group speed and group density within the group are concatenated to obtain the feature vector within the group; the features between groups are extracted to obtain the conflict value matrix between groups; The trajectory prediction module is configured as follows: feature fusion of intra-group feature vectors and inter-group conflict value matrices to obtain group aggregate features; concatenation of group aggregate features, current pedestrian coordinates, surrounding pedestrian features, and obstacle features to obtain multi-feature information representation; and inputting the multi-feature information representation into a trained multi-feature temporal convolutional network to obtain the predicted trajectory of the current pedestrian.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 7 is performed.

Images