Pedestrian trajectory prediction method and device based on heterogeneous relationship graph

Abstract

The present invention provides a pedestrian trajectory prediction method and device based on a heterogeneous relationship graph, which are applied to the field of artificial intelligence, wherein the above method includes: obtaining a target pedestrian, pedestrian motion information and obstacle information; inputting the pedestrian motion information and obstacle information into a heterogeneous relationship graph model to obtain the comprehensive trajectory characteristics of the target pedestrian output by the heterogeneous relationship graph model; inputting the comprehensive trajectory characteristics of the target pedestrian into a trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and feature fusion is performed based on a learnable prior vector and the heterogeneous relationship graph; the trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory characteristics; through the present application, more accurate pedestrian motion trajectory prediction can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pedestrian trajectory prediction method and device based on a heterogeneous relationship graph. Background Art

[0002] Pedestrian trajectory prediction aims to predict the movement trajectory of pedestrians in the future given their past or current motion trajectories.

[0003] In the related art, the environmental information around pedestrians is not explicitly considered, and obstacles are treated as static pedestrians when dealing with surrounding obstacles. However, the structural characteristics of obstacles are significantly different from those of pedestrians, and this simplification may lead to unstable prediction results.

[0004] It can be seen that the pedestrian trajectory prediction method in the related art has a technical problem that the prediction result is not accurate because it does not distinguish between obstacles and other pedestrians. Summary of the invention

[0005] The present invention provides a pedestrian trajectory prediction method and device based on a heterogeneous relationship graph, so as to solve the defect that the pedestrian trajectory prediction method in the prior art has low prediction accuracy due to failure to distinguish obstacles from other pedestrians, and realize more accurate pedestrian motion trajectory prediction.

[0006] The present invention provides a pedestrian trajectory prediction method based on a heterogeneous relationship graph, comprising the following steps.

[0007] Acquire a target pedestrian, pedestrian motion information and obstacle information; input the pedestrian motion information and the obstacle information into a heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model; input the comprehensive trajectory features of the target pedestrian into a trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and perform feature fusion with the heterogeneous relationship graph based on a learnable prior vector; the trajectory prediction module is used to predict the pedestrian's position at multiple future moments based on the comprehensive trajectory features.

[0008] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the heterogeneous relationship graph model includes: an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module; the pedestrian motion information and the obstacle information are input into the heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model, including: extracting motion features based on the pedestrian motion information through the individual motion feature extraction module to obtain the individual motion features of the target pedestrian; extracting motion features based on the obstacle feature extraction module through the obstacle feature extraction module Obstacle features are extracted from the obstacle information to obtain surrounding obstacle features of the target pedestrian; a heterogeneous relationship graph of the target pedestrian is constructed based on the individual motion features and the surrounding obstacle features through the global heterogeneous feature fusion module, wherein the node set of the heterogeneous relationship graph includes: individual motion features and surrounding obstacle features, and the edge set of the heterogeneous relationship graph includes: interaction relationships between obstacles and pedestrians and interaction relationships between pedestrians; comprehensive trajectory features of the target pedestrian are determined based on the heterogeneous relationship graph according to a learnable prior vector through the global heterogeneous feature fusion module.

[0009] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the individual motion feature extraction module includes: a multi-layer perceptron and a gated recurrent unit, and the motion feature extraction based on the pedestrian motion information to obtain the individual motion features of the target pedestrian includes: taking each moment in the pedestrian motion information as the current moment, and executing the following steps to obtain the individual motion features of the target pedestrian at each moment: inputting the observation value of the target pedestrian in the pedestrian motion information into the multi-layer perceptron with an activation function to obtain the embedded vector output by the multi-layer perceptron; inputting the embedded vector into the gated recurrent unit to obtain the individual motion features of the target pedestrian at the current moment output by the gated recurrent unit.

[0010] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the obstacle feature extraction module includes: an encoding layer, a multi-head self-attention module, a feedforward network layer and a maximum pooling layer; the obstacle feature extraction based on the obstacle information to obtain the surrounding obstacle features of the target pedestrian includes: determining the fan-shaped field of view of the target pedestrian based on the position of the target pedestrian and the speed of the target pedestrian; determining the intersection area of ​​the fan-shaped field of view and the obstacle position in the obstacle information as an obstacle polygon; inputting the feature vector corresponding to each line segment of the obstacle polygon into the encoding layer; The coding layer obtains the encoded feature vector output by the coding layer; the encoded feature vector is input into the multi-head self-attention module to obtain the query vector, key vector and value vector output by the multi-head self-attention module; the query vector, the key vector and the value vector are input into the feedforward network layer, and the output of the feedforward network layer is residually connected and layer-normalized with the encoded feature vector output by the coding layer to obtain the line segment features of each line segment; the line segment features of each line segment are input into the maximum pooling layer to obtain the surrounding obstacle features of the target pedestrian output by the maximum pooling layer.

[0011] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the comprehensive trajectory characteristics of the target pedestrian are determined based on the heterogeneous relationship graph according to a learnable prior vector, including: determining the attention allocation coefficient of the target pedestrian to each node in the heterogeneous relationship graph based on the learnable prior vector, wherein the prior vector is used to represent the prior attention coefficient of the interaction relationship between the obstacle and the pedestrian in the heterogeneous relationship graph, and the prior attention coefficient of the interaction relationship between the pedestrians; performing dot product attention based on the attention allocation coefficient of each node to obtain the heterogeneous interaction characteristics of the target pedestrian; fusing the heterogeneous interaction characteristics of the target pedestrian with the individual motion characteristics of the target pedestrian to obtain a fused feature; performing layer normalization and residual connection on the fused feature to obtain the comprehensive trajectory characteristics of the target pedestrian.

[0012] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the comprehensive trajectory features of the target pedestrian are input into a trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module, including: inputting the comprehensive trajectory features of the target pedestrian into a fully connected network of the trajectory prediction module to obtain the trajectory prediction result of the target pedestrian at the next moment; after determining the trajectory prediction results of all pedestrians, using the trajectory prediction results of all pedestrians as new pedestrian motion information to predict the pedestrian position at future moments.

[0013] The present invention also provides a pedestrian trajectory prediction device based on a heterogeneous relationship graph, comprising the following modules: an acquisition module, used to acquire target pedestrians, pedestrian motion information and obstacle information; a trajectory feature module, used to input the pedestrian motion information and the obstacle information into a heterogeneous relationship graph model, and obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model; a trajectory prediction module, used to input the comprehensive trajectory features of the target pedestrian into a trajectory prediction module, and obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and perform feature fusion with the heterogeneous relationship graph based on a learnable prior vector; the trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory features.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a pedestrian trajectory prediction method based on a heterogeneous relationship graph as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for predicting pedestrian trajectories based on a heterogeneous relationship graph as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned pedestrian trajectory prediction methods based on heterogeneous relationship graphs.

[0017] The pedestrian trajectory prediction method and device based on heterogeneous relationship graph provided by the present invention provide a rich data basis for subsequent trajectory prediction by obtaining target pedestrians, pedestrian motion information and obstacle information; the pedestrian motion information and obstacle information are input into the heterogeneous relationship graph model to construct a heterogeneous relationship graph of the target pedestrian; the heterogeneous relationship graph can represent the associations between different types of entities (such as pedestrians, obstacles), and how these associations change over time; by constructing the heterogeneous relationship graph, the complex interactive relationships between pedestrians and between pedestrians and obstacles can be captured; on the basis of constructing the heterogeneous relationship graph, a learnable prior vector is introduced for feature fusion to determine the comprehensive trajectory features; feature fusion improves the robustness of the model, and in the case of insufficient or ambiguous data, prior knowledge can help the model make more accurate processing; the comprehensive trajectory features of the target pedestrian are input into the trajectory prediction module to obtain the trajectory prediction result of the target pedestrian, thereby solving the defect of the pedestrian trajectory prediction method in the prior art that the prediction result is not accurate due to the failure to distinguish between obstacles and other pedestrians, and achieving more accurate pedestrian motion trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the pedestrian trajectory prediction method based on the heterogeneous relationship graph provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the overall framework of the pedestrian trajectory prediction method based on the heterogeneous relationship graph provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the heterogeneous relationship graph model structure provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the network structure of the obstacle feature extraction module provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the network structure of the global heterogeneous interactive feature fusion module provided by the present invention.

[0024] Figure 6 It is a module schematic diagram of a pedestrian trajectory prediction device based on a heterogeneous relationship graph provided by the present invention.

[0025] Figure 7 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In order to achieve the interactive relationship representation and pedestrian trajectory prediction in a multi-pedestrian environment, there are three types of processing methods in the existing methods:

[0028] Specific physical models or rules are used to model human-environment interactions, such as SocialForce, ORCA, etc. However, these methods involve many parameters, usually need to be adjusted individually, and may show great differences in different scenarios.

[0029] The interactive relationship between pedestrians and the environment is extracted by taking the motion state of pedestrians and environmental information as input. This type of method uses convolutional neural network (CNN) to extract environmental semantic features from bird's-eye view environmental images and high-definition semantic maps. However, the environment around pedestrians usually has a complex topological structure, and it is difficult for CNN to capture these features. At the same time, the receptive field of CNN is limited, and the use of CNN may cause the loss of geometric and semantic information. In addition, in these methods, the acquired environmental semantic information is usually fused with pedestrian trajectory features in a simple splicing method. This simple fusion method is difficult to capture the interaction relationship between environmental semantic information and pedestrian movement.

[0030] Without explicitly considering the surrounding environment information, obstacles are treated as static pedestrians when dealing with surrounding obstacles. However, the structural characteristics of obstacles are significantly different from those of pedestrians, and this simplification may lead to unstable prediction results.

[0031] According to the pedestrian trajectory prediction method based on heterogeneous relationship graph provided by the present invention, on the one hand, a feature representation method of environmental information vectorization is proposed, obstacle information is converted into graph structure data, and its structured features are extracted using graph neural network; on the other hand, a multimodal interaction feature fusion module based on heterogeneous relationship graph network is constructed, which realizes the fusion of heterogeneous interaction features between pedestrians and pedestrians, and pedestrians and obstacles.

[0032] Optionally, the pedestrian trajectory prediction method based on the heterogeneous relationship graph of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the pedestrian trajectory prediction method based on the heterogeneous relationship graph in the present embodiment being executed by a server.

[0033] Figure 1 is a flow chart of a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0034] Step 101, obtaining target pedestrians, pedestrian movement information and obstacle information;

[0035] In the embodiment of the present invention, basic information of the target pedestrian (such as identity, initial position, etc.), motion information of pedestrians in adjacent positions (for example, within a preset distance threshold) to the target pedestrian (such as speed, acceleration, direction, etc.) and obstacle information in the surrounding environment (such as the position, size, shape, etc. of the obstacle) are obtained. This information is the basis for constructing a heterogeneous relationship graph model and performing trajectory prediction in subsequent steps.

[0036] Step 102, inputting pedestrian motion information and obstacle information into a heterogeneous relationship graph model to obtain a comprehensive trajectory feature of a target pedestrian output by the heterogeneous relationship graph model;

[0037] refer to Figure 2 , Figure 2 It is a schematic diagram of the overall framework of the pedestrian trajectory prediction method based on the heterogeneous relationship graph provided by the present invention, which includes pedestrian motion information, obstacle information, a heterogeneous relationship graph model, and a trajectory prediction module; the heterogeneous relationship graph model includes an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module.

[0038] Among them, the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and obstacle information; and feature fusion is performed based on the learnable prior vector and the heterogeneous relationship graph.

[0039] The overall framework of the present invention is as follows Figure 2 As shown in the figure, in an environment with multiple pedestrians, by collecting pedestrian motion information and obstacle information around pedestrians, inputting them into the constructed heterogeneous relationship graph model, a comprehensive trajectory feature input trajectory prediction module is proposed, and finally the pedestrian trajectory prediction result output is obtained. Among them, the heterogeneous relationship graph model consists of an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module.

[0040] In the embodiment of the present invention, the heterogeneous relationship graph is a graph structure that can represent different types of nodes (such as pedestrians, obstacles) and the relationships between them (such as spatial position relationships, motion relationships). The heterogeneous relationship graph model dynamically adjusts the weights of nodes and edges in the graph according to pedestrian motion information and obstacle information to reflect dynamic changes in the actual scene.

[0041] The heterogeneous relationship graph model performs feature fusion based on a learnable prior vector (the learnable prior vector contains prior knowledge about the movement patterns of pedestrians, the impact of obstacles, etc.) and a heterogeneous relationship graph. Through this process, the heterogeneous relationship graph model can extract the comprehensive trajectory features of the target pedestrian, which includes the historical movement information of the target pedestrian and also reflects the impact of adjacent pedestrians and obstacles on the movement of the target pedestrian.

[0042] Among them, learnable prior vectors refer to vectors that can be used as prior knowledge input and learned by the machine learning model. These vectors usually contain additional information about the data or task, which helps the model better understand and process the data. The parameters of the learnable prior vectors are obtained during the training process.

[0043] Step 103, inputting the comprehensive trajectory features of the target pedestrian into a trajectory prediction module, and obtaining a trajectory prediction result of the target pedestrian output by the trajectory prediction module;

[0044] Among them, the trajectory prediction module is used to predict the pedestrian positions at multiple future moments based on comprehensive trajectory features.

[0045] In an embodiment of the present invention, the comprehensive trajectory features are input into a trajectory prediction module. Based on these features, the trajectory prediction module uses machine learning or deep learning algorithms (such as recurrent neural networks, convolutional neural networks, etc.) to predict the positions of pedestrians at multiple future moments. The prediction results can be the position coordinates of the pedestrians at one or more future time points.

[0046] Through the embodiments of the present invention, by obtaining target pedestrians, pedestrian motion information and obstacle information, a rich data basis is provided for subsequent trajectory prediction; the pedestrian motion information and obstacle information are input into a heterogeneous relationship graph model to construct a heterogeneous relationship graph of the target pedestrian; the heterogeneous relationship graph can represent the associations between different types of entities (such as pedestrians, obstacles), and how these associations change over time; by constructing a heterogeneous relationship graph, the complex interactive relationships between pedestrians and between pedestrians and obstacles can be captured; on the basis of constructing the heterogeneous relationship graph, a learnable prior vector is introduced for feature fusion to determine the comprehensive trajectory features; feature fusion improves the robustness of the model, and in the case of insufficient or ambiguous data, prior knowledge can help the model make more accurate processing; the comprehensive trajectory features of the target pedestrian are input into a trajectory prediction module to obtain a trajectory prediction result of the target pedestrian, thereby solving the defect of the pedestrian trajectory prediction method in the prior art that the prediction result is not accurate due to the failure to distinguish between obstacles and other pedestrians, and achieving more accurate pedestrian motion trajectory prediction.

[0047] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the heterogeneous relationship graph model includes: an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module;

[0048] refer to Figure 3 , Figure 3 It is a schematic diagram of the heterogeneous relationship graph model structure provided by the present invention, which includes: pedestrian social scene modeling, personal motion feature extraction module, obstacle (object) feature extraction module and global heterogeneous interaction feature fusion module (i.e., global heterogeneous feature fusion module).

[0049] in, They are used to represent different feature vectors corresponding to obstacle polygons. Represents the different eigenvectors corresponding to each line segment of the obstacle polygon, and Represents a linear combination of relative social features.

[0050] The pedestrian motion information and obstacle information are input into the heterogeneous relationship graph model to obtain the comprehensive trajectory characteristics of the target pedestrian output by the heterogeneous relationship graph model, which specifically includes the following steps.

[0051] Step 1: Extract motion features based on pedestrian motion information through an individual motion feature extraction module to obtain individual motion features of the target pedestrian.

[0052] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the individual motion feature extraction module includes: a multi-layer perceptron and a gated recurrent unit, which extracts motion features based on pedestrian motion information to obtain individual motion features of the target pedestrian, including:

[0053] Take each moment in the pedestrian motion information as the current moment, and perform the following steps to obtain the individual motion features of the target pedestrian at each moment:

[0054] The observation value of the target pedestrian in the pedestrian motion information is input into a multi-layer perceptron with an activation function to obtain an embedding vector output by the multi-layer perceptron;

[0055] The embedded vector is input into the gated recurrent unit to obtain the individual motion features of the target pedestrian at the current moment output by the gated recurrent unit.

[0056] In the individual motion feature extraction module, the embodiment of the present invention processes each pedestrian individually, such as Figure 3 As shown, at each moment t , the pedestrian at the current moment Observed value The input is fed into a multilayer perceptron (MLP) with a ReLU activation function, and the processed embedding vector is fed into a gated recurrent unit (GRU). The GRU updates the hidden state of the GRU at the current moment based on the input embedding vector and the previous GRU hidden state, so that the model can selectively retain the previous state information and integrate the new observation information. The feature vector output by the GRU The temporal motion characteristics of individual pedestrians are captured.

[0057] Multilayer Perceptron is a feedforward artificial neural network model that consists of an input layer, one or more hidden layers, and an output layer. The neurons in each layer are fully connected to the neurons in the next layer.

[0058] The input layer of the multi-layer perceptron is used to receive pedestrians At the moment t Observed value The hidden layer of a multilayer perceptron contains multiple neurons, each of which performs a linear transformation on the input through weights and biases, and then performs a nonlinear transformation through the ReLU activation function. The ReLU function is usually defined as f(x)=max(0,x) , which helps solve the gradient vanishing problem and speeds up training. The output layer of the multilayer perceptron outputs an embedding vector, which is the observation value Representation after processing by a multi-layer perceptron.

[0059] GRU is a variant of recurrent neural network (RNN) that solves the problem of gradient vanishing or gradient exploding that traditional RNNs are prone to when processing long sequences. GRU controls the flow of information through two gates (update gate and reset gate) to more effectively capture long-term dependencies in the sequence.

[0060] The input of GRU is the embedding vector output by MLP. The update gate of GRU determines how much previous information and how much new information should be retained in the hidden state at the current moment. The output of the update gate is a value between 0 and 1, which is used to weight the previous hidden state and the newly calculated state. The reset gate of GRU determines how much information of the hidden state at the previous moment should be forgotten in order to calculate the new state at the current moment. The output of the reset gate is also a value between 0 and 1, which is used to weight the contribution of the previous hidden state in calculating the new state. The hidden state update of GRU calculates the new hidden state based on the output of the update gate and reset gate, as well as the input embedding vector at the current moment.

[0061] Through the embodiments of the present invention, through multi-layer nonlinear transformation, MLP can extract higher-level feature representations from the original observation values; the embedding vector output by MLP is input into GRU, and the model can integrate observation information at different times to generate a feature vector that reflects the individual motion characteristics of the target pedestrian.

[0062] Step 2: Obstacle feature extraction module is used to extract obstacle features based on obstacle information to obtain surrounding obstacle features of the target pedestrian.

[0063] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the obstacle feature extraction module includes: an encoding layer, a multi-head self-attention module, a feedforward network layer and a maximum pooling layer;

[0064] Obstacle features are extracted based on obstacle information to obtain surrounding obstacle features of the target pedestrian, including:

[0065] Based on the position and speed of the target pedestrian, determine the fan-shaped field of view of the target pedestrian;

[0066] Determine the intersection area of ​​the fan-shaped field of view and the obstacle position in the obstacle information as the obstacle polygon;

[0067] Input the feature vector corresponding to each line segment of the obstacle polygon into the encoding layer to obtain the encoded feature vector output by the encoding layer;

[0068] Input the encoded feature vector into the multi-head self-attention module to obtain the query vector, key vector and value vector output by the multi-head self-attention module;

[0069] The query vector, key vector and value vector are input into the feedforward network layer, and the output of the feedforward network layer is residually connected and layer normalized with the encoded feature vector output by the encoding layer to obtain the segment features of each segment;

[0070] The line segment features of each line segment are input into the maximum pooling layer to obtain the surrounding obstacle features of the target pedestrian output by the maximum pooling layer.

[0071] The process of obstacle feature extraction and modeling is as follows: Figure 3 As shown, in this part, all operations are performed at the same time t To make the expression more concise, the subscripts are omitted. t . For example, the pedestrian's field of view is defined as a is the center point and the speed direction of pedestrians is a sector with a symmetry axis, and the radius of this sector is , the angle is ,The proposed method only considers obstacles that appear within the fan-shaped field of vision of pedestrians.

[0072] For pedestrians, first according to their location and speed , calculate its fan-shaped field of view Then determine the pedestrian's field of vision With surrounding obstacles Is there an intersection? If so, calculate the intersection area. .

[0073] The intersection area can be approximated as a polygon formed by a set of vertices connected in counterclockwise order, that is, ,in, Indicates k The two-dimensional coordinates of the vertices, K Represents the number of vertices of the polygon. The two-dimensional position of The direction of its velocity is for x Axis, establish a rectangular coordinate system. Each vertex of is converted to the pedestrian coordinate system. In order to better encode the obstacles within the field of view, it is necessary to extract reasonable and effective structural features from the line segments that make up the obstacles. The proposed method takes into account the length, direction, position and distance to the pedestrian of the line segment. Each line segment of k , the following feature vector is constructed :

[0074]

[0075] in, and Respectively represent k The coordinates of the starting point and end point of the line segment in the pedestrian coordinate system, Representative k The coordinates of the point on the line segment closest to the pedestrian, Represents pedestrians to k The shortest distance between line segments, Represents the Euclidean distance between the starting point and the end point of the line segment, Represents the unit vector of the line segment in the pedestrian coordinate system, that is, the direction of the line segment.

[0076] like Figure 3 As shown, in order to extract the spatial structural features of obstacles, for each obstacle polygon in the pedestrian field of view , all line segment features belonging to the polygon Constructed as a fully connected subgraph, each node in the graph represents the characteristics of each line segment .

[0077] refer to Figure 4 , Figure 4 It is a schematic diagram of the network structure of the obstacle feature extraction module provided by the present invention, which includes: input (segment feature), encoding layer, multi-head attention (module), feedforward network, layer normalization, pooling layer and output (obstacle feature).

[0078] The network structure of the obstacle feature extraction module is as follows: Figure 3 As shown in Figure 1, it mainly includes an encoding layer, a multi-head self-attention module, a feedforward network layer, and a maximum pooling layer. First, in order to better capture the local features of each line segment, the feature vector of each line segment is The input is fed into an encoding layer consisting of an MLP, and then a multi-head attention module is used to extract the spatial semantic information between the line segments. The multi-head attention allows the model to learn different attention weights in different subspaces. For each line segment’s feature vector , using the self-attention mechanism to map it into a query vector (Query) , key vector (Key) and value vector (Value) In each attention head of the multi-head attention module, the attention coefficient Indicates The line segment k The influence of the line segments can be calculated by the following formula:

[0079]

[0080] in, , and are the learnable parameter matrices for query, key, and value, respectively, is a matrix and The dimension of Represents matrix transpose, line segment k Aggregate the structural information of other line segments through dot product operations. Represents the target aggregation value, which depends on the attention coefficient With value vector :

[0081]

[0082] like Figure 4 As shown in Figure 1, the output of the multi-head attention module is passed to a feedforward neural network (or feedforward network) composed of a single-layer MLP with a ReLU activation function. The output of the feedforward layer is residually connected with the output of the encoding layer, and layer normalization is performed to obtain the features of each line segment. Finally, in order to obtain the characteristics of the obstacle level , all node features under each obstacle are input into a maximum pooling layer (MaxPooling):

[0083]

[0084] in, Features that represent obstacle levels (i.e., surrounding obstacle features), Represents the line segment features of each line segment, Represents a maximum pooling layer.

[0085] Through the embodiment of the present invention, the fan-shaped field of view is determined by combining the position and speed of the target pedestrian. This method can dynamically reflect the change of the field of view of the target pedestrian during movement, making the extraction of obstacle features more in line with the actual scene. The introduction of the multi-head self-attention module can capture the intrinsic connection and correlation between line segment features, and can enhance the model's understanding and response capabilities to complex obstacle situations, thereby improving the accuracy and comprehensiveness of feature extraction.

[0086] Step 3: Through the global heterogeneous feature fusion module, a heterogeneous relationship graph of the target pedestrian is constructed based on individual motion features and surrounding obstacle features.

[0087] The node set of the heterogeneous relationship graph includes: individual motion features and surrounding obstacle features, and the edge set of the heterogeneous relationship graph includes: interaction between obstacles and pedestrians and interaction between pedestrians.

[0088] like Figure 3 As shown in the figure, for each pedestrian, the social scene in which he is located is abstracted into a heterogeneous graph , where the node set Represents the feature vectors of each entity in the scene. For more convenient representation, the index of the obstacle feature is remapped, that is, Then use the symbol To unify the index of each entity node in the graph, according to its node type, the node set can be divided into individual motion features of pedestrians And surrounding obstacle characteristics , Represents a set of edges in a heterogeneous graph. Due to different types of source nodes, the edges connecting source nodes and target nodes can be divided into two types: the interaction relationship between obstacles and pedestrians and the interaction relationship between pedestrians. Since the obstacles within the field of view of each pedestrian are different and the interaction relationship between pedestrians is asymmetric, in the present invention, the interaction relationship between different entities is modeled as a directed edge.

[0089] Step 4: Through the global heterogeneous feature fusion module, the comprehensive trajectory characteristics of the target pedestrian are determined based on the heterogeneous relationship graph according to the learnable prior vector.

[0090] refer to Figure 5 , Figure 5 It is a schematic diagram of the network structure of the global heterogeneous interactive feature fusion module (i.e., the global heterogeneous feature fusion module) provided by the present invention, including: a multi-layer perceptron (MLP), multi-head attention, a feedforward network, and layer normalization.

[0091] In the embodiment of the present invention, the global heterogeneous feature fusion module is used for extracting the interaction relationship between pedestrians, extracting the interaction relationship between pedestrians and obstacles, and fusing heterogeneous interaction features.

[0092] In order to better capture the interactions between pedestrians, the asymmetric interaction relationship between pedestrians is considered, and the following two sets of interaction features are defined: and , which are used to describe the interaction relationship between pedestrians and the relative characteristics of pedestrians required to predict their future trajectories. Indicates that at the target node (pedestrian) From the perspective of relative social characteristics. Indicates that in the world coordinate system, the neighbor nodes Relative to the target node The relative social characteristics of , are defined as follows:

[0093]

[0094] In Features middle, Indicates the target node and its neighbor nodes The distance at the same time, Represents neighbor nodes To the target node The cosine of the azimuth, Indicates that at the target node In the local coordinate system of Relative to the target node speed. In the characteristics middle, Indicates that in the world coordinate system, the neighbor node Relative to the target node location, Represents neighbor nodes Relative to the target node speed.

[0095] When modeling the interaction relationship between pedestrians, the proposed heterogeneous relationship graph model not only considers the long-term dependency between pedestrian trajectories and takes the individual historical motion characteristics of pedestrians as nodes in the graph, but also considers the asymmetric relative spatial relationship between pedestrians, models the constructed relative social characteristics of pedestrians as directed connected edges, and uses the self-attention mechanism to extract the social semantic information between pedestrian nodes, and treats the target node as the node. Individual movement characteristics Mapped to query vector through linear transformation , the key vector Sum value vector Represented as individual motion features of neighboring nodes and relative social characteristics A linear combination of , unnormalized attention coefficients Represents neighbor nodes For the target node The influence of can be calculated by the following formula:

[0096]

[0097] in, W represents the learnable parameter matrix, Represents the transposed matrix, using The constructed key vector It can capture the directed interaction between neighbor nodes and target nodes. The constructed value vector It can capture the impact of the interaction between pedestrians on the future trajectory of the target pedestrian. A learnable parameter matrix representing the (target node) query vector; A learnable parameter matrix representing the key vectors of neighbor nodes; a learnable parameter matrix representing the bond vectors of relative social characteristics; A learnable parameter matrix representing the value vector of neighbor nodes; A learnable parameter matrix representing a vector of values ​​of relative social features.

[0098] The interaction between pedestrians and obstacles is modeled using a cross-attention mechanism. It should be noted that neighbor nodes include obstacle nodes. When the neighbor is an obstacle, the neighbor node can be regarded as an obstacle node. Its features are transformed by linear transformation. Mapped to key vectors Sum value vector , target node For each obstacle node around The unnormalized attention coefficient It can be expressed as:

[0099]

[0100] in, represents the weight matrix of the key vector, represents the weight matrix of the value vector, Represents the transposed matrix.

[0101] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, a comprehensive trajectory feature of a target pedestrian is determined based on a learnable prior vector and a heterogeneous relationship graph, including:

[0102] Based on a learnable prior vector, determine the attention allocation coefficient of the target pedestrian to each node in the heterogeneous relationship graph, where the prior vector is used to represent the prior attention coefficient of the interaction relationship between obstacles and pedestrians in the heterogeneous relationship graph, as well as the prior attention coefficient of the interaction relationship between pedestrians;

[0103] Based on the attention allocation coefficient of each node, dot product attention is performed to obtain the heterogeneous interaction features of the target pedestrian;

[0104] The heterogeneous interaction features of the target pedestrians are fused with the individual motion features of the target pedestrians to obtain fused features;

[0105] The fused features are layer-normalized and residual-connected to obtain the comprehensive trajectory features of the target pedestrian.

[0106] In the embodiment of the present invention, the prior vector is used to represent the prior attention coefficients of the interaction relationship between obstacles and pedestrians and the interaction relationship between pedestrians in the heterogeneous relationship graph; the prior vector can be used to determine the attention allocation coefficient of the target pedestrian to each node in the heterogeneous relationship graph. These coefficients reflect the degree of attention of the target pedestrian to different nodes when making decisions.

[0107] The target pedestrian’s prior attention allocation to different entities in the surrounding environment is usually different. Pedestrian behavior is usually more dynamic and social, while obstacles are usually relatively static. Therefore, compared with obstacles, the behavior of surrounding pedestrians may be more likely to attract the target pedestrian’s attention.

[0108] Based on this, the embodiment of the present invention adds a learnable prior vector , The prior attention coefficients of two interaction relationships (pedestrians and obstacles, pedestrians and pedestrians) are used to describe the differences in the prior attention allocation of the target pedestrian to different interaction relationships. The proposed method uses a multi-head attention module to extract the heterogeneous interaction features between pedestrians and different entities. In each attention head, the target pedestrian For each surrounding node (including obstacles and pedestrians) The attention allocation coefficient It can be expressed as:

[0109]

[0110] in, Represents the source node The type of There are two possible values: obstacle or pedestrian. Represents the prior attention coefficient of the corresponding interaction relationship. Represents the dimension of the key vector, the target pedestrian Through a dot product attention operation, the interaction information of different types of neighboring nodes is aggregated to obtain their heterogeneous interaction features. Represents the heterogeneous interaction features of the target pedestrian:

[0111]

[0112] The output of the multi-head attention layer is then passed to a feed-forward neural network composed of MLP, and the output of the feed-forward neural network is combined with the individual motion characteristics of the pedestrians. Residual connection is performed to fuse individual motion features and heterogeneous interactive semantic features. The fused features are input into a feedforward network with layer normalization before and after, and residual connection is performed at the output end to obtain the comprehensive trajectory features of pedestrians. .

[0113] Through the embodiments of the present invention, the comprehensive trajectory features of the target pedestrian obtained include the interaction information between the target pedestrian and other elements in the environment, as well as its own motion information. At the same time, the stability and expression ability of the features are improved through layer normalization and residual connection.

[0114] According to a pedestrian trajectory prediction method based on a heterogeneous relationship graph provided by the present invention, the comprehensive trajectory features of the target pedestrian are input into a trajectory prediction module, and the trajectory prediction result of the target pedestrian output by the trajectory prediction module is obtained, including:

[0115] The comprehensive trajectory features of the target pedestrian are input into the fully connected network of the trajectory prediction module to obtain the trajectory prediction result of the target pedestrian at the next moment;

[0116] After the trajectory prediction results of all pedestrians are determined, the trajectory prediction results of all pedestrians are used as new pedestrian motion information to predict the pedestrian positions at future moments.

[0117] In an embodiment of the present invention, the comprehensive trajectory features of the target pedestrian are input into the fully connected network (FCN) of the trajectory prediction module. A fully connected network is a basic neural network structure in which each neuron in each layer is connected to all neurons in the previous layer. In this scenario, the fully connected network is used to extract information related to the trajectory at the next moment from the input comprehensive trajectory features. Through the calculation of the fully connected network, the trajectory prediction result of the target pedestrian at the next moment can be obtained. The trajectory prediction result includes parameters such as position coordinates, speed, and acceleration.

[0118] The trajectory prediction module uses a fully connected network to predict the pedestrian trajectory at the next moment. Input the fully connected layer to get the pedestrian of t+1 After obtaining the next moment position of all pedestrians, add them to the observed trajectory and input them into the network for prediction again, we can get t+2 In actual application, the above process can be repeated as many times as needed to obtain the predicted trajectory of multiple future moments.

[0119] For example, pedestrians At a previous moment (e.g. from tn arrive t time, n The comprehensive trajectory features of the trajectory prediction module (where ) are input into the fully connected layer of the trajectory prediction module. The fully connected layer (also called dense layer or fully connected neural network layer) receives these input features and extracts information related to the trajectory at the next moment through a series of weighted summation, activation function and other operations. After processing by the fully connected layer, the pedestrian exist t+1 The trajectory prediction position at the moment.

[0120] Getting pedestrians exist t+1 After the predicted position at time , it is added to the observation trajectory. At this point, the observation trajectory has been updated to include tn arrive t+1 Pedestrian location information at each moment.

[0121] Next, the updated observation trajectory is used as a new input and input into the fully connected network again for prediction. This time the prediction target is the pedestrian exist t+2 The trajectory position at the moment.

[0122] By repeating this process, pedestrians can be obtained one by one. At multiple future moments (e.g. t+3 , t+4 , ...) of the predicted trajectories.

[0123] The above process is not only applicable to a single pedestrian , and can also be extended to all pedestrians in the scene. Repeating the above steps for each pedestrian can obtain the predicted trajectories of all pedestrians in the entire scene at multiple moments in the future.

[0124] Through the embodiments of the present invention, the comprehensive trajectory features of pedestrians are input into the fully connected network for processing, which can capture the complexity of pedestrian movement patterns, thereby achieving accurate prediction of pedestrian future trajectories; by repeating the prediction process multiple times, the predicted trajectories of pedestrians at multiple moments in the future can be obtained; it is not only applicable to the trajectory prediction of a single pedestrian, but can also be extended to all pedestrians in the scene. By processing the trajectory features of multiple pedestrians at the same time, the predicted trajectories of all pedestrians in the entire scene at multiple moments in the future can be obtained.

[0125] Through the above-mentioned embodiments of the present invention, a vectorized obstacle feature representation method is proposed, which converts obstacle information into graph structure data, and uses graph neural network to extract its structured features, thereby retaining the structural features and spatial relationships of obstacles. The complex interaction relationship between different types of entities (such as pedestrians and obstacles, pedestrians and pedestrians) is considered more comprehensively, and a heterogeneous graph network is used to model the process of pedestrians interacting with surrounding pedestrians and the environment. Ultimately, a more accurate pedestrian motion trajectory prediction effect is achieved.

[0126] The pedestrian trajectory prediction device based on the heterogeneous relationship graph provided by the present invention is described below. The pedestrian trajectory prediction device based on the heterogeneous relationship graph described below and the pedestrian trajectory prediction method based on the heterogeneous relationship graph described above can be referred to each other.

[0127] refer to Figure 6 , Figure 6 It is a module schematic diagram of a pedestrian trajectory prediction device based on a heterogeneous relationship graph provided by the present invention.

[0128] The acquisition module 601 is used to acquire target pedestrians, pedestrian movement information and obstacle information;

[0129] The trajectory feature module 602 is used to input the pedestrian motion information and obstacle information into the heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model;

[0130] The trajectory prediction module 603 is used to input the comprehensive trajectory features of the target pedestrian into the trajectory prediction module, and obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module;

[0131] Among them, the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and obstacle information; and feature fusion is performed based on the learnable prior vector and the heterogeneous relationship graph;

[0132] The trajectory prediction module is used to predict the pedestrian positions at multiple future moments based on comprehensive trajectory features.

[0133] Specifically, the pedestrian trajectory prediction device based on the heterogeneous relationship graph provided by the present invention can implement all the method steps implemented by the above-mentioned pedestrian trajectory prediction method embodiment based on the heterogeneous relationship graph, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0134] Figure 7 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the pedestrian trajectory prediction method based on the heterogeneous relationship graph, the method comprising: obtaining the target pedestrian, pedestrian motion information and obstacle information; inputting the pedestrian motion information and obstacle information into the heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model; inputting the comprehensive trajectory features of the target pedestrian into the trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct the heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and performing feature fusion with the heterogeneous relationship graph based on the learnable prior vector; the trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory features.

[0135] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0136] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pedestrian trajectory prediction method based on the heterogeneous relationship graph provided by the above-mentioned methods, the method including: obtaining the target pedestrian, pedestrian motion information and obstacle information; inputting the pedestrian motion information and obstacle information into a heterogeneous relationship graph model to obtain the comprehensive trajectory characteristics of the target pedestrian output by the heterogeneous relationship graph model; inputting the comprehensive trajectory characteristics of the target pedestrian into a trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and obstacle information; and feature fusion is performed based on a learnable prior vector and the heterogeneous relationship graph; the trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory characteristics.

[0137] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the pedestrian trajectory prediction method based on the heterogeneous relationship graph provided by the above-mentioned methods, the method comprising: obtaining the target pedestrian, pedestrian motion information and obstacle information; inputting the pedestrian motion information and obstacle information into a heterogeneous relationship graph model to obtain the comprehensive trajectory characteristics of the target pedestrian output by the heterogeneous relationship graph model; inputting the comprehensive trajectory characteristics of the target pedestrian into a trajectory prediction module to obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; wherein the heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and obstacle information; and feature fusion is performed based on a learnable prior vector and the heterogeneous relationship graph; the trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory characteristics.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pedestrian trajectory prediction method based on heterogeneous relationship graph, characterized in that: include: Obtain target pedestrians, pedestrian movement information, and obstacle information; Inputting the pedestrian motion information and the obstacle information into a heterogeneous relationship graph model to obtain a comprehensive trajectory feature of the target pedestrian output by the heterogeneous relationship graph model; Inputting the comprehensive trajectory features of the target pedestrian into a trajectory prediction module to obtain a trajectory prediction result of the target pedestrian output by the trajectory prediction module; The heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and to perform feature fusion with the heterogeneous relationship graph based on a learnable prior vector; The trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory features; The heterogeneous relationship graph model includes: an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module; The pedestrian movement information and the obstacle information are input into a heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model, including: By means of the individual motion feature extraction module, motion features are extracted based on the pedestrian motion information to obtain the individual motion features of the target pedestrian; by means of the obstacle feature extraction module, obstacle features are extracted based on the obstacle information to obtain the surrounding obstacle features of the target pedestrian; by means of the global heterogeneous feature fusion module, a heterogeneous relationship graph of the target pedestrian is constructed based on the individual motion features and the surrounding obstacle features, wherein the node set of the heterogeneous relationship graph includes: individual motion features and surrounding obstacle features, and the edge set of the heterogeneous relationship graph includes: interaction relationships between obstacles and pedestrians and interaction relationships between pedestrians; by means of the global heterogeneous feature fusion module, the comprehensive trajectory features of the target pedestrian are determined based on the heterogeneous relationship graph according to a learnable prior vector; The obstacle feature extraction module includes: an encoding layer, a multi-head self-attention module, a feedforward network layer and a maximum pooling layer; The extracting obstacle features based on the obstacle information to obtain surrounding obstacle features of the target pedestrian includes: Based on the position of the target pedestrian and the speed of the target pedestrian, determine the fan-shaped field of view of the target pedestrian; determine the intersection area of ​​the fan-shaped field of view and the obstacle position in the obstacle information as an obstacle polygon; input the feature vector corresponding to each line segment of the obstacle polygon into the encoding layer to obtain the encoded feature vector output by the encoding layer; input the encoded feature vector into the multi-head self-attention module to obtain the query vector, key vector and value vector output by the multi-head self-attention module; input the query vector, the key vector and the value vector into the feedforward network layer, and perform residual connection and layer normalization on the output of the feedforward network layer and the encoded feature vector output by the encoding layer to obtain the line segment features of each line segment; input the line segment features of each line segment into the maximum pooling layer to obtain the surrounding obstacle features of the target pedestrian output by the maximum pooling layer.

2. The pedestrian trajectory prediction method based on heterogeneous relationship graph according to claim 1 is characterized in that: The individual motion feature extraction module includes: a multi-layer perceptron and a gated recurrent unit. The motion feature extraction based on the pedestrian motion information to obtain the individual motion feature of the target pedestrian includes: Taking each moment in the pedestrian motion information as the current moment, perform the following steps to obtain the individual motion features of the target pedestrian at each moment: Inputting the observation value of the target pedestrian in the pedestrian motion information into the multi-layer perceptron with an activation function to obtain an embedding vector output by the multi-layer perceptron; The embedding vector is input into the gated recurrent unit to obtain the individual motion features of the target pedestrian at the current moment output by the gated recurrent unit.

3. The pedestrian trajectory prediction method based on heterogeneous relationship graph according to claim 1 is characterized in that: The step of determining the comprehensive trajectory features of the target pedestrian based on the heterogeneous relationship graph according to the learnable prior vector includes: Based on a learnable prior vector, determining an attention allocation coefficient of the target pedestrian to each node in the heterogeneous relationship graph, wherein the prior vector is used to represent the prior attention coefficient of the interaction relationship between the obstacle and the pedestrian in the heterogeneous relationship graph, and the prior attention coefficient of the interaction relationship between the pedestrians; Performing dot product attention based on the attention allocation coefficient of each node to obtain the heterogeneous interaction features of the target pedestrian; fusing the heterogeneous interaction features of the target pedestrian with the individual motion features of the target pedestrian to obtain a fused feature; The fusion features are layer-normalized and residual-connected to obtain the comprehensive trajectory features of the target pedestrian.

4. The pedestrian trajectory prediction method based on heterogeneous relationship graph according to claim 1 is characterized in that: The step of inputting the comprehensive trajectory features of the target pedestrian into a trajectory prediction module to obtain a trajectory prediction result of the target pedestrian output by the trajectory prediction module includes: Inputting the comprehensive trajectory features of the target pedestrian into the fully connected network of the trajectory prediction module to obtain the trajectory prediction result of the target pedestrian at the next moment; After the trajectory prediction results of all pedestrians are determined, the trajectory prediction results of all pedestrians are used as new pedestrian motion information to predict the positions of pedestrians at future moments.

5. A pedestrian trajectory prediction device based on a heterogeneous relationship graph, characterized in that: include: An acquisition module is used to acquire target pedestrians, pedestrian movement information, and obstacle information; A trajectory feature module, used to input the pedestrian motion information and the obstacle information into a heterogeneous relationship graph model to obtain a comprehensive trajectory feature of the target pedestrian output by the heterogeneous relationship graph model; A trajectory prediction module, used to input the comprehensive trajectory features of the target pedestrian into the trajectory prediction module, and obtain the trajectory prediction result of the target pedestrian output by the trajectory prediction module; The heterogeneous relationship graph model is used to construct a heterogeneous relationship graph of the target pedestrian based on the pedestrian motion information and the obstacle information; and perform feature fusion with the heterogeneous relationship graph based on a learnable prior vector; The trajectory prediction module is used to predict the pedestrian position at multiple future moments based on the comprehensive trajectory features; The heterogeneous relationship graph model includes: an individual motion feature extraction module, an obstacle feature extraction module and a global heterogeneous feature fusion module; The pedestrian movement information and the obstacle information are input into a heterogeneous relationship graph model to obtain the comprehensive trajectory features of the target pedestrian output by the heterogeneous relationship graph model, including: By means of the individual motion feature extraction module, motion features are extracted based on the pedestrian motion information to obtain the individual motion features of the target pedestrian; by means of the obstacle feature extraction module, obstacle features are extracted based on the obstacle information to obtain the surrounding obstacle features of the target pedestrian; by means of the global heterogeneous feature fusion module, a heterogeneous relationship graph of the target pedestrian is constructed based on the individual motion features and the surrounding obstacle features, wherein the node set of the heterogeneous relationship graph includes: individual motion features and surrounding obstacle features, and the edge set of the heterogeneous relationship graph includes: interaction relationships between obstacles and pedestrians and interaction relationships between pedestrians; by means of the global heterogeneous feature fusion module, the comprehensive trajectory features of the target pedestrian are determined based on the heterogeneous relationship graph according to a learnable prior vector; The obstacle feature extraction module includes: an encoding layer, a multi-head self-attention module, a feedforward network layer and a maximum pooling layer; The extracting obstacle features based on the obstacle information to obtain surrounding obstacle features of the target pedestrian includes: Based on the position of the target pedestrian and the speed of the target pedestrian, determine the fan-shaped field of view of the target pedestrian; determine the intersection area of ​​the fan-shaped field of view and the obstacle position in the obstacle information as an obstacle polygon; input the feature vector corresponding to each line segment of the obstacle polygon into the encoding layer to obtain the encoded feature vector output by the encoding layer; input the encoded feature vector into the multi-head self-attention module to obtain the query vector, key vector and value vector output by the multi-head self-attention module; input the query vector, the key vector and the value vector into the feedforward network layer, and perform residual connection and layer normalization on the output of the feedforward network layer and the encoded feature vector output by the encoding layer to obtain the line segment features of each line segment; input the line segment features of each line segment into the maximum pooling layer to obtain the surrounding obstacle features of the target pedestrian output by the maximum pooling layer.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the pedestrian trajectory prediction method based on the heterogeneous relationship graph as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pedestrian trajectory prediction method based on a heterogeneous relationship graph as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the pedestrian trajectory prediction method based on a heterogeneous relationship graph as described in any one of claims 1 to 4 is implemented.

Images