Automatic driving track prediction method and system, storage medium and equipment

Abstract

The invention provides an automatic driving track prediction method and system, a storage medium and equipment, and relates to the field of automatic driving, the method is applied to a deep learning network model, and the deep learning network model comprises a VectorNet network, a graph neural network GNN, a FusionNet fusion module and a GRU decoder. By setting a complete deep learning network model, the method aims to improve the ability of recognizing lane structure features in complex urban roads and enhance interaction between data, so that the accuracy of target vehicle trajectory prediction is improved, and the driving safety is ensured.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving trajectory prediction method, system, storage medium and device. Background Art

[0002] As vehicles become more intelligent, trajectory prediction modules become more and more important.

[0003] In the existing technology, traditional machine learning methods generally include hidden Markov models (HMM), dynamic Bayesian networks (DBN), etc. These methods make predictions by learning patterns in historical data. Their advantage is that they can process time series data, but they have limitations in processing complex interactions and uncertainties, which in turn affects the accuracy of trajectory prediction during vehicle autonomous driving. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide an automatic driving trajectory prediction method, system, storage medium and device, which are used to improve the accuracy of trajectory prediction during vehicle automatic driving.

[0005] On one hand, the present invention provides an autonomous driving trajectory prediction method, which is applied to a deep learning network model, wherein the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module, and a GRU decoder; the method includes: Based on the VectorNet network, the intelligent agent trajectory and the map features of the original map are obtained, the intelligent agent includes a vehicle, the intelligent agent trajectory and the map features are respectively converted into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and the plurality of trajectory vectors or map vectors are sequentially connected to generate a polyline, the polyline includes a trajectory polyline and a map polyline, the trajectory polyline is an approximation of the intelligent agent trajectory, and the map polyline is an approximation of the original map; the VectorNet network includes a local subgraph encoder and a global graph encoder; The local subgraph encoder encodes the historical trajectory and nearby road segments of each traffic participant into local features to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant in the local area and its related road segments are constructed as an independent subgraph. The subgraphs interact with each other through the graph neural network GNN to transmit information. The subgraphs include nodes, which are the features of the trajectory points and road segments of the agent. The features of the trajectory points and road segments of the agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs. Inputting the global interaction graph into a FusionNet fusion module to fuse dynamic objects and scene information, performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features; The local embedded features are fused and encoded with the global embedded features based on the GRU decoder, and the features fused by the GRU decoder are decoded using a multi-layer perceptron MLP to generate a multimodal prediction result and assign a confidence score to each possible trajectory. The final trajectory is obtained according to the confidence score to achieve trajectory prediction.

[0006] The above-mentioned autonomous driving trajectory prediction method sets up a complete deep learning network model, which includes VectorNet network, graph neural network GNN, FusionNet fusion module and GRU decoder, aiming to improve the ability to identify lane structure features in complex urban roads, enhance the interaction between data, thereby improving the accuracy of target vehicle trajectory prediction and ensuring driving safety. Specifically, by building a deep learning network model based on VectorNet network, unlike the traditional road feature extraction based on rasterization and image, the graph neural network GNN is introduced to process local information and global information, and the features of all sub-graphs are integrated through further message passing mechanism at the global level to capture the association relationship and context information between sub-graphs. This hierarchical aggregation method enables the VectorNet network to effectively capture the subtle structure and global relationship in roads and scenes. The FusionNet fusion module performs information fusion at different levels through a multi-level feature interaction mechanism, which can fully capture the complex relationship between multimodal data and enhance the expressiveness and robustness of the deep learning network model.

[0007] In addition, the automatic driving trajectory prediction method according to the present invention may also have the following additional technical features: Furthermore, the steps of converting the agent trajectory and the map feature into a plurality of trajectory vectors and map vectors respectively according to a preset time interval and a preset spatial interval include: An original map is obtained, a starting point and a direction are selected according to the original map, map key points are uniformly sampled from a spline with a preset spatial interval according to the starting point and the direction, and adjacent map key points are sequentially connected to obtain a map vector to obtain a map feature; trajectory key points are sampled according to a preset time interval, and the sampled trajectory key points are sequentially connected to form a trajectory vector, wherein the map vector and the trajectory vector both include a plurality of nodes, each node is an element in a subgraph, and a node represents a historical trajectory point of a traffic participant or a road segment.

[0008] Furthermore, each vector belonging to a polyline is represented as a node, and the feature of the node is defined as: ; v i represents a vector; Indicates the starting point; Indicates the end point; d Indicates 2D coordinates (x, y) or 3D coordinates (x, y, z) for representation; a i Represents attribute features, including object type, trajectory timestamp, road feature type, or lane speed limit; j Indicates p j The integer ID of v i ∈p j , p j A node feature representing an integer ID.

[0009] Furthermore, single-layer subgraph propagation is defined as: ; In the formula, Representation Node i In the l Node characteristics of the layer; Representation Node i In the l +1 layer node features; Represents neighbor nodes j In the l Node characteristics of the layer; g enc (﹒) represents a multi-layer perceptron MLP, whose weights are shared among all nodes; f agg (﹒) represents a maximum pooling operation to obtain the most obvious behavioral characteristics of neighboring agents; f rel (﹒) represents the relationship operator between a node and its neighbors.

[0010] Furthermore, the method also includes: normalizing the coordinates of the trajectory vector and the map vector so that the position of the input node feature relative to the target intelligent body remains unchanged.

[0011] Furthermore, the step of inputting the global interaction graph into the FusionNet fusion module to fuse the dynamic object and scene information includes: The high-order interactions on node features are modeled through the global interaction graph, and the modeling equation is: ; In the formula, Represented as a node i In the l The feature set of the layer; Representation Node i In the l +1 layer feature set; GNN(﹒) represents a single layer of graph neural network GNN; A The adjacency matrix representing the feature set of the polyline nodes.

[0012] Another aspect of the present invention provides an autonomous driving trajectory prediction system, which is applied to a deep learning network model, wherein the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module, and a GRU decoder; the system includes: An acquisition module is used to acquire the trajectory of an intelligent agent and the map features of an original map based on a VectorNet network, wherein the intelligent agent includes a vehicle, and the trajectory of the intelligent agent and the map features are respectively converted into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and the plurality of trajectory vectors or map vectors are sequentially connected to generate a polyline, wherein the polyline includes a trajectory polyline and a map polyline, wherein the trajectory polyline is an approximation of the trajectory of the intelligent agent, and the map polyline is an approximation of the original map; the VectorNet network includes a local subgraph encoder and a global graph encoder; An interaction module is used to encode the historical trajectory and nearby road segments of each traffic participant into local features through a local subgraph encoder to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant and its related road segments in the local area are constructed as an independent subgraph. The subgraphs interact with each other through a graph neural network GNN to transmit information. The subgraphs include nodes, which are the features of the trajectory points and road segments of the agent. The features of the trajectory points and road segments of the agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs. A fusion module, used for inputting the global interaction graph into a FusionNet fusion module to fuse dynamic objects and scene information, and performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features; The prediction module is used to fuse and encode the local embedded features with the global embedded features based on the GRU decoder, decode the features fused by the GRU decoder using a multi-layer perceptron MLP, generate a multimodal prediction result and assign a confidence score to each possible trajectory, and obtain the final trajectory according to the confidence score to achieve trajectory prediction.

[0013] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic driving trajectory prediction method as described above.

[0014] On the other hand, the present invention also provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the automatic driving trajectory prediction method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the automatic driving trajectory prediction method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the principle structure of the present invention.

[0016] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0019] In recent years, deep learning methods have made significant progress in trajectory prediction. Graph neural networks (GNNs) are particularly suitable for processing non-Euclidean spatial data and can capture the interactive relationships between traffic participants. Graph neural networks (GNNs) can use road vectors as information sources to extract features, thereby better preserving the detailed features of the road. This feature further improves the application effect of graph neural networks (GNNs) in road trajectory prediction, making improving prediction accuracy a key development direction.

[0020] In order to improve the accuracy of trajectory prediction during vehicle autonomous driving, the present application provides an autonomous driving trajectory prediction method, system, storage medium and device, which is designed to improve the ability to identify lane structure features in complex urban roads, enhance the interaction between data, and thus improve the accuracy of target vehicle trajectory prediction and ensure driving safety by setting a complete deep learning network model, which includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module and a GRU decoder. Specifically, by building a deep learning network model based on the VectorNet network, unlike the traditional road feature extraction based on rasterization and image, a graph neural network GNN is introduced to process local information and global information, and the features of all subgraphs are integrated at the global level through a further message passing mechanism to capture the association relationship and contextual information between subgraphs. This hierarchical aggregation method enables the VectorNet network to effectively capture the subtle structure and global relationship in roads and scenes. The FusionNet fusion module performs information fusion at different levels through a multi-level feature interaction mechanism, which can fully capture the complex relationship between multimodal data and enhance the expressiveness and robustness of the deep learning network model.

[0021] In order to facilitate understanding of the present invention, several embodiments of the present invention are provided below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0022] Embodiment 1 See also Figure 1 , which shows the automatic driving trajectory prediction method in the first embodiment of the present invention, applied to a deep learning network model, the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module and a GRU decoder, and the method includes steps S101 to S104: S101. Acquire the trajectory of an intelligent agent and map features of an original map based on a VectorNet network, wherein the intelligent agent includes a vehicle. Convert the trajectory of the intelligent agent and the map features into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and sequentially connect the plurality of trajectory vectors or map vectors to generate a polyline, wherein the polyline includes a trajectory polyline and a map polyline, wherein the trajectory polyline is an approximation of the trajectory of the intelligent agent, and the map polyline is an approximation of the original map; the VectorNet network includes a local sub-graph encoder and a global graph encoder.

[0023] like Figure 2As shown, in this embodiment, the original map is obtained, and a starting point and a direction are selected according to the original map to vectorize the intelligent body trajectory and map features, specifically: map key points are uniformly sampled from splines with preset spatial intervals according to the starting point and the direction, and adjacent map key points are sequentially connected to obtain map vectors to obtain map features; trajectory key points are sampled according to preset time intervals, and the sampled trajectory key points are sequentially connected into trajectory vectors, and the map vector and the trajectory vector both include multiple nodes, each node is an element in a subgraph, and the node represents a historical trajectory point of a traffic participant or a road segment.

[0024] In this embodiment, each vector belonging to a polyline is represented as a node, and the feature of the node is defined as: ; v i represents a vector; Indicates the starting point; Indicates the end point; d Represents 2D coordinates (x, y) or 3D coordinates (x, y, z); a i Represents attribute features, including object type, trajectory timestamp, road feature type, or lane speed limit; j Indicates p j The integer ID of v i ∈p j , p j A node feature representing an integer ID.

[0025] In order to make the input node features unchanged for the position of the target agent, in this embodiment, the coordinates of the trajectory vector and the map vector are normalized so that they are centered on the position of the target agent at the last observed time step, so that the trajectory of the target agent can be predicted in parallel.

[0026] S102. The historical trajectory and nearby road segments of each traffic participant are encoded into local features through a local subgraph encoder to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant and its related road segments in the local area are constructed as an independent subgraph. The subgraphs interact with each other through a graph neural network GNN to transmit information. The subgraphs include nodes. The nodes are the features of the trajectory points and road segments of the intelligent agent. The features of the trajectory points and road segments of the intelligent agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs.

[0027] In order to exploit the spatial and semantic locality of nodes, in this embodiment, a subgraph is first constructed at the vector level, where all vector nodes belonging to the same polyline are connected to each other. v 1, v 2,…, v n}Node features p , in this application, single-layer subgraph propagation is defined as: ; In the formula, Representation Node i In the l Node characteristics of the layer; Representation Node i In the l +1 layer node features; Represents neighbor nodes j In the l Node characteristics of the layer; g enc (﹒) represents a multi-layer perceptron MLP whose weights are shared across all nodes. In the VectorNet network, specifically, the multi-layer perceptron MLP contains a single fully connected layer, followed by a normalization layer, and then a ReLU nonlinear layer; f agg (﹒) represents the maximum pooling operation to obtain the most obvious behavioral characteristics of neighboring agents; f rel (﹒) represents the relationship operator between a node and its neighbors.

[0028] Finally, in order to obtain the polyline node features, the polyline node features are calculated by the maximum pooling operation again p , where the node features of the polyline p The calculation formula is: ; In the formula, f agg (﹒) represents the maximum pooling operation; Representation Node i In the L p The feature vector of the layer.

[0029] By embedding the sorting information into the vector, constraining the connectivity of the subgraph based on polyline grouping, and encoding the attributes as node features, it is suitable for encoding structured map annotations and agent trajectories.

[0030] S103, inputting the global interaction graph into the FusionNet fusion module to fuse the dynamic object and scene information, performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features.

[0031] In this embodiment, the node feature matrix P={ p 1, p 2,…, p i} is used to model the high-order interactions on the , and the modeling equation is: ; In the formula, Represented as a node i The feature set of Representation Node i In the l +1 layer feature set; GNN(﹒) represents a single layer of graph neural network GNN; A The adjacency matrix representing the feature set of the polyline nodes.

[0032] Graph neural network GNN is implemented through self-attention mechanism: ; In the formula, P is the node feature matrix; P Q , P V is the linear projection of the node feature matrix; Represents the transpose of the linear projection of the node feature matrix. The feature map or feature vector output by the VectorNet network is used as the input feature of the FusionNet fusion module, which contains rich scene information and interaction information between dynamic objects.

[0033] Each module in the FusionNet fusion module can further process the input features of the FusionNet fusion module, fuse the relationship between dynamic objects and scene information, and develop a spatial attention layer for A2L, L2L, L2A, and A2A. The spatial attention layer is applicable to each module in the FusionNet fusion module in the same way. Taking A2L as an example, given a participant node i, the features of its surrounding lane nodes j are aggregated as follows: ; In the formula, y i For Node i Updated features;x i Indicates i Node features of nodes; w is the weight matrix; f It is a combination of layer normalization and ReLU nonlinear layer; j Indicates j nodes; Δ i is the relative position feature between the node and the lane; x j Indicates j Node features of nodes; concat (﹒) indicates that the node i Features x i and relative position feature Δ i And lane nodes j Features x j Make splicing.

[0034] S104, based on the GRU decoder, the local embedded features are fused and encoded with the global embedded features, and the features fused by the GRU decoder are decoded using a multi-layer perceptron MLP to generate a multimodal prediction result and assign a confidence score to each possible trajectory, and the final trajectory is obtained according to the confidence score to achieve trajectory prediction.

[0035] In this embodiment, a GRU-based decoder is used to decode the vector that fuses the features of each element. The FusionNet fusion module outputs a series of fused features, which include the state information of different elements in the scene, and outputs a tensor represented as: [ batch_size,num_objects,fusion_feature_dim ]; The local embedding features are passed to the GRU decoder to initialize the hidden state of the GRU decoder. The local embedding will be repeatedly expanded to multiple prediction modes to meet the requirements of multimodal prediction tasks. The global embedding features are passed to the GRU decoder to guide the decoding process. During the input process, the shape will be adjusted to adapt to the time step dimension of the GRU decoder, that is, expanded to future time steps to ensure consistent global information input at each time step.

[0036] In the linear layer, the input vector x is mapped to the output vector y , the mapping process can be expressed by the following formula: y = wx + b ,in w is the weight matrix, bis the bias vector. Each row of this weight matrix corresponds to a set of weights for an output neuron, and the value of each output neuron is the dot product of the input vector and the corresponding weight vector plus the bias term. Although linear layers are linear in nature, by stacking multiple linear layers and combining them with nonlinear activation functions, deep networks with complex decision boundaries can be constructed that can represent very complex functions.

[0037] Layer Normalization (LN) is a regularization technique used to speed up the training process of neural networks and improve the stability of the model. Its main goal is to normalize the features of each sample so that the output has the same mean and variance. For a given input X =[ x 1 ,x 2 , … ,x H ], where H represents the dimension of the feature, that is, the number of input features of the sample at this layer. The specific operation of layer normalization can be described by the following formula: Calculate the mean: ; Calculate the variance: ; Normalization: ; Zoom and Pan: ; In the formula, i Indicates i nodes; e represents a small constant used to avoid division by zero; γ represents the scaling factor, β represents the offset, both of which are learnable parameters; y i Represents the first i feature values.

[0038] Through normalization, the output of each layer of the network is made to have a mean of 0 and a variance of 1, thereby alleviating the problem of gradient vanishing and gradient exploding, especially in deep networks. Layer normalization does not depend on the batch size, which makes it effective when the batch size is small or changes, and has better flexibility. The learnable parameters allow the model to automatically adjust the normalized values ​​during the learning process to adapt to different data distributions. By standardizing the input features, layer normalization can reduce the overfitting of the model to specific training samples and improve the generalization ability of the model.

[0039] ReLU nonlinear layer is a common activation function, which mainly introduces nonlinear characteristics to enable neural networks to process complex data. Mathematical expression: f ( x )=max(0,x ). The calculation is very simple, just comparison and maximum value operation, the calculation speed is fast, suitable for the training and reasoning of large-scale deep learning models.

[0040] The local and global embedding features are integrated into the input of the GRU decoder, and the future trajectory position and possible uncertainty are predicted based on these inputs; the probability of the predicted trajectory is finally output.

[0041] In summary, the automatic driving trajectory prediction method in the above-mentioned embodiment of the present invention, by setting up a complete deep learning network model, the deep learning network model includes VectorNet network, graph neural network GNN, FusionNet fusion module and GRU decoder, aims to improve the ability to identify lane structure features in complex urban roads, enhance the interaction between data, thereby improving the accuracy of target vehicle trajectory prediction and ensuring driving safety. Specifically, by building a deep learning network model based on VectorNet network, different from the traditional road feature extraction based on rasterization and image, the graph neural network GNN is introduced to process local information and global information, and the features of all subgraphs are integrated through further message passing mechanism at the global level to capture the association relationship and context information between subgraphs. This hierarchical aggregation method enables the VectorNet network to effectively capture the subtle structure and global relationship in roads and scenes. The FusionNet fusion module performs information fusion at different levels through a multi-level feature interaction mechanism, which can fully capture the complex relationship between multimodal data and enhance the expressiveness and robustness of the deep learning network model.

[0042] Embodiment 2 Embodiment 2 of the present invention provides an autonomous driving trajectory prediction system, the system is applied to a deep learning network model, the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module and a GRU decoder, and the system includes: An acquisition module is used to acquire the trajectory of an intelligent agent and the map features of an original map based on a VectorNet network, wherein the intelligent agent includes a vehicle, and the trajectory of the intelligent agent and the map features are respectively converted into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and the plurality of trajectory vectors or map vectors are sequentially connected to generate a polyline, wherein the polyline includes a trajectory polyline and a map polyline, wherein the trajectory polyline is an approximation of the trajectory of the intelligent agent, and the map polyline is an approximation of the original map; the VectorNet network includes a local subgraph encoder and a global graph encoder; An interaction module is used to encode the historical trajectory and nearby road segments of each traffic participant into local features through a local subgraph encoder to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant and its related road segments in the local area are constructed as an independent subgraph. The subgraphs interact with each other through a graph neural network GNN to transmit information. The subgraphs include nodes, which are the features of the trajectory points and road segments of the agent. The features of the trajectory points and road segments of the agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs. A fusion module, used for inputting the global interaction graph into a FusionNet fusion module to fuse dynamic objects and scene information, and performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features; The prediction module is used to fuse and encode the local embedded features with the global embedded features based on the GRU decoder, decode the features fused by the GRU decoder using a multi-layer perceptron MLP, generate a multimodal prediction result and assign a confidence score to each possible trajectory, and obtain the final trajectory according to the confidence score to achieve trajectory prediction.

[0043] In summary, the automatic driving trajectory prediction system in the above-mentioned embodiment of the present invention, by setting up a complete deep learning network model, the deep learning network model includes VectorNet network, graph neural network GNN, FusionNet fusion module and GRU decoder, aims to improve the ability to identify lane structure features in complex urban roads, enhance the interaction between data, thereby improving the accuracy of target vehicle trajectory prediction and ensuring driving safety. Specifically, by building a deep learning network model based on VectorNet network, different from the traditional road feature extraction based on rasterization and image, the graph neural network GNN is introduced to process local information and global information, and the features of all subgraphs are integrated through further message passing mechanism at the global level to capture the association relationship and context information between subgraphs. This hierarchical aggregation method enables the VectorNet network to effectively capture the subtle structure and global relationship in roads and scenes. The FusionNet fusion module performs information fusion at different levels through a multi-level feature interaction mechanism, which can fully capture the complex relationship between multimodal data and enhance the expressiveness and robustness of the deep learning network model.

[0044] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0045] In addition, an embodiment of the present invention further provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method in the above embodiment when executing the program.

[0046] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0047] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0048] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0050] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An automatic driving trajectory prediction method, characterized in that: The method is applied to a deep learning network model, wherein the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module, and a GRU decoder, and the method includes: Based on the VectorNet network, the intelligent agent trajectory and the map features of the original map are obtained, the intelligent agent includes a vehicle, the intelligent agent trajectory and the map features are respectively converted into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and the plurality of trajectory vectors or map vectors are sequentially connected to generate a polyline, the polyline includes a trajectory polyline and a map polyline, the trajectory polyline is an approximation of the intelligent agent trajectory, and the map polyline is an approximation of the original map; the VectorNet network includes a local subgraph encoder and a global graph encoder; The local subgraph encoder encodes the historical trajectory and nearby road segments of each traffic participant into local features to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant in the local area and its related road segments are constructed as an independent subgraph. The subgraphs interact with each other through the graph neural network GNN to transmit information. The subgraphs include nodes, which are the features of the trajectory points and road segments of the agent. The features of the trajectory points and road segments of the agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs. Inputting the global interaction graph into a FusionNet fusion module to fuse dynamic objects and scene information, performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features; The local embedded features are fused and encoded with the global embedded features based on the GRU decoder, and the features fused by the GRU decoder are decoded using a multi-layer perceptron MLP to generate a multimodal prediction result and assign a confidence score to each possible trajectory. The final trajectory is obtained according to the confidence score to achieve trajectory prediction.

2. The automatic driving trajectory prediction method according to claim 1, characterized in that: The steps of converting the agent trajectory and the map feature into a plurality of trajectory vectors and map vectors respectively according to a preset time interval and a preset spatial interval include: An original map is obtained, a starting point and a direction are selected according to the original map, map key points are uniformly sampled from a spline with a preset spatial interval according to the starting point and the direction, and adjacent map key points are sequentially connected to obtain a map vector to obtain a map feature; trajectory key points are sampled according to a preset time interval, and the sampled trajectory key points are sequentially connected to form a trajectory vector, wherein the map vector and the trajectory vector both include a plurality of nodes, each node is an element in a subgraph, and a node represents a historical trajectory point of a traffic participant or a road segment.

3. The automatic driving trajectory prediction method according to claim 1, characterized in that: Each vector belonging to a polyline is represented as a node, and the feature of the node is defined as: ； v i represents a vector; Indicates the starting point; Indicates the end point; d Represents 2D coordinates (x, y) or 3D coordinates (x, y, z); a i Represents attribute features, including object type, trajectory timestamp, road feature type, or lane speed limit; j Indicates p j The integer ID of v i ∈p j , p j A node feature representing an integer ID.

4. The automatic driving trajectory prediction method according to claim 1, characterized in that: Single-layer subgraph propagation is defined as: ； In the formula, Representation Node i In the l Node characteristics of the layer; Representation Node i In the l +1 layer node features; Represents neighbor nodes j In the l Node characteristics of the layer; g enc (﹒) represents a multi-layer perceptron MLP, whose weights are shared among all nodes; φ agg (﹒) represents the maximum pooling operation to obtain the most obvious behavioral characteristics of neighboring agents; φ rel (﹒) represents the relationship operator between a node and its neighbors.

5. The automatic driving trajectory prediction method according to claim 1, characterized in that: The method further comprises: The coordinates of the trajectory vector and the map vector are normalized to make the input node features remain unchanged for the position of the target agent.

6. The automatic driving trajectory prediction method according to claim 1, characterized in that: The step of inputting the global interaction graph into the FusionNet fusion module to fuse dynamic objects and scene information includes: The high-order interactions on node features are modeled through the global interaction graph, and the modeling equation is: ； In the formula, Represented as a node i In the l The feature set of the layer; Representation Node i In the l +1 layer feature set; GNN(﹒) represents a single layer of graph neural network GNN; A The adjacency matrix representing the feature set of the polyline nodes.

7. An automatic driving trajectory prediction system, characterized in that: The system is applied to a deep learning network model, wherein the deep learning network model includes a VectorNet network, a graph neural network GNN, a FusionNet fusion module and a GRU decoder, and the system includes: An acquisition module is used to acquire the trajectory of an intelligent agent and the map features of an original map based on a VectorNet network, wherein the intelligent agent includes a vehicle, and the trajectory of the intelligent agent and the map features are respectively converted into a plurality of trajectory vectors and map vectors according to a preset time interval and a preset spatial interval, and the plurality of trajectory vectors or map vectors are sequentially connected to generate a polyline, wherein the polyline includes a trajectory polyline and a map polyline, wherein the trajectory polyline is an approximation of the trajectory of the intelligent agent, and the map polyline is an approximation of the original map; the VectorNet network includes a local subgraph encoder and a global graph encoder; An interaction module is used to encode the historical trajectory and nearby road segments of each traffic participant into local features through a local subgraph encoder to construct a local area. Each local area includes a trajectory polyline and a map polyline. Each traffic participant and its related road segments in the local area are constructed as an independent subgraph. The subgraphs interact with each other through a graph neural network GNN to transmit information. The subgraphs include nodes, which are the features of the trajectory points and road segments of the agent. The features of the trajectory points and road segments of the agent include position, speed and acceleration. The global graph encoder fuses the subgraphs for global interaction to generate the global features of the scene and construct a global interaction graph. The global interaction graph includes subgraphs and edges. The edges are used to connect two subgraphs to represent the interaction relationship between different subgraphs. A fusion module, used for inputting the global interaction graph into a FusionNet fusion module to fuse dynamic objects and scene information, and performing model training on the FusionNet fusion module to obtain a trained FusionNet fusion module, and outputting fusion features according to the trained FusionNet fusion module, wherein the fusion features include local embedding features and global embedding features; The prediction module is used to fuse and encode the local embedded features with the global embedded features based on the GRU decoder, decode the features fused by the GRU decoder using a multi-layer perceptron MLP, generate a multimodal prediction result and assign a confidence score to each possible trajectory, and obtain the final trajectory according to the confidence score to achieve trajectory prediction.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the automatic driving trajectory prediction method as described in any one of claims 1 to 6 is implemented.

9. A data processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the automatic driving trajectory prediction method as described in any one of claims 1 to 6 is implemented.

Images