An agent action prediction method based on multi-scale space perception
By using multi-scale graph convolutional neural networks and feature fusion technology, the problem of integrating all-element interaction relationships in agent action prediction was solved, achieving accurate multimodal trajectory prediction in complex map environments and improving the reliability and multi-task applicability of agent actions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2024-09-30
- Publication Date
- 2026-06-09
AI Technical Summary
Existing agent action prediction methods cannot effectively integrate the interaction relationships of all elements in complex map environments, resulting in unreliable predictions and a lack of multi-task versatility. Furthermore, existing neural network methods cannot meet the requirements for complete extraction of high-definition map information and trajectory prediction.
Multi-scale graph convolutional neural networks are used to extract map features, and convolutional neural networks and feature pyramid networks are combined to extract trajectory features. Features are fused through self-attention and cross-attention mechanisms, and action prediction is performed using multimodal trajectory prediction. The results output multimodal complete trajectory coordinates and confidence scores.
It achieves accurate prediction of agent actions in complex map environments, provides multimodal action prediction with full-element information fusion, improves the interpretability and reliability of prediction, and is suitable for multi-task scenarios.
Smart Images
Figure CN119399570B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of autonomous action prediction and intelligent behavior decision-making, and specifically to a method for predicting the actions of intelligent agents based on multi-scale spatial perception. Background Technology
[0002] The construction of intelligent agent models mainly covers entity models, perception models, behavior models, decision-making models, and control models. Among them, the behavior model is the driving force that enables the entity model to effectively achieve specific goals. Based on the extraction of perception features, it forms the data source and basis for decision-making and control. At present, the design of behavior models still focuses on prediction based on dynamics, rules, and expert experience data. Rule-based methods for specific tasks require huge human and resource consumption for design, and cannot effectively resist knowledge blind spots and achieve innovative exploration, easily falling into the dilemma of local suboptimal results.
[0003] Current mainstream action prediction methods leverage the features of rasterized map data to provide crucial geometric information. Convolutional Neural Networks (CNNs) extract spatial features based on the relative positions of nodes within the map scene, providing useful background information for motion prediction. Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), extract temporal features from historical trajectory data of agents and use linear or nonlinear methods to fit the temporal relationships of autonomous actions, generating future trajectory coordinates. These methods inevitably lose some key information from high-resolution maps and ignore the interaction relationships between historical trajectories with similar temporal features within the same scene. Graph Neural Network-based trajectory prediction methods currently only consider the node interactions between action entities, without integrating the complex and numerous map element node information.
[0004] Therefore, existing rule-based action prediction methods have technical problems such as complex design and lack of versatility for multiple tasks. Furthermore, the existing neural network-related prediction methods cannot meet the requirements for full-element interaction relationship coverage in complex map environments. Summary of the Invention
[0005] The purpose of this invention is to provide a method for predicting the actions of intelligent agents based on multi-scale spatial perception. By fully combining the powerful semantic representation capabilities of graph neural networks, it enables accurate and reasonable autonomous actions, providing reliable and trustworthy multimodal action predictions for intelligent agent simulation based on the fusion of all elements of information.
[0006] To achieve the above objectives, the present invention employs the following technical solution:
[0007] A method for predicting agent actions based on multi-scale spatial perception includes: map spatial structure modeling, historical trajectory feature extraction, feature fusion, and multimodal action prediction, wherein:
[0008] The map spatial structure construction utilizes a multi-scale graph convolutional neural network to extract map features from the two-dimensional vector map data in the application scenario map, thereby obtaining high-dimensional map feature information.
[0009] The historical trajectory feature extraction employs convolutional neural networks and feature pyramid networks to extract high-dimensional trajectory data features of all agents within the application scenario map.
[0010] The feature fusion uses a self-attention mechanism to perform correlation modeling and fusion of high-dimensional map feature information and high-dimensional trajectory data feature information obtained from map spatial structure modeling and historical trajectory feature extraction, thereby obtaining intelligent agent trajectory fusion features with directional information.
[0011] The multimodal trajectory prediction utilizes the trajectory fusion features of the agent to perform prediction regression and confidence scoring, outputting the multimodal complete trajectory coordinates and corresponding confidence scores of the action prediction, providing auxiliary decision-making for the agent's actions.
[0012] Furthermore, the application scenario map should be able to provide map data in a two-dimensional bird's-eye view vector coordinate format. The map data includes the historical behavioral trajectory data of all intelligent agents within the application scenario map, and the lane centerline has lane coordinate information, including the initial and end position coordinates of the lane.
[0013] Furthermore, the map spatial structure construction utilizes a multi-scale graph convolutional neural network to extract map features from the two-dimensional vector map data in the application scenario, obtaining high-dimensional map feature information, including:
[0014] 91-1) Using an MLP network to map lane coordinate information, represented as... Among them l i This represents the i-th lane node in the model. and v represents the initial and ending coordinates of the center line of the i-th lane in the map data, respectively. i Indicates the position of the center point of the i-th lane; MLP shape This indicates that the length of lane nodes is extracted using an MLP network; MLP loc This indicates the use of an MPL network to extract the center location features of lane nodes; the application scenario involves all lane nodes within a map. i The high-dimensional map data L is composed of l i Let L be the i-th row;
[0015] (1-2) Based on the high-dimensional map data L, the GCN network is used to learn the association information between the nodes of the high-dimensional map data; among them, the GCN network processes the forward, backward, left neighbor and right neighbor according to the four connection types of lane nodes, which is expressed as: F=LW0+∑ j∈{q，h，z，y} A j LW j Where W0 is the weight of the original information of the lane node, set as an identity matrix; j represents the i-th connection type, q, h, z, y represent the physical connection relationships of the lane node's forward, backward, left, and right neighbors in the map, respectively, and A j and W j Let represent the adjacency matrix and weight matrix for the j-th connection type, respectively;
[0016] (1-3) Selecting whether lane nodes have ground markings or signs as an additional weight for A j The restriction is implemented as follows: We get F = LW0 + ∑ j∈{q，h，z，y} A′ j LW j Where I is the identification matrix related to ground markings, that is, the connection relationship matrix of lane nodes that do not have ground markings;
[0017] (1-4) Based on step (1-3), obtain the selected A′. j Finally, the high-dimensional map feature information is represented as follows: Where C is the number of expansion scale factors, and k c It is the c-th scaling factor.
[0018] Furthermore, the historical trajectory feature extraction employs convolutional neural networks and feature pyramid networks to extract high-dimensional trajectory data features of all agents within the application scenario map, including:
[0019] A CNN network is used to extract features from the historical behavior trajectory data of all agents in the scene to obtain historical trajectory features T. At the same time, a feature pyramid network (FPN) is used to fuse the feature values of different scales in the historical trajectory features T to output high-dimensional trajectory data features T′ with time series characteristics.
[0020] Furthermore, T′ is obtained by concatenating information from different nodes in the historical trajectory feature T, and is represented as T′ = concat(t mi Δ mi,mj , t mj ), where t mi It is the feature information of the mi-th historical trajectory node in the feature variables T, Δ mi,mj =MLP(t) mj -tmi ) indicates that MLP encoding is used for different historical trajectory nodes t mj t mi The process of extracting position difference feature information is then performed, followed by direct concatenation to obtain high-dimensional trajectory data features T′.
[0021] Furthermore, the feature fusion uses a self-attention mechanism to perform correlation modeling and fusion of high-dimensional map feature information and high-dimensional trajectory data feature information obtained from map spatial structure modeling and historical trajectory feature extraction, to obtain agent trajectory fusion features with directional information, including:
[0022] (3-1) The historical trajectory fusion information T′ obtained by extracting historical trajectory features is fused using a self-attention mechanism. The calculation process is as follows: in d is the dimension of the key vector K. The layer normalization and ReLU operation are represented by `softmax`, which represents the normalization exponential function. The query, key, and value vectors of the self-attention mechanism are represented by Q, K, and V, respectively, and are obtained by passing the high-dimensional trajectory data feature T′ through three different linear layers, expressed as: Q = T′W Q K = T′W K V = T′W V Among them, W Q W K W V Let Q, K, and V represent the weight matrices, respectively.
[0023] (3-2) Historical trajectory feature information The high-dimensional map feature information F is further fused through a cross-attention mechanism, and is represented as: Where <-, ·> denote the inner product operation, that is, the weight matrix of the cross-attention operator is obtained by transforming the query vector of the attention mechanism. and bond vector K F The result is obtained through multiplication, and then processed through the value vector V. F The weighted values are then used to output the agent trajectory fusion feature Z, which contains directional information.
[0024] Furthermore, the multimodal trajectory prediction utilizes agent trajectory fusion features for prediction regression and confidence scoring, outputting the multimodal complete trajectory coordinates and corresponding confidence scores for action prediction, providing reasonable auxiliary decision-making for agent actions, including:
[0025] (4-1) The extracted agent trajectory fusion feature Z is subjected to regression operation through the first MLP branch network to form the multimodal complete trajectory BEV coordinates, denoted as: This represents the BEV coordinates of the T-th step of the k-th action of the m-th agent, where K is a settable parameter representing the predicted trajectory of K actions;
[0026] (4-2) Using the multimodal complete trajectory BEV coordinates P obtained in (4-1) m，reg The fusion feature Z with the agent's trajectory is used to form confidence scores for K action predictions through a second MLP branch network, denoted as O. m，cls =(c m，0 C m，1 c m，K-1 ), where C m，K-1 This represents the confidence score of the prediction of the Kth behavioral trajectory of the m-th agent.
[0027] Furthermore, during the MLP training process for multimodal trajectory prediction, the BEV coordinates P of the complete multimodal trajectory are used. m，reg and confidence score O m，cls Based on the ground truth trajectory values of each agent in the existing dataset, the classification loss and regression loss in the loss function adopt the maximum marginal loss and the smoothed L1 loss respectively, i.e., L = L cls +L reg ,in The loss function is used to train the model; specifically, M is the number of agents, and ∈ represents the boundary value of the loss function, which takes the value of 0.2. It is the true value of the BEV coordinates of agent m at step t, reg(x) = ∑ i d(x i ), x i It is the i-th element of x, d(x) i ) is a smoothed L1 loss.
[0028] An action prediction device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, it implements the intelligent agent action prediction method based on multi-scale spatial perception.
[0029] A computer-readable storage medium storing a computer program; when executed by a processor, the computer program implements the intelligent agent action prediction method based on multi-scale spatial perception.
[0030] Compared with the prior art, the present invention has the following technical features:
[0031] 1. This invention constructs a multi-scale graph neural network to extract features from map element information with long-distance dependencies and directional information, overcoming the problem of unacceptable and uninterpretable predictions caused by the lack of utilization of highly correlated semantic information of map nodes in existing action prediction methods.
[0032] 2. This invention constructs a data fusion network to capture complete information interaction between map features and intelligent agents. Specifically, it includes the fusion of all element node data such as target intelligent agent information, dynamic historical information, and static environmental information, to achieve effective modeling of spatial topology and to be used for multimodal action prediction. Based on complete spatial information, it can obtain K (K is a settable parameter) action trajectory predictions with interpretable and acceptable data that are closer to real experience data. Attached Figure Description
[0033] Figure 1 This is a diagram illustrating the action prediction framework of the present invention;
[0034] Figure 2 This is a network architecture diagram of the map spatial structure modeling module of the present invention;
[0035] Figure 3 This is a network architecture diagram of the historical trajectory feature extraction module of the present invention;
[0036] Figure 4 This is a network architecture diagram for feature fusion in this invention;
[0037] Figure 5 This is a network architecture diagram for multimodal action prediction in this invention;
[0038] Figure 6 This is a schematic diagram of the application process of an embodiment of the present invention;
[0039] Figure 7 This is a visualization of the simulation results of the present invention. Detailed Implementation
[0040] This invention provides a method for predicting agent actions based on multi-scale spatial perception. This method can be directly applied to application scenarios with two-dimensional vector map data. For greater generality, this invention uses the open-source Argoverse dataset for specific implementation. This dataset provides high-precision and diverse autonomous driving scenarios and visual data. The Argoverse dataset provides lane-level precision map information, specifically including center lines, intersections, and pedestrian crossings. Its motion prediction dataset contains approximately 320,000 scene sequences. These scenes all feature a primary agent, and are captured from the perspective of the primary agent's trajectory. The total duration is 5 seconds, with a sampling rate of 10Hz (10 frames per second). The motion prediction task is to predict the trajectory of the primary agent in the last 3 seconds, given the first two seconds of its motion information and high-precision map information. Trajectory prediction involves complex interactions between different agents, and is generally captured in areas with complex traffic conditions. The dataset size is sufficient for training complex models.
[0041] like Figure 1 As shown, this invention provides an agent action prediction method based on multi-scale spatial perception, including map spatial structure modeling, historical trajectory feature extraction, feature fusion, and multimodal action prediction, wherein:
[0042] The map spatial structure construction utilizes a multi-scale graph convolutional neural network (GCN) to extract map features from two-dimensional vector map data in the application scenario, thereby obtaining high-dimensional map feature information;
[0043] The historical trajectory feature extraction employs a highly parallel computing-efficient convolutional neural network (CNN) and feature pyramid (FPN) network to extract high-dimensional trajectory data features of all agents within the scene;
[0044] The feature fusion method uses a self-attention mechanism to perform correlation modeling and fusion of high-dimensional map feature information and high-dimensional trajectory data feature information obtained from map spatial structure modeling and historical trajectory feature extraction, thereby obtaining intelligent agent trajectory fusion features with directional information.
[0045] The multimodal trajectory prediction utilizes the trajectory fusion features of the agent to perform prediction regression and confidence scoring, outputting the BEV coordinates of the multimodal complete trajectory of the action prediction and the corresponding confidence score, providing reasonable auxiliary decision-making for the agent's actions.
[0046] 1. Map spatial structure modeling
[0047] The map spatial structure construction process mainly includes: modeling lane nodes and ground marking information using multilayer perceptron (MLP) network layers, multi-scale graph convolutional neural networks (GCN), and linear layers to obtain high-dimensional map feature information. The network structure is as follows: Figure 2 As shown.
[0048] (1-1) First, an MLP network layer is used to preprocess the attributes such as lane centerlines, intersections, and traffic signs provided by the application scenario map; the application scenario map should be able to provide map data in a two-dimensional bird's-eye view (BEV) vector coordinate format, and the map data includes the historical behavioral trajectory data of all agents within the application scenario map. The lane centerline has lane coordinate information, including the initial and ending position coordinates of the lane, represented as v. start (x, y) and v end (x, y); Lane coordinate information is mapped using an MLP network, represented as...
[0049] Among them, l i This represents the i-th lane node in the model. and v represents the initial and ending coordinates of the center line of the i-th lane in the map data, respectively.i This represents the position of the i-th lane center point, calculated using the average of the initial and final coordinates of the lane centerline. The application scenario involves all lanes within the map. i The high-dimensional map data L is composed of l i This is the i-th row of L. MLP shape This indicates that the length of lane nodes is extracted using an MLP network; MLP loc This indicates that the center position features of lane nodes are extracted using an MPL network.
[0050] (1-2) Based on the high-dimensional map data L obtained in step (1-1), the GCN network is used to learn the association information between the nodes of the high-dimensional map data; wherein, the GCN network processes the forward, backward, left neighbor and right neighbor according to the four connection types of lane nodes, which is expressed as: F=LW0+∑ j∈{q，h，z，y} A j LW j Where W0 is the weight of the original information of the lane node, which is generally set as an identity matrix; j represents the j-th connection type, q, h, z, and y represent the physical connection relationships of the lane node's forward, backward, left, and right neighbors in the map, respectively, and A j and W j Let A and B represent the adjacency matrix and weight matrix for the j-th connection type, respectively; where A q and A h This is obtained by moving the identity matrix one step to the upper right and lower left. Left and right neighbors refer to the node connections that allow lane changes without violating traffic rules, denoted as A. z and A y The connection type and direction of lane nodes are encoded by setting the crossable lane node positions to 1 in the GCN network structure model described above.
[0051] (1-3) Based on the lane node connection type and direction obtained in step (1-2), this scheme selects the ground marking and sign types coupled with F as information expansion, and applies this to the lane node adjacency matrix A. i Perform constraint design.
[0052] In fact, the selection of beneficial coupling information is relevant to the dataset characteristics and the target task. Without loss of generality, this scheme selects whether lane nodes have ground markings or signs as an additional weight for A. j By applying restrictions, more geometric information can be encoded, specifically as follows: The connection relationships are initially filtered, resulting in F = LW0 + ∑ j∈{q，h，z，y} A′ j LW jWhere I is the identification matrix related to ground markings, that is, the connection relationship matrix of lane nodes that do not have ground markings.
[0053] (1-4) Based on step (1-3), obtain the selected A′. j Subsequently, to further capture the long-distance forward dependencies between lane nodes and meet the needs of action prediction with a time span, this scheme extends the lane information of the forward relationships; the final high-dimensional map feature information is represented as follows:
[0054] Where C is the number of scaling factors, which can be flexibly set according to the task; in this scheme, it is set to 5; k c This is the c-th scaling factor. The scaling factor used in this invention is k. c =2 c The selected adjacency matrix A′ is then scaled using a scaling factor. j According to k c Perform matrix exponentiation; for example, when c = 5, k c =2 5 =32, W j＝q，c That is, W q or W c .
[0055] This invention uses multi-scale operation technology to aggregate forward lane node information over a longer distance into the current node. The larger the scale factor (such as the lane node with a scale factor of 5 mentioned above), the smaller the aggregation effect. Then, high-dimensional map feature information F is output through a linear layer, with a dimension of N×128, where N is the number of lane nodes in the scene.
[0056] 2. Historical Trajectory Feature Extraction
[0057] Historical trajectory feature extraction primarily utilizes Convolutional Neural Networks (CNNs) and Feature Pyramid Networks (FPNs) to extract high-dimensional trajectory data features from all agents within the scene. The network structure is as follows: Figure 3 As shown.
[0058] First, a CNN network with high parallel computing efficiency is used to extract features from the historical behavioral trajectory data of all agents in the scene. The CNN network uses three sets of one-dimensional convolutional networks with a kernel size of 3. Each convolutional network consists of two residual networks, with the first residual block having a stride of 2 and the second residual block having a stride of 1. After extraction by the CNN network, the historical trajectory features T are obtained. At the same time, a Feature Pyramid Network (FPN) is used to fuse the feature values of different scales in the historical trajectory features T, outputting a high-dimensional trajectory data feature T′ with time series characteristics and a dimension of 128. Specifically:
[0059] T′ is obtained by concatenating information from different nodes in the historical trajectory feature T, and is represented as T′=concat(t mi Δ mi,mj , t mj ), where t mi It is the feature information of the mi-th historical trajectory node in the feature variables T, Δ mi,mj =MLP(t) mj -t mi ) indicates that MLP encoding is used for different historical trajectory nodes t mj t mi The process of extracting position difference feature information is then performed, followed by direct concatenation to obtain high-dimensional trajectory data features T′.
[0060] During feature extraction, the historical action trajectory data of different agents in the scene are padded with zeros to obtain point displacement coordinate data with the same duration (taking 2 seconds as an example). Then, the same number of 0 / 1 masks are connected to the displacement coordinate data to indicate whether the coordinate point is padded data. Thus, a historical behavior trajectory tensor T with dimensions of M×20×3 is formed for the next step of feature extraction, where M is the number of all agents in the scene, 20 represents the number of trajectory collection points within the 2-second duration, and 3 represents the (x, y) coordinates and mask data of each point displacement coordinate.
[0061] By extracting historical trajectory features, a high-dimensional trajectory data feature T with time characteristics is output, with a dimension of 128, that is, an output dimension of M×20×128.
[0062] 3. Feature fusion
[0063] Feature fusion encodes trajectory position difference information through an MLP network layer, then fuses historical trajectories using a self-attention network, and finally fuses the extracted high-dimensional vector map features F with the high-dimensional trajectory data features T′ using a cross-attention module to obtain the agent's trajectory fusion feature Z; the network structure is as follows. Figure 4 As shown.
[0064] (3-1) The historical trajectory fusion information T′ obtained by extracting historical trajectory features is fused using a self-attention mechanism. The calculation process is as follows: in d is the dimension of the key vector K. Dividing by the square root of d prevents the inner product from becoming too large. The representation layer uses normalization and ReLU operations, with softmax representing the normalization exponential function; the self-attention module outputs the fused historical trajectory feature information.
[0065] The query, key, and value vectors of the self-attention mechanism are represented as Q, K, and V, respectively. These are obtained by passing the high-dimensional trajectory data feature T′ through three different linear layers, and are expressed as: Q = T′W Q K = T′W K V = T′W V Among them, W Q W K W V These represent the weight matrices for Q, K, and V, respectively, and are typically initialized randomly.
[0066] (3-2) The fused historical trajectory feature information obtained in step (3-1) The high-dimensional map feature information F obtained in step 1 is further fused with the information through a cross-attention mechanism. This process encodes the relative position information of different historical trajectory nodes and lane nodes. The key is to uniformly encode both trajectory and map node information using a weight matrix, and to mask unrelated node information using a masking attention mechanism. The specific calculation process is as follows:
[0067] Where <·, ·> represent inner product operations, that is, the weight matrix of the cross-attention operator is obtained by transforming the query vector of the attention mechanism. and bond vector K F The result is obtained through multiplication, and then processed through the value vector V. F Weighted summaries are applied to output a directional agent trajectory fusion feature Z, with dimensions M×128. The query, key, and value vectors for the cross-attention mechanism are respectively derived from... K F =FW K V F =FW V get.
[0068] 4. Multimodal trajectory prediction
[0069] The multimodal trajectory prediction module uses the agent trajectory fusion feature Z obtained in step 3 to perform trajectory prediction regression and confidence score, and outputs the predicted multimodal complete trajectory BEV coordinate information and score information to provide reasonable auxiliary decision-making for agent actions; that is, the network finally outputs the predicted agent coordinates and the corresponding score. In practical applications, the prediction result with a higher confidence score can be selected as a decision reference.
[0070] The multimodal trajectory prediction consists of two pre-trained MLP branch architectures. Each MLP architecture comprises two sets of linear layers, group normalization, and ReLU modules. The first MLP branch network outputs the BEV coordinates of the complete multimodal trajectory of the action prediction, with input and output dimensions of M×128 and M×K×60, respectively. The other MLP branch network performs confidence scoring on the action prediction results, with input and output dimensions of M×128 and M×K×1, respectively. It scores the confidence of the action prediction results for each modality, where K is a configurable parameter representing K action trajectory predictions. The network structure is as follows: Figure 5 As shown.
[0071] (4-1) The extracted agent trajectory fusion feature Z is subjected to regression operation through the first MLP branch network to form the multimodal complete trajectory BEV coordinates, denoted as: Let P represent the BEV coordinates of the m-th agent in the k-th action at step T. T is typically taken as 3 seconds. Since the sampling frequency is 10Hz, this represents a predicted coordinate value of 30 for each trajectory. The coordinate values are represented using two-dimensional coordinates (x, y). Therefore, the output is the 60-dimensional multimodal complete trajectory BEV coordinates P. m，reg ;
[0072] (4-2) Using the multimodal complete trajectory BEV coordinates P obtained in (4-1) m，reg The fusion feature Z with the agent's trajectory is used to form confidence scores for K action predictions through a second MLP branch network, denoted as O. m，cls =(c m，0 C m，1 c m，K-1 ), where C m，K-1 This represents the confidence score of the prediction of the Kth behavioral trajectory of the m-th agent.
[0073] In the MLP training process for multimodal trajectory prediction, the BEV coordinates P of the complete multimodal trajectory obtained in steps (4-1) and (4-2) are used. m，reg and confidence score O m，cls Based on the ground truth trajectory values of each agent in the existing dataset, the classification loss and regression loss in the loss function adopt the maximum marginal loss and the smoothed L1 loss respectively, i.e., L = L cls +L reg ,in The loss function is used to train the model; specifically, M is the number of agents, ∈ represents the boundary value of the loss function, which is a non-negative real number, and in this embodiment, it is taken as 0.2; It is the true value of the BEV coordinates of agent m at step t, reg(x) = ∑ i d(xi ), x i It is the f-th element of x, d(x) i ) is the smoothed L1 loss, defined as:
[0074] This invention utilizes map and scene data to perform end-to-end model training on multi-scale GCN networks, CNN networks, feature fusion networks, and multimodal trajectory prediction networks. Therefore, the trained model framework facilitates algorithm transfer and application. For greater generality, the method requires a simple data source format and does not require specific preprocessing; the deployment and application of the model can be achieved primarily by combining two-dimensional vector map data and trajectory data within the scene.
[0075] Figure 6 A flowchart of the practical application of this invention is provided. After loading the dataset or map data in the actual scene, the method generally selects the main intelligent agent as the research object, collects historical trajectory data with time series characteristics of all intelligent agents in the scene and map vector data in the current scene; uses a CNN network to extract the features of historical behavior trajectory, uses a multi-scale GCN to learn features of vector map data, and fuses the learned historical behavior trajectory features and map element features through a fusion network with attention mechanism as the core. The fused feature information is input into the action prediction network to perform multimodal prediction and evaluation of the future action trajectory of the intelligent agent. The number of predicted trajectories can be flexibly set according to the application scenario.
[0076] This invention provides an agent action prediction method based on multi-scale spatial perception. It extracts features from the agent's historical trajectory and scene map in the target simulation data, corresponding to different network structures. The map data feature extraction directly uses vector map data, eliminating the need for rasterization and avoiding the loss of map element information in the scene. It can establish a structured map element map based on complete vector map data. When using a multi-scale GCN network to aggregate information from multiple forward lane nodes, it can accurately model the characteristics of fast-moving agents with large displacements, achieving feature extraction from lane nodes with long-distance dependencies. The agent's trajectory prediction output is further updated, and a confidence score is calculated based on the updated trajectory data, enabling the agent to explore multi-modal behavioral strategies and improving its exploration efficiency.
[0077] Example:
[0078] See appendix Figure 6This embodiment uses the publicly available Argoverse trajectory prediction benchmark to verify and visualize the technology. The Argoverse dataset contains 324,557 scenes, each lasting 5 seconds, with a sampling rate of 10Hz. The trajectory of each tracked object is represented by its centroid trajectory. The first 2 seconds of data from the training and validation sets in the dataset are used as input data, and the last 3 seconds are used as the model's true prediction values. The map file provides relevant BEV vector coordinate map data within the 5-second scene, which can meet the map data acquisition application requirements of this invention.
[0079] First, using the original BEV coordinates of the two-dimensional initial and final positions of lane centerlines and ground marking elements provided in the Argoverse dataset map file, lane node and ground marking information are modeled through a multilayer perceptron (MLP) network layer, a multi-scale graph convolutional neural network (GCN), and a linear layer to obtain high-dimensional map feature information with a dimension of N×128, where N is the number of lane nodes in the scene.
[0080] This invention uses the action data of all agents within the scene in the Argoverse dataset for the first 2 seconds as the input data for historical trajectories. Data that does not meet the 2-second time requirement is padded using a 0-padding method, resulting in 20 displacement coordinate data points with the same duration and a sampling frequency of 10Hz. The displacement coordinate data is then concatenated with the same number of 0 / 1 masks to indicate whether the coordinate point is padded data, thus forming a historical behavior trajectory tensor of dimension M×20×3 for the next step of feature extraction. Here, dimension M represents the number of all agents in the scene, 20 represents the number of trajectory sampling points for each agent within 2 seconds, and 3 represents the x, y coordinates and mask data of each displacement point. Through the historical trajectory feature extraction network designed in this invention, historical trajectory data with time-series characteristics is output, with a dimension of 128, i.e., the output historical trajectory data is M×20×128.
[0081] Based on the aforementioned high-dimensional map and historical motion trajectory feature information, the relevant information is fused and modeled through an attention mechanism to obtain the position coordinates and driving direction information related to the agent's trajectory. Finally, the fused high-dimensional features are used for prediction regression and confidence scoring to output the multimodal complete trajectory coordinates and scoring information of the agent, providing reasonable auxiliary decision-making for the agent's actions.
[0082] from Figure 7The simulation results visualization of this scheme show that the dataset scenarios selected in this scheme are mostly at intersections, where vehicles turn left or right and change lanes—areas with relatively complex traffic. This provides complex training conditions for the agent's action prediction, enabling the model to reliably predict trajectories in complex road conditions and real-time obstacles by combining map element feature information. This invention adopts an end-to-end model training mechanism, facilitating deployment on various simulation platforms. The map data format corresponding to the algorithm model is simple, requiring no specific preprocessing. Trajectory prediction modeling can be achieved with only basic data such as historical behavior trajectory coordinates with time-series characteristics, vector lane node centerlines, and ground markings. The output is a flexibly configurable multimodal behavior trajectory and evaluation results. The evaluation results of the corresponding trajectories can provide a good basis for evaluating the autonomous behavior capabilities of intelligent models in industrial applications, demonstrating significant application advantages.
[0083] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for predicting agent actions based on multi-scale spatial perception, characterized in that, This includes: map spatial structure modeling, historical trajectory feature extraction, feature fusion, and multimodal action prediction, among which: The map spatial structure modeling utilizes a multi-scale graph convolutional neural network to extract map features from two-dimensional vector map data in the application scenario map, thereby obtaining high-dimensional map feature information. The historical trajectory feature extraction employs convolutional neural networks and feature pyramid networks to extract high-dimensional trajectory data features of all agents within the application scenario map. The feature fusion method uses a self-attention mechanism to perform correlation modeling and fusion of high-dimensional map feature information and high-dimensional trajectory data feature information obtained from map spatial structure modeling and historical trajectory feature extraction, to obtain agent trajectory fusion features with directional information, including: (3-1) High-dimensional trajectory data features obtained by extracting historical trajectory features Feature fusion is performed using a self-attention mechanism, and the computation process is as follows: ,in , Key vector Dimensions This indicates layer normalization and ReLU operation. The normalized exponential function is represented as follows; the query, key, and value vectors of the self-attention mechanism are represented as follows: Utilizing the characteristics of high-dimensional trajectory data After passing through three different linear layers, the results are expressed as follows: ;in, They represent The weight matrix; (3-2) Historical trajectory feature information and high-dimensional map feature information Further fusion is achieved through a cross-attention mechanism, represented as follows: ;in The weight matrix of the inner product operation, i.e., the cross-attention operator, is obtained by applying the query vector of the attention mechanism. and key vector The result is obtained through multiplication, and then processed through the value vector. Weighted summaries are then applied to output the agent trajectory fusion features containing directional information. ; The multimodal action prediction utilizes the trajectory fusion features of the agent to perform prediction regression and confidence scoring, outputting the multimodal complete trajectory coordinates and corresponding confidence scores of the action prediction, providing auxiliary decision-making for the agent's actions.
2. The agent action prediction method based on multi-scale spatial perception according to claim 1, characterized in that, The application scenario map provides map data in a two-dimensional bird's-eye view vector coordinate format. The map data includes the historical behavior trajectory data of all intelligent agents within the application scenario map. The lane centerline has lane coordinate information, including the initial and end position coordinates of the lane.
3. The agent action prediction method based on multi-scale spatial perception according to claim 1, characterized in that, The map spatial structure modeling utilizes a multi-scale graph convolutional neural network to extract map features from two-dimensional vector map data in the application scenario, obtaining high-dimensional map feature information, including: (1-1) Using an MLP network to map lane coordinate information, represented as follows: ,in The first step in modeling Each lane node and These represent the first and second parts of the map data, respectively. Initial and ending coordinates of the center lines of each lane Indicates the first The center point location of each lane; This indicates that the length of lane nodes is extracted using an MLP network. This indicates the use of an MPL network to extract the center location features of lane nodes; the application scenario is all within the map. Constructing high-dimensional map data ,in for The OK; (1-2) Based on high-dimensional map data The GCN network is used to learn the association information between nodes in high-dimensional map data. Specifically, the GCN network processes the forward, backward, left, and right neighbors based on the four connection types of lane nodes, as shown below: ,in, The weights of the original information of the lane nodes are set as an identity matrix; j Indicates the first j Types of connections, These represent the physical connections of the lane node's forward, backward, left, and right neighbors on the map, respectively. and They represent the first Adjacency matrices and weight matrices for various connection types; (1-3) Selecting whether lane nodes have ground markings or signs as an additional weight. The restriction is implemented as follows: ;get ;in, This is the identification matrix related to ground markings and signs, i.e., the connection relationship matrix of lane nodes that do not have ground markings and signs. (1-4) Based on step (1-3), the filtered results are obtained. Finally, the high-dimensional map feature information is represented as follows: ;in To expand the number of scaling factors, It is the first Each scale factor.
4. The agent action prediction method based on multi-scale spatial perception according to claim 1, characterized in that, The historical trajectory feature extraction employs convolutional neural networks and feature pyramid networks to extract high-dimensional trajectory data features of all agents within the application scenario map, including: A CNN network is used to extract features from the historical behavioral trajectory data of all agents in the scene, thus obtaining historical trajectory features. T Simultaneously, a Feature Pyramid Network (FPN) is used to fuse historical trajectory features. T The feature values at different scales are used to output high-dimensional trajectory data features with time series characteristics. .
5. The agent action prediction method based on multi-scale spatial perception according to claim 4, characterized in that, High-dimensional trajectory data features Based on historical trajectory characteristics T The information of different nodes in the data is obtained through a concatenation operation, and is represented as follows: ,in yes The first of the characteristic variables Historical trajectory node feature information This indicates that MLP encoding is used for different historical trajectory nodes. , The process of extracting location difference feature information, and then through Perform a direct stitching operation to obtain high-dimensional trajectory data features. .
6. The agent action prediction method based on multi-scale spatial perception according to claim 1, characterized in that, The multimodal action prediction utilizes agent trajectory fusion features for prediction regression and confidence scoring, outputting the multimodal complete trajectory coordinates and corresponding confidence scores of the action prediction, providing reasonable auxiliary decision-making for agent actions, including: (4-1) Fuse the extracted agent trajectory features Regression calculations are performed using the first MLP branch network to generate the BEV coordinates of the complete multimodal trajectory, represented as follows: , Indicates the first The first intelligent agent Article 1 of the action Step BEV coordinates, For configurable parameters, it means Predicting movement trajectories; (4-2) Using the BEV coordinates of the multimodal complete trajectory obtained in (4-1) Features fused with agent trajectory Formed through a second MLP branch network The confidence score for the prediction of each action is expressed as: ,in Indicates the first The first agent of the intelligent agent The confidence score for predicting behavioral trajectories.
7. The agent action prediction method based on multi-scale spatial perception according to claim 1, characterized in that, During the training process of the MLP for multimodal action prediction, the BEV coordinates of the complete multimodal trajectory are used. and confidence score Based on the ground truth trajectory values of each agent in the existing dataset, the classification loss and regression loss in the loss function adopt the maximum marginal loss and the smoothed L1 loss, respectively. ,in , The loss function is used to train the model; specifically, The number of agents. This represents the boundary value of the loss function, which is 0.
2. It is an intelligent agent m exist The true value of the BEV coordinates of the step. , yes The One element, It is a smoothed L1 loss.
8. An action prediction device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes the computer program, it implements the agent action prediction method based on multi-scale spatial perception as described in any one of claims 1-7.
9. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements the agent action prediction method based on multi-scale spatial perception as described in any one of claims 1-7.