Heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search

Abstract

The invention discloses a heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search. The method comprises the following steps: modeling a heterogeneous scene graph in a multi-agent automatic driving scene; constructing a multi-agent trajectory prediction model based on graph neural architecture search; and performing trajectory prediction in the heterogeneous scene graph by using the multi-agent trajectory prediction model. The invention provides a novel trajectory prediction framework based on graph neural architecture search. The innovative design of the framework is composed of a flexible, customizable and efficient search space, a super network search strategy and a subnet contribution evaluation strategy. The invention further designs a new NAS loss function which integrates the road semantic layout, the utility constraint, the model calculation complexity constraint and the multi-modal prediction trajectory precision loss so as to optimize the balance between the reasoning speed and the model precision.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search. Background Art

[0002] Multi-agent trajectory prediction is of great significance in fields such as autonomous driving and intelligent robotics. It aims to predict the future trajectories of multiple agents in a driving scenario based on their current and historical locations, traffic conditions, and their intentions and behavior patterns. Graph neural networks have been adopted by most trajectory prediction algorithms due to their superior ability to understand agent interactions and model the scene's semantic context. Due to data limitations, early graph network-based trajectory prediction methods typically focused on predicting trajectories based on interactions between a single type of traffic participant. Recent advances in optical cameras, lidar, and 3D point cloud technologies have led to the release of large-scale autonomous driving datasets, which have led to a greater diversity of scenarios and data types for trajectory prediction. Input data may include the motion state of the agent, traffic lights with positional information, and lane polygons, making it possible to obtain rich information about interactions between heterogeneous traffic participants. Furthermore, scene semantic context and social behavioral constraints have become increasingly research topics. Scene semantics and social constraints refer to the fact that in mixed traffic scenarios, the movement of agents is often influenced by the scene's semantic layout, traffic rules, and road utilities. Traffic participants evaluate their behavior based on the current traffic scenario and take appropriate measures to avoid collisions or violations of traffic rules. HRG-HSG introduces heterogeneous risk graphs and scene graphs to explicitly simulate the impact of road semantic layout on road scene syntax, thereby enabling trajectory prediction for heterogeneous agents. EDRF employs an enhanced driving risk field model, combining collision risk with a Gaussian distribution model to quantitatively measure the uncertainty of traffic participants' behavior.

[0003] However, most existing methods rely on manually designed neural network architectures, which often encounter the following difficulties:

[0004] 1) The neural network design process relies on extensive manual work and professional experience. Designing a satisfactory network often requires continuous experimentation and adjustment of the constructed network. This process is tedious and time-consuming. The designed network often fails to fully explore the impact of different network architectures and hyperparameters on the accuracy of the resulting model, and typically only achieves a local optimum.

[0005] 2) As the amount of data information and features available in datasets increases, manually designed neural networks often find it difficult to balance their inference speed and accuracy levels, posing a challenge to edge deployment with limited computing resources.

[0006] Some methods have proposed using neural architecture search to address the aforementioned issues. However, these works typically decompose the trajectory prediction task into multiple stages, then search the modular operations of each stage using specific search spaces and strategies to obtain the corresponding CNN or RNN models. However, this staged search approach can easily lead to the accumulation of errors during the search process, affecting search efficiency. In addition, current models based on CNN or RNN structures often fail to capture the interactions of heterogeneous scene information and cannot effectively utilize the rich semantic features contained in maps. Furthermore, due to inconsistent evaluation criteria, these methods often ignore physical and social behavioral constraints, and the resulting models may not be applicable to different driving scenarios or datasets and lack interpretability. Summary of the Invention

[0007] This paper aims to address these issues through graph neural architecture search and related supporting technologies. Specifically, it predicts the trajectories of multiple agents based on heterogeneous feature input data from multiple interacting agents of different semantic types and physical and social behavioral constraints, automatically performing an end-to-end search task to find the optimal graph network model for trajectory prediction.

[0008] To achieve the above objectives, the present invention provides a heterogeneous multi-agent trajectory prediction method based on graph neural architecture search, comprising the following steps:

[0009] Modeling heterogeneous scene graphs in multi-agent autonomous driving scenarios;

[0010] Build a multi-agent trajectory prediction model based on graph neural architecture search;

[0011] The multi-agent trajectory prediction model is used to perform trajectory prediction in the heterogeneous scene graph.

[0012] Preferably, the heterogeneous scene graph blocks constructed are:

[0013] G=(V,E)

[0014] Among them, the node V represents the isomer; the edge E represents the interaction between isomers.

[0015] Preferably, the constructed multi-agent trajectory prediction model consists of many encapsulated convolution operator modules, which participate in the calculation of each search layer; each search layer determines a convolution operator as its search result.

[0016] Preferably, the method for performing the trajectory prediction includes: modeling the collision risk as a graph structure, and considering the representation of the agent at time t as a node:

[0017]

[0018] in, is the final node representation at time t, LSTM() represents the LSTM network constructed by the fully connected layer and the ReLU function; (a1, a2, ..., a j ) are the physical properties of all agents, including position, velocity, orientation angle, and size at the current time step;

[0019] Use collision risk as edge feature:

[0020]

[0021] Among them, |T ij | represents the collision time, Indicates the risk of movement direction, is the agent attribute risk.

[0022] Preferably, the collision risk is inversely proportional to the collision time, and the formula is expressed as:

[0023]

[0024] Where, and represents the coordinates of agent i and agent j at time t, α and β represent the angles between their forward directions and the horizontal axis, γ represents the angle between the connecting line of the two agents and the horizontal axis, and v i and v j represents the speed of agents i and j;

[0025] If agent i appears within the field of view of agent j, it is determined that there is a collision risk between them in the moving direction:

[0026]

[0027] Among them, μ is a heterogeneous distance parameter that reflects the radius of the area considered for potential risks, and the μ value is different for different types of traffic participants. t is the cosine value of the heading angle deviation between agent i and agent j.

[0028] Preferably, the mixed weights are mixed by introducing the mixed weight back propagation operator of the softmax calculation and updating it in the supernet. Finally, the final output feature is obtained:

[0029]

[0030]

[0031] Where z out is the final representation of the node; represents the features of the i-th intermediate layer; σ represents a nonlinear activation function; represents the network weights shared by each node aggregator from the candidate architectures in the search space; represents the input hidden features; θ n represents the node aggregation operation; θ l represents the inter-layer aggregation operation; θ s Represents a skip connection operation.

[0032] The present invention also provides a heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, which is used to implement the above method and includes: a scene construction module, a model construction module and a trajectory prediction module;

[0033] The scenario construction module is used to model heterogeneous scene graphs in multi-agent autonomous driving scenarios;

[0034] The model building module is used to build a multi-agent trajectory prediction model based on graph neural architecture search;

[0035] The trajectory prediction module is used to perform trajectory prediction in the heterogeneous scene graph using the multi-agent trajectory prediction model.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This paper proposes a novel trajectory prediction framework based on graph neural architecture search. Its innovative design consists of a flexible, customizable, and efficient search space, a hypernetwork search strategy, and a subnetwork contribution evaluation strategy. TrajGNAS is able to autonomously perform end-to-end search tasks to produce advanced trajectory prediction models with strong performance. This paper also designs a new NAS loss function that integrates road semantic layout, utility constraints, model computational complexity constraints, and multimodal prediction trajectory accuracy loss to optimize the balance between inference speed and model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of a multi-agent trajectory prediction model according to an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the multi-agent position relationship according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] like Figure 1 FIG. 1 is a flow chart of the method of this embodiment, and the steps include:

[0046] S1. Modeling heterogeneous scene graphs in multi-agent autonomous driving scenarios.

[0047] Before trajectory prediction, a heterogeneous scene graph G = (V, E) needs to be modeled, where nodes V represent heterogeneous entities (e.g., agents, lanes, or stop signs) and edges E represent the interactions between them.

[0048] S2. Construct a multi-agent trajectory prediction model based on graph neural architecture search.

[0049] Generally speaking, GNN (graph neural network) models are unified by neighbor aggregation or message passing models, where the representation of each node is learned by iteratively aggregating the embeddings of its neighbors. A key component of this is the predefined aggregation function, which, as a convolution operator, varies across different graph neural networks. Node aggregation functions and inter-layer aggregation functions are often the focus of manually designed graph neural networks. Many trajectory prediction models based on graph neural networks choose to stack multiple layers of the same aggregation function to aggregate adjacent features.

[0050] This embodiment combines different aggregation functions to further improve the performance of the graph network. Figure 2 As shown in , the constructed multi-agent trajectory prediction model (TrajGNAS) focuses on searching for aggregation functions in graph networks. Figure 2 As shown in Figure 2, TrajGNAS focuses on searching for aggregation functions in graph networks.

[0051] The search space of TrajGNAS consists of many encapsulated convolution operator modules, called units, which participate in the calculation of each search layer; ultimately, each search layer determines a convolution operator as its search result. Therefore, as a fundamental part of the search process, the type of operator modules involved in the search directly affects the final model structure. In order to optimize the search space, this embodiment refers to classic trajectory prediction and graph network task solving methods, and further introduces efficient convolution operator types. In addition, considering that the residual mechanism can improve the performance of graph neural networks, residual connectivity is also used as part of the search process.

[0052] (1) Node message aggregation operation:

[0053] For node convolution operator candidates, this example selects five convolution operators based on popular GNN models: GCN, GAT, ST-GCN, AI-GAT, and EDGE, as well as a parameter-free operation (non-GNN). GCN and GAT are powerful and widely used basic graph aggregation functions. They have excellent neighbor message aggregation capabilities and a simple structure, which helps reduce their weight. Autonomous driving scenes often contain rich spatiotemporal information; therefore, ST-GCN is widely used as a spatiotemporal graph convolutional network model for trajectory prediction. It dynamically captures the relationships between nodes in time and space, which is crucial for long-term trajectory prediction tasks, while AI-GAT and EDGE focus on the heterogeneous elements of the scene and their interactions. AI-GAT uses an improved graph aggregation function to improve its graph structure. AI-GAT uses an enhanced graph attention module to construct weight matrices for semantically distinct nodes. In contrast, EDGE overcomes the limitations of GAT, which can only use binary edge features, and GCN, which can only use one-dimensional edge features. It utilizes edge connection enhancement to fully utilize the edge connection characteristics between different types of nodes.

[0054] (2) Inter-layer aggregation and connection operations:

[0055] This example selects three inter-layer aggregation functions as candidate functions: CONCAT, MAX, and LSTM. Furthermore, two jump operators are added to the search space: ZERO and Identity. The ZERO operation prohibits message passing and helps eliminate redundant aggregation operations in the architecture, while the Identity operation allows message passing without aggregating neighbor information to the next layer. This design makes the overall search space more flexible.

[0056] S3. Use a multi-agent trajectory prediction model to perform trajectory prediction in heterogeneous scene graphs.

[0057] Real-world traffic scenarios are often dynamic and full of uncertainty. Uncertainty in cognitive predictions and lack of risk awareness are major challenges involving autonomous driving systems. Therefore, it is important to introduce a large amount of additional information when optimizing architecture search and prediction results, especially when training data is insufficient or scenarios are diverse. Therefore, in order to consider the role of risk in the search for model aggregation functions, this embodiment chooses to model collision risk as a graph structure. Specifically, the representation of the agent at time t is regarded as a node, and the node representation of traffic participants is usually modeled by their physical characteristics, such as their position, speed, facing angle, and size. It can be expressed as:

[0058]

[0059] in, is the final node representation at time t, and LSTM() represents the LSTM network constructed by the fully connected layer and the ReLU function. (a1, a2, ..., a j ) are the physical properties of all agents, namely their position, velocity, facing angle, and size at the current time step.

[0060] For edge representation of the graph, this embodiment uses collision risk as the edge feature. The distance, attributes, offset angle, and speed between agents are all key factors in estimating collision risk. Therefore, the collision risk between agents is quantified as:

[0061]

[0062] where |T ij | represents the collision time, Indicates the risk of movement direction, is the agent attribute risk. The details are as follows:

[0063] The collision risk is inversely proportional to the collision time, and the formula is:

[0064]

[0065] Where, and represents the coordinates of agent i and agent j at time t, α and β represent the angles between their forward directions and the horizontal axis, γ represents the angle between the connecting line of the two agents and the horizontal axis, and v i and v j represents the speed of agents i and j, such as Figure 3 shown.

[0066] The moving direction risk means that if agent i appears within the field of view of agent j, there is a collision risk between them in the moving direction. The formula is expressed as:

[0067]

[0068] Among them, μ is a heterogeneous distance parameter that reflects the radius of the area considered for potential risks, and the μ value is different for different types of traffic participants. t is the cosine value of the heading angle deviation between agents i and j. The closer their moving angles are, the greater the risk of collision.

[0069] Agent attribute risk refers to the fact that different types of traffic participants (pedestrians, vehicles, cyclists) typically have different physical attributes (speed, mass, volume) and social attributes (vulnerable groups, whether they are protected by traffic rules, etc.), and their assessment criteria for road risks may also vary. With this in mind, the search strategy distinguishes the risk factors of different semantic types of agents in the scene. A simple MLP structure is used to model the interaction between different agent attributes. Agent attribute risk can be expressed as:

[0070]

[0071] in, and are node representations obtained by formula (1), which are used here to characterize the physical properties they contain. i and m j are the qualities of agent i and agent j, respectively, c type is a variable representing the agent type, which indicates the specific risk coefficient of different types of traffic participants.

[0072] For a multi-layer GNN, the search space is usually composed of aggregation operators and jump operations of multiple nodes. A discrete search space is formed by selecting appropriate operators on each edge. This embodiment relaxes the classification selection of specific operations to the softmax of all possible operations:

[0073]

[0074] Among them, the operator mixing weight of nodes i and j is composed of the parameter vector express, Represents the search space, which includes node message aggregation operation, inter-layer aggregation operation and connection operation; this embodiment adopts three corresponding θ n ,θ l ,θ s To distinguish different types of operation weights. These coefficients can distinguish the importance of different operators and will participate in the training process of the super network. o(x) represents the input hidden feature of the current layer of GNN

[0075] By introducing the mixed weights of the softmax calculation, the operator mixed weights can be back-propagated And update it in the supernet. Finally, the final output feature can be obtained, which is expressed as:

[0076]

[0077]

[0078] Where z out is the final representation of the node; represents the features of the i-th intermediate layer; σ represents a nonlinear activation function; represents the network weights shared by each node aggregator from the candidate architectures in the search space; represents the input hidden features; θ n represents the node aggregation operation; θ l represents the inter-layer aggregation operation; θ s Represents a skip connection operation.

[0079] After getting the final node representation z out Finally, different loss or guidance mechanisms are used according to the needs of the task.

[0080] The task of TrajGNAS architecture search is to learn the continuous variable θ n ,θ l ,θ s and network weights w. The architecture parameters θ are optimized on the validation set, while the network weights w are optimized on the training set. It is worth noting that when optimizing the architecture parameters and network weights, this embodiment not only searches the heterogeneous agent graph, but also the constructed risk interaction perception graph structure, using a pre-designed task-adaptive loss function. This combines agent historical trajectories, agent interaction mechanisms, and collision risk at the hypernet feature level to guide the search for the optimal network architecture.

[0081] In the fine-tuning part, this example also optimizes the hyperparameters of the risk interaction perception graph. The gradient descent method is used to update the parameters:

[0082]

[0083] Where w is the weight of the current risk interaction perception graph, ξ is the internal optimization learning rate of w, and this embodiment only needs to use a single training step to adjust w to approximate w * (θ).

[0084] Example 2

[0085] Trajectory prediction models in real-world driving scenarios require not only high accuracy but also inference speed to cope with emergencies and ensure safety and reliability. Furthermore, in order to fully utilize the risk-awareness guidance mechanism of the present invention in search, reflect the constraints imposed by physical and social behavior on the agent's trajectory, and enhance interpretability, this embodiment designs a loss function based on posterior knowledge and social constraints as a key metric for evaluating search network models.

[0086] (1) Semantic region loss:

[0087] In the real world, the behavior of traffic participants is usually subject to traffic rules and social ethics. For example, on the road, vehicles should drive on the motor vehicle lane, pedestrians should go on the sidewalk or crosswalk, and bicycles should drive on the non-motor vehicle lane. Therefore, the trajectory predicted by the model is used as a posteriori knowledge to verify the rationality of the agent in exercising the right of way and whether it complies with the constraints of traffic rules. If it is within the constraint area, its regional loss is defined as 0. Otherwise, the normalized distance from it to the constraint area is calculated, and the minimum value is taken as the regional loss of the point. The final regional loss is the sum of the squared normalized distances for all time steps and objects. Their calculation formula is as follows:

[0088]

[0089] Among them, d t，n It is the prediction point The distance to the nearest boundary of the region, for traffic participants whose predicted trajectories are not on their corresponding road semantics. The normalization factor is the maximum radius of the region d max ε is a custom coefficient. This loss encourages the model to fully consider road rules and social constraints when predicting the trajectories of different agents, ensuring that the predicted trajectories are as close to the corresponding area as possible, and the farther away from the boundary, the greater the penalty.

[0090] (2) Joint loss of accuracy and complexity:

[0091] This embodiment also takes into account the balance between model computational complexity and accuracy, and designs a multi-index loss function, which includes an accuracy loss function, as well as considerations of network sparsity, inference speed, number of parameters, and computational effort. The evaluation of model prediction accuracy is achieved through the negative log-likelihood loss (NLL-loss) of the bivariate normal distribution:

[0092]

[0093] Among them, σ x and σ yis the standard deviation in the x and y directions, ρ is the correlation coefficient between x and y, and Δx and Δy represent the deviation between the predicted value and the true value.

[0094] To constrain model complexity, this embodiment integrates multiple neural network computational properties and guides TrajGNAS's search direction by backpropagating loss values during super-network training. First, the computational loss and sparsity loss are defined. The computational loss reflects the overall computational complexity of the model, focusing on floating-point numbers and the number of parameters. In addition, in actual deployment, trajectory prediction algorithms are often limited by the computing power of hardware devices, so the model's FLOPS should be limited. This embodiment provides a new limit coefficient for the floating-point loss when calculating it, so that the model's FLOPS during training is more severely penalized when it exceeds the hardware's tolerance. The formula is:

[0095]

[0096] Where, is the set of all modules of the model, W conv is the convolution kernel weight tensor, K h With K w is the height and width of the convolution kernel, C out is the number of output channels, W fc is the weight tensor of the fully connected layer, represents the total number of parameters, λ f is the computational loss weight parameter. Is the penalty coefficient used to adjust the value of the computational burden loss. It can be expressed as:

[0097]

[0098] Where T is the expected Flops limit, and τ is a value set in actual deployment, which can be adjusted according to the situation and is usually much larger than 1.

[0099] In addition, enhancing the sparsity of neural networks is of great significance for improving computational efficiency and enhancing model generalization performance. The loss of network sparsity is defined as:

[0100]

[0101] Where P is the set of all model parameters, ||P i ||1 is the L1 norm of the parameter, λ s is the sparsity loss weight.

[0102] The final loss of this embodiment is a weighted average of all the above losses. By introducing a multi-index loss function based on posterior knowledge and social constraints, the trajectory prediction model can achieve an optimal balance between accuracy and speed while being interpretable. The total loss function is expressed as:

[0103]

[0104] in The weights for each loss can be defined, and users can adjust the weight values as needed to achieve a more flexible network architecture search.

[0105] Example 3

[0106] In order to verify the advancement of the present invention compared to the prior art, the following experiments were conducted for comparison.

[0107] Experiment 1

[0108] Experiments were conducted on three large-scale autonomous driving datasets, nuScenes, Agroverse, and ApolloScape, comparing the proposed method with classic and state-of-the-art trajectory prediction models. The experimental results on ApolloScape are shown in Table 1. The TrajGANS framework outperforms other candidate methods in both the wADE and wFDE metrics, achieving improvements of 3.8% and 3.2% respectively over the second-place candidate. The experimental results on the Agroverse dataset, shown in Table 2, show that the proposed method ranks third in the mADE metric but is the state-of-the-art method in the mFDE metric. The experimental results on nuScenes are shown in Table 3. When K = 5 and K = 10, the proposed model achieves the best performance in terms of the MR evaluation metric, achieving average improvements of 31.6% and 50% compared to the second-place method, DMAP, respectively. This result demonstrates that the proposed model is able to more accurately predict the final position of the agent. Furthermore, the inference time of TrajGNAS was evaluated, as shown in Table 4. TrajGNAS significantly outperforms other methods in terms of inference speed. Therefore, faced with more complex scenarios and larger data volumes, the simple and unified network structure searched by TrajGNAS is crucial for performing fast real-time analysis and processing tasks and maximizing safety. Therefore, TrajGNAS is well suited for the autonomous driving industry.

[0109] Table 1

[0110] method years WSADE ADE ADEp ADEb WSFDE FDE FDEp FDE TPNet 2020 1.280 2.210 0.740 1.850 2.340 3.860 1.410 3.400 Grip++ 2020 1.259 2.240 0.714 1.802 2.363 4.076 1.373 3.416 MCCITP 2021 1.200 2.090 0.700 1.720 2.240 3.770 1.340 3.210 S2TNet 2021 1.168 1.987 0.683 1.700 2.180 3.578 1.305 3.215 AI-TP 2022 1.160 - - - 2.130 - - - D2-Tpred 2022 1.020 - - - 1.690 - - - HRG-HSG 2023 1.036 - - - 1.321 - - - STF 2023 1.384 2.403 0.799 2.001 1.707 2.972 1.012 2.392 MDSTF 2024 1.117 1.868 0.674 1.601 2.044 3.333 1.272 2.907 TP-EGT 2024 1.190 2.050 0.700 1.720 2.140 3.530 1.280 3.160 TrajGNAS (the present invention) - 0.982 1.258 0.701 1.470 1.291 1.905 0.802 2.022

[0111] Table 2

[0112] method years mADE mFDE NLNI 2021 0.79 1.25 THOMAS 2021 0.94 1.44 GOHOME 2022 0.89 1.29 DenseTNT 2021 0.88 1.28 LaneGCN 2020 0.87 1.36 LTP 2022 0.83 1.29 FRM 2023 0.82 1.27 HiVT 2022 0.77 1.17 ProphNet 2023 0.77 1.14 LAformer 2024 0.77 1.16 TrajGNAS (the present invention) - 0.82 1.10

[0113] Table 3

[0114]

[0115] Table 4

[0116] Model Batch size Inference time (ms) HRG-HSG 8 43.75 DenseTNT 16 88.5 HiVT 16 6.3 PGP 12 17.9 TrajGNAS 8 35.65 TrajGNAS-loss 12 17.5

[0117] Experiment 2

[0118] In order to demonstrate the effectiveness of the proposed method, various variants of the model are evaluated in Table 5. Among them, (1) only using graph neural architecture search; (2) adding risk interaction guidance mechanism; (3) introducing regional semantic constraint loss; (4) introducing computational complexity constraint. The results show that each module designed by the present invention plays its own role. The risk interaction mechanism and regional semantic constraint loss proposed by the present invention significantly improve the accuracy of the model. In terms of accuracy indicators, the introduction of the risk interaction mechanism significantly improves the mADE, mFDE and MR indicators of the model. In addition, by introducing the computational loss function, TrajGNAS effectively guides the network search process, reducing the number of parameters and computational complexity of the final search model; however, this configuration has no significant impact on the accuracy. mADE and mFDE only increased by 7% and 10.6% respectively, and the MR performance index did not decrease. It is worth noting that since model 1 only uses GNAS, model 1 has fewer parameters and lower computational burden compared with models 2 and 3; however, under the computational loss constraint, the number of parameters of structure 4 is still smaller than that of structure 1, and the mADE and mFDE indicators predicted by the model are improved by 53% and 44% respectively.

[0119] Table 5

[0120] Model (ID) GNAS RI Region-Loss Compute-Loss mADE mFDE MR Params FLOPS(MB) 1 √ 1.532 1.911 0.53 1089K 8.961 2 √ √ 1.103 1.518 0.29 1315K 9.346 3 √ √ √ 0.990 1.312 0.09 1364K 9.764 4 √ √ √ √ 0.997 1.326 0.09 1031K 8.716 .

[0121] Example 4

[0122] This embodiment also provides a heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, comprising a scenario construction module, a model construction module, and a trajectory prediction module. The scenario construction module is used to model the heterogeneous scene graph in a multi-agent autonomous driving scenario; the model construction module is used to construct a multi-agent trajectory prediction model based on graph neural architecture search; and the trajectory prediction module is used to perform trajectory prediction in the heterogeneous scene graph using the multi-agent trajectory prediction model.

[0123] The following will describe in detail how the present invention solves technical problems in practical work in conjunction with this embodiment.

[0124] First, the scenario construction module is used to model the heterogeneous scene graph in the multi-agent autonomous driving scenario.

[0125] Before trajectory prediction, a heterogeneous scene graph G = (V, E) needs to be modeled, where nodes V represent heterogeneous entities (e.g., agents, lanes, or stop signs) and edges E represent the interactions between them.

[0126] The model building module then constructs a multi-agent trajectory prediction model based on graph neural architecture search.

[0127] Generally speaking, GNN (graph neural network) models are unified by neighbor aggregation or message passing models, where the representation of each node is learned by iteratively aggregating the embeddings of its neighbors. A key component of this is the predefined aggregation function, which, as a convolution operator, varies across different graph neural networks. Node aggregation functions and inter-layer aggregation functions are often the focus of manually designed graph neural networks. Many trajectory prediction models based on graph neural networks choose to stack multiple layers of the same aggregation function to aggregate adjacent features.

[0128] This embodiment combines different aggregation functions to further improve the performance of the graph network. Figure 2 As shown in , the constructed multi-agent trajectory prediction model (TrajGNAS) focuses on searching for aggregation functions in graph networks. Figure 2 As shown in Figure 2, TrajGNAS focuses on searching for aggregation functions in graph networks.

[0129] The search space of TrajGNAS consists of many encapsulated convolution operator modules, called units, which participate in the calculation of each search layer; ultimately, each search layer determines a convolution operator as its search result. Therefore, as a fundamental part of the search process, the type of operator modules involved in the search directly affects the final model structure. In order to optimize the search space, this embodiment refers to classic trajectory prediction and graph network task solving methods, and further introduces efficient convolution operator types. In addition, considering that the residual mechanism can improve the performance of graph neural networks, residual connectivity is also used as part of the search process.

[0130] (1) Node message aggregation operation:

[0131] For node convolution operator candidates, this example selects five convolution operators based on popular GNN models: GCN, GAT, ST-GCN, AI-GAT, and EDGE, as well as a parameter-free operation (non-GNN). GCN and GAT are powerful and widely used basic graph aggregation functions. They have excellent neighbor message aggregation capabilities and a simple structure, which helps reduce their weight. Autonomous driving scenes often contain rich spatiotemporal information; therefore, ST-GCN is widely used as a spatiotemporal graph convolutional network model for trajectory prediction. It dynamically captures the relationships between nodes in time and space, which is crucial for long-term trajectory prediction tasks, while AI-GAT and EDGE focus on the heterogeneous elements of the scene and their interactions. AI-GAT uses an improved graph aggregation function to improve its graph structure. AI-GAT uses an enhanced graph attention module to construct weight matrices for semantically distinct nodes. In contrast, EDGE overcomes the limitations of GAT, which can only use binary edge features, and GCN, which can only use one-dimensional edge features. It utilizes edge connection enhancement to fully utilize the edge connection characteristics between different types of nodes.

[0132] (2) Inter-layer aggregation and connection operations:

[0133] This example selects three inter-layer aggregation functions as candidate functions: CONCAT, MAX, and LSTM. Furthermore, two jump operators are added to the search space: ZERO and Identity. The ZERO operation prohibits message passing and helps eliminate redundant aggregation operations in the architecture, while the Identity operation allows message passing without aggregating neighbor information to the next layer. This design makes the overall search space more flexible.

[0134] Finally, the path prediction module uses a multi-agent trajectory prediction model to perform trajectory prediction in heterogeneous scene graphs.

[0135] Real-world traffic scenarios are often dynamic and full of uncertainty. Uncertainty in cognitive predictions and lack of risk awareness are major challenges involving autonomous driving systems. Therefore, it is important to introduce a large amount of additional information when optimizing architecture search and prediction results, especially when training data is insufficient or scenarios are diverse. Therefore, in order to consider the role of risk in the search for model aggregation functions, this embodiment chooses to model collision risk as a graph structure. Specifically, the representation of the agent at time t is regarded as a node, and the node representation of traffic participants is usually modeled by their physical characteristics, such as their position, speed, facing angle, and size. It can be expressed as:

[0136]

[0137] in, is the final node representation at time t, and LSTM() represents the LSTM network constructed by the fully connected layer and the ReLU function. (a1, a2, ..., a j) are the physical properties of all agents, namely their position, velocity, facing angle, and size at the current time step.

[0138] For edge representation of the graph, this embodiment uses collision risk as the edge feature. The distance, attributes, offset angle, and speed between agents are all key factors in estimating collision risk. Therefore, the collision risk between agents is quantified as:

[0139]

[0140] where |T ij | represents the collision time, Indicates the risk of movement direction, is the agent attribute risk. The details are as follows:

[0141] The collision risk is inversely proportional to the collision time, and the formula is:

[0142]

[0143] Where, and represents the coordinates of agent i and agent j at time t, α and β represent the angles between their forward directions and the horizontal axis, γ represents the angle between the connecting line of the two agents and the horizontal axis, and v i and v j represents the speed of agents i and j, such as Figure 3 shown.

[0144] The moving direction risk means that if agent i appears within the field of view of agent j, there is a collision risk between them in the moving direction. The formula is expressed as:

[0145]

[0146] Among them, μ is a heterogeneous distance parameter that reflects the radius of the area considered for potential risks, and the μ value is different for different types of traffic participants. t is the cosine value of the heading angle deviation between agents i and j. The closer their moving angles are, the greater the risk of collision.

[0147] Agent attribute risk refers to the fact that different types of traffic participants (pedestrians, vehicles, cyclists) typically have different physical attributes (speed, mass, volume) and social attributes (vulnerable groups, whether they are protected by traffic rules, etc.), and their assessment criteria for road risks may also vary. With this in mind, the search strategy distinguishes the risk factors of different semantic types of agents in the scene. A simple MLP structure is used to model the interaction between different agent attributes. Agent attribute risk can be expressed as:

[0148]

[0149] in, and are node representations obtained by formula (17), which are used here to characterize the physical properties they contain. i and m j are the qualities of agent i and agent j, respectively, c type is a variable representing the agent type, which indicates the specific risk coefficient of different types of traffic participants.

[0150] For a multi-layer GNN, the search space is usually composed of aggregation operators and jump operations of multiple nodes. A discrete search space is formed by selecting appropriate operators on each edge. This embodiment relaxes the classification selection of specific operations to the softmax of all possible operations:

[0151]

[0152] Among them, the operator mixing weight of nodes i and j is composed of the parameter vector express, Represents the search space, which includes node message aggregation operation, inter-layer aggregation operation and connection operation; this embodiment adopts three corresponding θ n ,θ l ,θ s To distinguish different types of operation weights. These coefficients can distinguish the importance of different operators and will participate in the training process of the super network. o(x) represents the input hidden feature of the current layer of GNN

[0153] By introducing the mixed weights of the softmax calculation, the operator mixed weights can be back-propagated And update it in the supernet. Finally, the final output feature can be obtained, which is expressed as:

[0154]

[0155]

[0156] Where zout is the final representation of the node; represents the features of the i-th intermediate layer; σ represents a nonlinear activation function; represents the network weights shared by each node aggregator from the candidate architectures in the search space; represents the input hidden features; θ n represents the node aggregation operation; θ l represents the inter-layer aggregation operation; θ s Represents a skip connection operation.

[0157] After getting the final node representation z out Finally, different loss or guidance mechanisms are used according to the needs of the task.

[0158] The task of TrajGNAS architecture search is to learn the continuous variable θ n ,θ l ,θ s and network weights w. The architecture parameters θ are optimized on the validation set, while the network weights w are optimized on the training set. It is worth noting that when optimizing the architecture parameters and network weights, this embodiment not only searches the heterogeneous agent graph, but also the constructed risk interaction perception graph structure, using a pre-designed task-adaptive loss function. This combines agent historical trajectories, agent interaction mechanisms, and collision risk at the hypernet feature level to guide the search for the optimal network architecture.

[0159] In the fine-tuning part, this example also optimizes the hyperparameters of the risk interaction perception graph. The gradient descent method is used to update the parameters:

[0160]

[0161] Where w is the weight of the current risk interaction perception graph, ξ is the internal optimization learning rate of w, and this embodiment only needs to use a single training step to adjust w to approximate w * (θ).

[0162] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A heterogeneous multi-agent trajectory prediction method based on graph neural architecture search, characterized by the following steps: include: Modeling heterogeneous scene graphs in multi-agent autonomous driving scenarios; Build a multi-agent trajectory prediction model based on graph neural architecture search; The multi-agent trajectory prediction model is used to perform trajectory prediction in the heterogeneous scene graph.

2. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1 is characterized in that The heterogeneous scene graph constructed: G=(V,E) Among them, the node V represents the isomer; the edge E represents the interaction between isomers.

3. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1 is characterized in that The constructed multi-agent trajectory prediction model consists of many encapsulated convolution operator modules, which participate in the calculation of each search layer; each search layer determines a convolution operator as its search result.

4. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1 is characterized in that The method for performing the trajectory prediction includes modeling the collision risk as a graph structure, and considering the representation of the agent at time t as a node: in, is the final node representation at time t, LSTM() represents the LSTM network constructed by the fully connected layer and the ReLU function; (a1, a2, ..., a j ) are the physical properties of all agents, including position, velocity, orientation angle, and size at the current time step; Use collision risk as edge feature: Among them, |T ij | represents the collision time, Indicates the risk of movement direction, is the agent attribute risk.

5. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 4 is characterized in that: The collision risk is inversely proportional to the collision time, and the formula is: Where, and represents the coordinates of agent i and agent j at time t, α and β represent the angles between their forward directions and the horizontal axis, γ represents the angle between the connecting line of the two agents and the horizontal axis, and v i and v j represents the speed of agents i and j; If agent i appears within the field of view of agent j, it is determined that there is a collision risk between them in the moving direction: Among them, μ is a heterogeneous distance parameter that reflects the radius of the area considered by potential risks. Different types of traffic participants have different μ values; Δ t is the cosine value of the heading angle deviation between agent i and agent j.

6. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 5 is characterized in that: By introducing the mixed weight back propagation operator of the softmax calculation and updating it in the supernet, the final output feature is obtained: Where z out is the final representation of the node; represents the features of the i-th intermediate layer; σ represents a nonlinear activation function; represents the network weights shared by each node aggregator from the candidate architectures in the search space; Represents the input hidden features; θ n represents the node aggregation operation; θ l Represents the inter-layer aggregation operation; θ s Represents a skip connection operation.

7. A heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, the system being used to implement the method according to any one of claims 1 to 6, characterized in that: include: Scenario building module, model building module and trajectory prediction module; The scenario construction module is used to model heterogeneous scene graphs in multi-agent autonomous driving scenarios; The model building module is used to build a multi-agent trajectory prediction model based on graph neural architecture search; The trajectory prediction module is used to perform trajectory prediction in the heterogeneous scene graph using the multi-agent trajectory prediction model.

Images