A heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search

Abstract

The application discloses a heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search, wherein the method steps comprise: modeling a heterogeneous scene graph in a multi-agent automatic driving scene; constructing a multi-agent trajectory prediction model based on graph neural architecture search; and performing trajectory prediction in the heterogeneous scene graph by using the multi-agent trajectory prediction model. The application proposes a novel trajectory prediction framework based on graph neural architecture search, which is composed of a flexible, customizable and efficient search space, a super network search strategy and a subnet contribution evaluation strategy. The application also designs a new NAS loss function, which integrates road semantic layout, utility constraint, model calculation complexity constraint and multi-modal prediction trajectory accuracy loss to optimize the balance between inference speed and model accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to a heterogeneous multi-agent trajectory prediction method and system based on graph neural architecture search. BACKGROUND

[0002] Multi-agent trajectory prediction has great significance in the fields of automatic driving and intelligent robots. Multi-agent trajectory prediction aims to predict the future trajectories of multiple agents in a driving scene according to the current and historical position information of traffic participants, traffic conditions, and the intentions and behavior patterns of agents. Graph neural networks have been adopted by most trajectory prediction algorithms because of their excellent ability to understand agent interactions and model semantic context of the scene. Due to the limitation of data volume, early trajectory prediction methods based on graph networks usually only focus on the interaction between a single type of traffic participant to predict its trajectory. In recent years, with the development of optical cameras, lidar, 3D point cloud technology, and the release of some large-scale autonomous driving datasets, the trajectory prediction scene and data types have become more diverse. The input data may include the motion state of the agent, traffic lights with position information, lane line polygons, etc., which makes it possible to obtain rich interaction information between heterogeneous traffic participants. In addition, scene semantic context and social behavior constraints have also gradually become a research hotspot. Scene semantics and social constraints refer to the fact that in mixed traffic scenes, the motion of agents is often influenced by the semantic layout of the scene, traffic rules, and road utility. Traffic participants assess their behavior according to the current traffic scene and take appropriate measures to avoid collisions or violations of traffic rules. HRG-HSG introduces heterogeneous risk graphs and heterogeneous scene graphs to explicitly simulate the influence of road semantic layout on road scene syntax, thereby achieving trajectory prediction of heterogeneous agents. EDRF adopts an enhanced driving risk field model to quantitatively measure the uncertainty of traffic participant behavior by combining collision risk with a Gaussian distribution model.

[0003] However, most existing methods rely on artificially designed neural network architectures, which often encounter the following difficulties:

[0004] 1) The neural network design process relies on a large amount of manual work and professional experience. To design a satisfactory network, it is often necessary to continuously experiment and adjust the constructed network. This process is tedious and time-consuming, and the designed network often cannot fully explore the impact of different network architectures and hyperparameters on the accuracy of the resulting model, usually only achieving local optimization.

[0005] 2) As the amount of data information and features available in the dataset increases, artificially designed neural networks often struggle to balance their reasoning speed and accuracy level, posing challenges to edge deployment with limited computing resources.

[0006] Some methods propose to use neural architecture search to solve the above problems, but these works usually decompose the trajectory prediction task into multiple stages, and then search for modular operations for each stage through a specific search space and strategy, obtaining the corresponding CNN or RNN model. However, this stage-by-stage search method is prone to accumulate errors during the search process, affecting the efficiency of the search. In addition, the current models based on CNN or RNN structure often cannot capture the interaction of heterogeneous scene information, and cannot effectively utilize the rich semantic features contained in the map. In addition, due to inconsistent evaluation criteria, these methods often ignore the physical and social behavior constraints, and the resulting model may not be applicable to different driving scenarios or data sets, and lacks interpretability. SUMMARY

[0007] The present application aims to solve the existing problems through graph neural architecture search and related supporting technologies. Specifically, based on the heterogeneous feature input data from multiple interacting agents of different semantic types and physical and social behavior constraints, the trajectories of multiple agents are predicted, and an end-to-end search task is automatically performed to find the optimal graph network model for trajectory prediction.

[0008] To achieve the above purpose, the present application provides a heterogeneous multi-agent trajectory prediction method based on graph neural architecture search, the steps comprising:

[0009] Modeling the heterogeneous scene graph in the multi-agent autonomous driving scenario;

[0010] Constructing a multi-agent trajectory prediction model based on graph neural architecture search;

[0011] Using the multi-agent trajectory prediction model to perform trajectory prediction in the heterogeneous scene graph.

[0012] Preferably, the constructed heterogeneous scene graph plate:

[0013] G=(V,E)

[0014] Wherein, the node V represents the isomer; the edge E represents the interaction between the isomers.

[0015] Preferably, the constructed multi-agent trajectory prediction model is composed of many encapsulated convolution operator modules, which participate in the calculation of each search layer; each search layer determines a convolution operator as its search result.

[0016] Preferably, the method for performing the trajectory prediction comprises: modeling the collision risk as a graph structure, and regarding the representation of the agent at time t as a node:

[0017]

[0018] Wherein, LSTM() represents the LSTM network built by the fully connected layer and the ReLU function; (a1, a2,..., a j ) are the physical properties of all agents, including the position, velocity, facing angle and size at the current time step;

[0019] Use the collision risk as the edge feature:

[0020]

[0021] Where |T ij | represents the collision time, represents the motion direction risk, is the agent attribute risk.

[0022] Preferably, the collision risk is inversely proportional to the collision time, and the formula is:

[0023]

[0024] In the formula, and represent the coordinates of agent i and agent j at time t, α and β represent the angles between their forward directions and the horizontal axis, γ represents the angle between the connecting line of the two agents and the horizontal axis, v i and v j represent the speeds of agents i and j;

[0025] If agent i appears in the field of view of agent j, it is determined that there is a collision risk between them in the moving direction:

[0026]

[0027] Where μ is a heterogeneity distance parameter, which reflects the field radius considered by the potential risk, and the μ value of different types of traffic participants is different. Δ t is the heading angle deviation cosine value of agent i and agent j.

[0028] Preferably, the hybrid weight is updated in the super network by introducing the hybrid weight back propagation operator calculated by the softmax, and finally the final output feature is obtained:

[0029]

[0030]

[0031] In the formula, z out is the final representation of the node; represents the feature of the i-th intermediate layer; σ represents a nonlinear activation function; denotes the network weights shared by each node aggregator from the candidate architecture in the search space; denotes the input hidden features; θ n denotes the node aggregation operation; θ l denotes the inter-layer aggregation operation; θ s denotes the skip connection operation.

[0032] The application also provides a heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, which is used to implement the above method, comprising a scene construction module, a model construction module and a trajectory prediction module.

[0033] The scene construction module is used to model the heterogeneous scene graph in the multi-agent autonomous driving scene.

[0034] The model construction module is used to construct a multi-agent trajectory prediction model based on graph neural architecture search.

[0035] The trajectory prediction module is used to perform trajectory prediction in the heterogeneous scene graph using the multi-agent trajectory prediction model.

[0036] Compared with the prior art, the application has the following beneficial effects:

[0037] The application proposes a new trajectory prediction framework based on graph neural architecture search, which is composed of a flexible, customizable and efficient search space, a super network search strategy and a subnetwork contribution evaluation strategy. TrajGNAS can autonomously perform end-to-end search tasks to produce advanced trajectory prediction models with strong performance. The application also designs a new NAS loss function, which integrates road semantic layout, utility constraints, model computational complexity constraints and multi-modal prediction trajectory accuracy loss to optimize the balance between inference speed and model accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0039] Figure 1 The method flowchart of the embodiment of the present application;

[0040] Figure 2 The multi-agent trajectory prediction model schematic diagram of the embodiment of the present application;

[0041] Figure 3 The multi-agent position relationship schematic diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0043] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0044] Embodiment One

[0045] As shown in the method flowchart of the present embodiment, the steps include: Figure 1

[0046] S1. Model the heterogeneous scene graph under the multi-agent autonomous driving scene.

[0047] Before trajectory prediction, the heterogeneous scene graph G=(V, E) needs to be modeled, where the node V represents the heteroatom (for example, an agent, a lane or a parking sign), and the edge E represents the interaction between them.

[0048] S2. Build a multi-agent trajectory prediction model based on graph neural architecture search.

[0049] Generally, the GNN (graph neural network) model is unified by neighbor aggregation or message passing model, where the representation of each node is learned by iteratively aggregating the embeddings of its neighbors. Among them, the pre-defined aggregation function is a key component, which is different in different graph neural networks as a convolution operator. The node aggregation function and the inter-layer aggregation function are usually the focus of the artificial design of the graph neural network, and many graph neural network-based trajectory prediction models choose to stack multiple layers of the same aggregation function to aggregate adjacent features.

[0050] The present embodiment combines different aggregation functions to further improve the performance of the graph network. As shown in the figure, Figure 2 The multi-agent trajectory prediction model (TrajGNAS) constructed focuses on searching the aggregation function in the graph network. As shown in the figure, Figure 2 The multi-agent trajectory prediction model (TrajGNAS) constructed focuses on searching the aggregation function in the graph network.

[0051] ​The search space of TrajGNNAS is composed of many encapsulated convolutional operator modules, referred to as cells, which participate in the computation of each search layer; finally, each search layer determines a convolutional operator as its search result. Therefore, as a basic part of the search process, the types of operator modules involved in the search directly affect the final model structure. To optimize the search space, this embodiment refers to the classical trajectory prediction and graph network task solving method, and further introduces efficient convolutional operator types. In addition, considering that the residual mechanism can improve the performance of the graph neural network, residual connectivity is also used as part of the search process.

[0052] (1) Node message aggregation operation:

[0053] For node convolutional operator candidates, this embodiment selects five convolutional operators based on popular GNN models, namely GCN, GAT, ST-GCN, AI-GAT and EDGE, and a parameter-free operation, namely Non-GNN. GCN and GAT are powerful and widely used basic graph aggregation functions, which have excellent neighbor message aggregation capabilities and simple structures, which help to reduce their weights. Autonomous driving scenarios usually contain rich spatio-temporal information; therefore, ST-GCN is widely used in spatio-temporal graph convolutional network models for trajectory prediction. It dynamically captures the relationship between nodes at the time and space levels, which is crucial for long time series trajectory prediction tasks, while AI-GAT and EDGE focus on the heterogeneous elements of the scene and their interactions. AI-GAT uses an improved graph aggregation function to improve its graph structure. AI-GAT uses an enhanced graph attention module to construct a weight matrix for semantically different nodes. In contrast, EDGE overcomes the limitations of GAT, which can only use binary edge features, and GCN, which can only use one-dimensional edge features, and makes full use of the edge connection features between different types of nodes by using edge connection enhancement.

[0054] (2) Inter-layer aggregation and connection operation:

[0055] This embodiment selects three inter-layer aggregation functions as candidate functions: CONCAT, MAX and LSTM. In addition, two jump operator symbols, ZERO and Identity, are added to the search space. The ZERO operation prohibits message passing and helps to eliminate redundant aggregation operations in the architecture, while the Identity operation allows message passing without aggregating its neighbor information to the next layer. Such a design makes the overall search space more flexible.

[0056] S3. Trajectory prediction in a heterogeneous scene graph using a multi-agent trajectory prediction model.

[0057] Real-world traffic scenarios are usually dynamic and full of uncertainties. The uncertainty of cognitive prediction and lack of risk awareness are the main challenges involved in autonomous driving systems. Therefore, it is important to introduce a large amount of additional information when optimizing the architecture search and prediction results, especially in the case of insufficient training data or scenario diversification. Therefore, in order to consider the role of risk in model aggregation function search, the embodiment selects to model the collision risk as a graph structure. Specifically, the representation of the agent at time t is regarded as a node, and the node representation of the traffic participant is usually modeled through their physical characteristics, such as their position, speed, facing angle and size. Expressed in formula as:

[0058]

[0059] Where, is the final node representation at time t, LSTM() represents the LSTM network constructed by the full connection layer and the ReLU function. (a1, a2,..., a j ) are the physical properties of all agents, that is, their position, speed, facing angle and size at the current time step.

[0060] For the edge representation of the graph, the embodiment uses the collision risk as the edge feature. The distance, attributes, bias angle and speed between agents are all key factors in estimating the collision risk. Therefore, the collision risk between agents is quantified as:

[0061]

[0062] Where |T ij | represents the collision time, represents the motion direction risk, is the agent attribute risk. Specifically as follows:

[0063] The collision risk is inversely proportional to the collision time, expressed in formula as:

[0064]

[0065] In the formula, and represent the coordinates of agent i and agent j at time t, and α and β represent the angle between their forward direction and the horizontal axis, and γ represents the angle between the connecting line of the two agents and the horizontal axis, v i and v j represent the speed of agent i and j, as shown in Figure 3

[0066] The moving direction risk represents that if agent i appears in the field of view of agent j, it is considered that there is a collision risk between them in the moving direction. Expressed in formula as: ​

[0067]

[0068] where μ is a heterogeneous distance parameter that reflects the radius of the domain considered for potential risk, and μ values are different for different types of traffic participants. t is the cosine value of the heading angle deviation between agent i and agent j, the closer their moving angles are, the greater the risk of collision.

[0069] Agent attribute risk refers to the fact that different types of traffic participants (pedestrians, vehicles, cyclists) usually have different physical attributes (speed, mass, volume) and social attributes (vulnerable groups, whether protected by traffic rules, etc.), and their evaluation criteria for road risk will also differ. For this reason, the search strategy distinguishes between different semantic types of agents in the scene in terms of risk coefficients. A simple MLP structure is used to model the interaction between different agent attributes. Agent attribute risk can be represented as:

[0070]

[0071] where, and are node representations obtained through formula (1), which are used here to represent the physical attributes they contain. i and m j are the masses of agents i and j, respectively, c type is a variable representing the type of agent, representing the specific risk coefficient of different types of traffic participants.

[0072] For a multi-layer GNN, the search space is usually the aggregation operator of multiple nodes and the jump operation, and the discrete search space is formed by selecting appropriate operators on each edge. This embodiment relaxes the categorical selection of specific operations to a softmax over all possible operations:

[0073]

[0074] where the operator mixing weights of nodes i and j are represented by the parameter vector , represents the search space, which includes node message aggregation operations, inter-layer aggregation operations and connection operations; this embodiment uses three corresponding θ n , θ l , θ s to distinguish between different types of operation weights. These coefficients can distinguish the importance of different operators and will be involved in the training process of the super network. o(x) represents the input hidden features of the current layer of the GNN

[0075] By introducing the mixed weight of softmax calculation, the operator mixed weight can be back propagated And update it in the hypernetwork. Finally, the final output feature can be obtained, which is expressed by the formula:

[0076]

[0077]

[0078] where z out is the final representation of the node; represents the feature of the i-th intermediate layer; σ represents a nonlinear activation function; represents the network weight shared by each node aggregator from the candidate architecture in the search space; represents the input hidden feature; θ n represents the node aggregation operation; θ l represents the inter-layer aggregation operation; θ s represents the skip connection operation.

[0079] After obtaining the final representation z out of the node, different losses or guidance mechanisms are used according to the needs of the task.

[0080] The task of TrajGNAS architecture search is to learn continuous variables θ n , θ l , θ s and network weights w. The architecture parameters θ are optimized on the validation set, and the network weights w are optimized on the training set. It is worth noting that when optimizing the architecture parameters and network weights, the embodiment not only searches for the heterogeneous graph of the agent, but also constructs the completed risk interaction perception graph as the search object, and uses a pre-designed loss function with task adaptability. In this way, the search for the best architecture of the network is guided by combining the agent historical trajectory, agent interaction mechanism and collision risk on the hypernetwork feature level.

[0081] In the fine-tuning part, the embodiment also optimizes the hyperparameters of the risk interaction perception graph. The gradient descent method is used to update the parameters:

[0082]

[0083] where w is the weight of the current risk interaction perception graph, ξ is the internal optimization learning rate of w, and the embodiment only needs to use a single training step to adjust w to approximate w * (θ).

[0084] Embodiment two

[0085] For trajectory prediction models in real driving scenarios, in addition to achieving good accuracy, their inference speed needs to be considered to respond to unexpected situations and ensure safety and reliability. In addition to accuracy and speed, in order to make the risk perception guidance mechanism of the present application fully play a role in search and reflect the constraints of physical and social behavior on the trajectory of the agent, and enhance explainability, the embodiment designs a loss function under posterior knowledge and social constraints as an important indicator for evaluating the search network model.

[0086] (1) Semantic region loss:

[0087] The behavior of traffic participants in the real world is usually constrained by traffic rules and social ethics, for example, on the road, vehicles should drive on the motorway, pedestrians should advance on the sidewalk or pedestrian crossing, and bicycles should drive on the non-motorway. Therefore, the trajectory predicted by the model is taken as posterior knowledge, and the rationality of the agent in exercising road rights and whether it conforms to the constraints of traffic rules are verified. For the prediction point If it is within the constraint region, the region loss of the point is defined as 0, otherwise, the normalized distance of the point to the constraint region is calculated, and the minimum value is taken as the region loss of the point. The final region loss is the normalized distance square accumulation sum of all time steps and objects, and their calculation formula is as follows:

[0088]

[0089] where d t，n is the distance from the prediction point to the nearest boundary of the region, which is for traffic participants whose predicted trajectories are not in their corresponding road semantics. The normalization factor is the maximum radius d max of the region. ε is a self-defined coefficient. This loss encourages the model to fully consider road rules and social constraints when predicting different agent trajectories, ensuring that the predicted trajectory is as close as possible to the corresponding region, and the farther the distance from the boundary, the greater the penalty.

[0090] (2) Precision and complexity joint loss:

[0091] The embodiment also considers the balance between model calculation complexity and accuracy, and designs a multi-index loss function, which includes an accuracy loss function and a consideration of network sparsity, inference speed, parameter quantity and calculation amount. The evaluation of the prediction accuracy of the model is realized by the negative log-likelihood loss (NLL-loss) of the bivariate normal distribution:

[0092]

[0093] where σ x and σ yis the standard deviation in x, y direction, p is the correlation coefficient between x, y, and Δx, Δy represent the deviation between predicted value and true value.

[0094] For the constraint of model complexity, the embodiment comprehensively considers multiple neural network computing properties, and guides the search direction of TrajGNAS by backpropagating the loss value in the super network training process. Firstly, the computational loss and the sparsity loss are defined. The computational loss reflects the overall computational complexity of the model, and focuses on the floating point number and the parameter quantity. In addition, in actual deployment, the trajectory prediction algorithm is often limited by the computing power of the hardware device, so the FLops of the model should be limited. In the embodiment, a new limit coefficient is provided for the floating point loss, so that the Flops of the model in the training process is more severely punished when it exceeds the hardware bearing range. The formula is:

[0095]

[0096] In the formula, is the set of all modules of the model, W conv is the convolution kernel weight tensor, K h and K w is the convolution kernel height and width, C out is the output channel number, W fc is the fully connected layer weight tensor, represents the total parameter quantity, λ f is the weight parameter of the computational loss. is the penalty coefficient, used to adjust the value of the computational burden loss. It can be represented as:

[0097]

[0098] Where T is the expected Flops limit set, and τ is a value set in actual deployment, which can be adjusted according to the situation, usually much larger than 1.

[0099] In addition, enhancing the sparsity of the neural network is of great significance to improve the computing efficiency and enhance the generalization performance of the model. The loss of network sparsity is defined as:

[0100]

[0101] Where P is the set of all parameters of the model, ||P i is the L1 norm of the parameter, λ s is the sparsity loss weight.

[0102] The final loss of the embodiment is a weighted average of all the above losses. By introducing a multi-index loss function based on posterior knowledge and social constraints, the trajectory prediction model can achieve an optimal balance between accuracy and speed, while also being interpretable. The total loss function is represented as:

[0103]

[0104] wherein is the definable weight of each loss, and the user can adjust the value of the weight according to the need to achieve a more flexible network architecture search.

[0105] Embodiment three

[0106] To verify the advancement of the present application compared to the prior art, the following experiments are set up for comparison.

[0107] Experiment one

[0108] Experiments were conducted on three large-scale autonomous driving datasets, nuScenes, Agroverse, and ApolloScape, and the proposed method was compared with classic and state-of-the-art trajectory prediction models. The experimental results on ApolloScape are shown in Table 1, and the TrajGANS framework is superior to other candidate methods in terms of wADE and wFDE indicators, with an increase of 3.8% and 3.2% respectively compared to the second candidate. Table 2 shows the experimental results on the Agroverse dataset, which shows that the proposed method ranks third in mADE metrics, but is the most advanced method in mFDE metrics. The experimental results on nuScenes are shown in Table 3, and when K = 5 and K = 10, the model achieves the best performance in MR evaluation indicators, with an average increase of 31.6% and 50% respectively compared to DMAP (the second method). This result shows that the model of the present application can more accurately predict the final position of the agent. In addition, the present application also evaluates the inference time of TrajGNAS, as shown in Table 4, which is significantly better than other methods in inference speed. Therefore, in the face of more complex scenarios and larger data volumes, the simple and unified network structure searched by TrajGNAS is crucial for performing fast real-time analysis and processing tasks and maximizing safety. Therefore, TrajGNAS is very suitable for the autonomous driving industry.

[0109] Table 1

[0110] Method Year WSADE ADEv ADEp ADEb WSFDE FDEv FDEp FDEb TPNet 2020 1.280 2.210 0.740 1.850 2.340 3.860 1.410 3.400 Grip++ 2020 1.259 2.240 0.714 1.802 2.363 4.076 1.373 3.416 MCCITP 2021 1.200 2.090 0.700 1.720 2.240 3.770 1.340 3.210 S2TNet 2021 1.168 1.987 0.683 1.700 2.180 3.578 1.305 3.215 AI-TP 2022 1.160 - - - 2.130 - - - D2-Tpred 2022 1.020 - - - 1.690 - - - HRG-HSG 2023 1.036 - - - 1.321 - - - STF 2023 1.384 2.403 0.799 2.001 1.707 2.972 1.012 2.392 MDSTF 2024 1.117 1.868 0.674 1.601 2.044 3.333 1.272 2.907 TP-EGT 2024 1.190 2.050 0.700 1.720 2.140 3.530 1.280 3.160 TrajGNAS (Invention) - 0.982 1.258 0.701 1.470 1.291 1.905 0.802 2.022

[0111] Table 2

[0112] Method Year mADE mFDE NLNI 2021 0.79 1.25 THOMAS 2021 0.94 1.44 GOHOME 2022 0.89 1.29 DenseTNT 2021 0.88 1.28 LaneGCN 2020 0.87 1.36 LTP 2022 0.83 1.29 FRM 2023 0.82 1.27 HiVT 2022 0.77 1.17 ProphNet 2023 0.77 1.14 LAformer 2024 0.77 1.16 TrajGNAS (Invention) - 0.82 1.10

[0113] Table 3

[0114]

[0115] Table 4

[0116] Model Batch size Inference time (ms) HRG-HSG 8 43.75 DenseTNT 16 88.5 HiVT 16 6.3 PGP 12 17.9 TrajGNAS 8 35.65 TrajGNAS-loss 12 17.5

[0117] Experiment two

[0118] To evaluate the effectiveness of the method proposed in the present application, various variants of the model are evaluated in Table 5. Among them, (1) only the graph neural architecture search is used; (2) the risk interaction guiding mechanism is added; (3) the regional semantic constraint loss is introduced; (4) the computational complexity constraint is introduced. The results show that each module designed in the present application plays its own role. The risk interaction mechanism and the regional semantic constraint loss proposed in the present application significantly improve the accuracy of the model. In terms of precision indicators, the introduction of the risk interaction mechanism significantly improves the mADE, mFDE and MR indicators of the model. In addition, by introducing the computational loss function, TrajGNAS effectively guides the network search process, reducing the number of parameters and the amount of calculation of the final search model; however, this configuration has no significant effect on the accuracy. The mADE and mFDE are only increased by 7% and 10.6%, respectively, and the MR performance indicator does not decrease. It is worth noting that since model 1 only uses GNAS, the number of parameters of model 1 is less and the computational burden is lower than that of models 2 and 3; however, under the constraint of computational loss, the number of parameters of structure 4 is still less than that of structure 1, and the mADE and mFDE indicators of the model prediction are increased by 53% and 44%, respectively.

[0119] Table 5

[0120] Model (ID) GNAS RI Region-Loss Compute-Loss mADE mFDE MR Params FLOPS (MB) 1 √ 1.532 1.911 0.53 1089K 8.961 2 √ √ 1.103 1.518 0.29 1315K 9.346 3 √ √ √ 0.990 1.312 0.09 1364K 9.764 4 √ √ √ √ 0.997 1.326 0.09 1031K 8.716

[0121] Example four

[0122] The present embodiment also provides a heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, comprising a scene construction module, a model construction module and a trajectory prediction module. The scene construction module is used to model the heterogeneous scene graph in the multi-agent autonomous driving scene; the model construction module is used to construct a multi-agent trajectory prediction model based on graph neural architecture search; and the trajectory prediction module is used to perform trajectory prediction in the heterogeneous scene graph by using the multi-agent trajectory prediction model.

[0123] In the following, the present application will be described in detail in combination with the present embodiment, how the present application solves the technical problems in practical work.

[0124] First, the scene construction module is used to model the heterogeneous scene graph in the multi-agent autonomous driving scene.

[0125] Before trajectory prediction, the heterogeneous scene graph G=(V, E) needs to be modeled, where the nodes V represent the heterostructures (e.g., agents, lanes, or parking signs), and the edges E represent the interactions between them.

[0126] Then the model construction module constructs a multi-agent trajectory prediction model based on graph neural architecture search.

[0127] Generally, the GNN (graph neural network) model is unified by a neighbor aggregation or message passing model, in which the representation of each node is learned by iteratively aggregating the embeddings of its neighbors. Among them, the pre-defined aggregation function is a key component, which acts as a convolution operator and is different in different graph neural networks. The node aggregation function and the inter-layer aggregation function are usually the focus of the artificial design of the graph neural network, and many graph neural network-based trajectory prediction models choose to stack multiple layers of the same aggregation function to aggregate adjacent features.

[0128] The embodiment combines different aggregation functions to further improve the performance of the graph network. As shown in Figure 2 , the constructed multi-agent trajectory prediction model (TrajGNAS) focuses on searching the aggregation function in the graph network. As shown in Figure 2 , TrajGNAS focuses on searching the aggregation function in the graph network.

[0129] The search space of TrajGNAS is composed of many encapsulated convolution operator modules, called units, which participate in the calculation of each search layer; finally, each search layer determines a convolution operator as its search result. Therefore, as the basic part of the search process, the type of operator module involved in the search directly affects the final model structure. In order to optimize the search space, the embodiment refers to the classical trajectory prediction and graph network task solving method, and further introduces efficient convolution operator types. In addition, considering that the residual mechanism can improve the performance of the graph neural network, the residual connectivity is also used as part of the search process.

[0130] (1) Node message aggregation operation:

[0131] For node convolutional operator candidates, the embodiment selects 5 kinds of convolutional operators based on popular GNN models, namely GCN, GAT, ST-GCN, AI-GAT and EDGE, and a parameter-free operation, namely Non-GNN. GCN and GAT are powerful and widely used basic graph aggregation functions, which have excellent neighbor message aggregation capability and simple structure, which helps to reduce their weights. Autonomous driving scenarios usually contain rich spatio-temporal information; therefore, ST-GCN is widely used in spatio-temporal graph convolutional network models for trajectory prediction. It dynamically captures the relationship between nodes at the time and space levels, which is crucial for long time series trajectory prediction tasks, while AI-GAT and EDGE focus on the heterogeneous elements of the scene and their interactions. AI-GAT uses an improved graph aggregation function to improve its graph structure. AI-GAT uses an enhanced graph attention module to construct a weight matrix for semantically different nodes. In contrast, EDGE overcomes the limitations of GAT, which can only use binary edge features, and GCN, which can only use one-dimensional edge features, and makes full use of edge connection features between different types of nodes by using edge connection enhancement.

[0132] (2) Inter-layer aggregation and connection operations:

[0133] The embodiment selects three inter-layer aggregation functions as candidate functions: CONCAT, MAX and LSTM. In addition, two jump operator symbols, ZERO and Identity, are added to the search space. The ZERO operation prohibits message passing and helps to eliminate redundant aggregation operations in the architecture, while the Identity operation allows message passing without aggregating its neighbor information to the next layer. Such a design makes the overall search space more flexible.

[0134] Finally, the path prediction module uses a multi-agent trajectory prediction model to predict trajectories in a heterogeneous scene graph.

[0135] Real-world traffic scenarios are usually dynamic and full of uncertainties. The uncertainty of cognitive prediction and the lack of risk awareness are major challenges involving autonomous driving systems. Therefore, it is important to introduce a large amount of additional information when optimizing the architecture search and prediction results, especially in the case of insufficient training data or scene diversification. Therefore, in order to consider the role of risk in model aggregation function search, the embodiment selects to model the collision risk as a graph structure. Specifically, the representation of the agent at time t is regarded as a node, and the node representation of the traffic participant is usually modeled through their physical characteristics, such as their position, velocity, facing angle and size. It can be expressed in the formula as:

[0136]

[0137] wherein, LSTM() represents the LSTM network built by fully connected layers and ReLU functions for the final node representation at time t. (a1, a2,..., a j) are the physical properties of all agents, i.e., their position, velocity, facing angle, and size at the current time step.

[0138] For the edge representation of the graph, this embodiment uses the collision risk as the edge feature. The distance, properties, bias angle, and velocity between agents are all key factors to estimate the collision risk. Therefore, the collision risk between agents is quantified as:

[0139]

[0140] where |T ij | represents the collision time, represents the moving direction risk, is the agent property risk. Specifically, as follows:

[0141] The collision risk is inversely proportional to the collision time, which is expressed as:

[0142]

[0143] In the formula, and represent the coordinates of agent i and agent j at time t, and a and b represent the angles between their forward directions and the horizontal axis, and g represents the angle between the connecting line of the two agents and the horizontal axis, v i and v j represent the velocities of agents i and j, as shown in Figure 3

[0144] The moving direction risk represents that if agent i appears in the field of view of agent j, it is considered that there is a collision risk between them in the moving direction. The formula is expressed as:

[0145]

[0146] where μ is a heterogeneous distance parameter that reflects the field radius considered by the potential risk, and the value of μ is different for different types of traffic participants. Δ t is the heading angle deviation cosine value of agent i and agent j, and the closer their moving angles are, the greater the risk of collision is.

[0147] ​Agent attribute risk refers to the fact that different types of traffic participants (pedestrians, vehicles, cyclists) usually have different physical attributes (speed, mass, volume) and social attributes (vulnerable groups, whether protected by traffic rules, etc.), and their evaluation criteria for road risk will also be different. In view of this, the search strategy distinguishes the risk coefficients of different semantic types of agents in the scene. A simple MLP structure is used to model the interaction between different agent attributes. The agent attribute risk can be represented as:

[0148]

[0149] where, and are the node representations obtained by formula (17), which are used here to represent the physical attributes they contain. m i and m j are the masses of agent i and agent j, c type is a variable representing the type of agent, representing the specific risk coefficient of different types of traffic participants.

[0150] For a multi-layer GNN, the search space is usually the aggregation operator of multiple nodes and the jump operation, and the discrete search space is formed by selecting the appropriate operator on each edge. The embodiment relaxes the classification selection of specific operations to softmax of all possible operations:

[0151]

[0152] where the operator mixing weight of node i and j is represented by the parameter vector , represents the search space, which includes node message aggregation operations, inter-layer aggregation operations and connection operations; the embodiment adopts three corresponding θ n , θ l , θ s to distinguish different types of operation weights. These coefficients can distinguish the importance of different operators and will participate in the training process of the super network. o(x) represents the input hidden features of the current layer of the GNN

[0153] By introducing the mixed weight calculated by softmax, the operator mixing weight can be backpropagated and updated in the super network, and finally the final output feature can be obtained, which is represented by the formula:

[0154]

[0155]

[0156] where zout is the final representation of the node; is the feature of the i-th intermediate layer; σ is a nonlinear activation function; is the network weight shared by each node aggregator from the candidate architecture in the search space; is the input hidden feature; θ n is the node aggregation operation; θ l is the inter-layer aggregation operation; θ s is the skip connection operation.

[0157] After obtaining the final representation of the node z out , different losses or guidance mechanisms are used according to the needs of the task.

[0158] The task of the TrajGNA architecture search is to learn the continuous variable θ n , θ l , θ s and the network weight w. The architecture parameter θ is optimized on the validation set, and the network weight w is optimized on the training set. It is worth noting that when optimizing the architecture parameter and the network weight, the embodiment not only searches for the heterogeneous graph of the agent, but also constructs the completed risk interaction perception graph as the search object, and uses a pre-designed loss function with task adaptability. In this way, the search for the best architecture of the network is guided by combining the agent historical trajectory, agent interaction mechanism and collision risk on the super network feature level.

[0159] In the fine-tuning part, the embodiment also performs hyperparameter optimization on the risk interaction perception graph. The gradient descent method is used to update the parameters:

[0160]

[0161] where w is the weight of the current risk interaction perception graph, ξ is the internal optimization learning rate of w, and the embodiment only needs to use a single training step to adjust w to approximate w * (θ).

[0162] The above-described embodiments only describe the preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those of ordinary skill in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A heterogeneous multi-agent trajectory prediction method based on graph neural architecture search, characterized by the following steps: include: Modeling heterogeneous scene graphs in multi-agent autonomous driving scenarios; Construct a multi-agent trajectory prediction model based on graph neural architecture search; The multi-agent trajectory prediction model is used to predict trajectories in the heterogeneous scene graph. The method for performing the trajectory prediction includes: modeling the collision risk as a graph structure, and treating the agent's representation at time t as nodes: in, Let be the final node representation at time t, and LSTM() represent an LSTM network constructed from fully connected layers and the ReLU function; These are the physical attributes of all intelligent agents, including position, velocity, facing angle, and size at the current time step; Using collision risk as an edge feature: in, Indicates the collision time. Indicates the risk of movement direction. Risks associated with agent attributes.

2. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1, characterized in that, The constructed heterogeneous scene graph modules: G = (V,E) In this context, node V represents an isomer; edge E represents the interaction between isomers.

3. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1, characterized in that, The constructed multi-agent trajectory prediction model consists of many encapsulated convolution operator modules, which participate in the computation of each search layer; each search layer determines a convolution operator as its search result.

4. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 1, characterized in that, Collision risk is inversely proportional to collision time, as expressed by the formula: In the formula and Let represent the coordinates of agents i and j at time t, α and β represent the angles between their directions of movement and the horizontal axis, and γ represent the angle between the line connecting the two agents and the horizontal axis. and Indicate the speeds of agents i and j; If agent i appears within the field of view of agent j, then it is determined that there is a risk of collision between them in the direction of movement: Here, μ is a heterogeneous distance parameter that reflects the radius of the area considered for potential risks, and the value of μ varies for different types of traffic participants; Let be the cosine of the heading angle deviation between agents i and j.

5. The heterogeneous multi-agent trajectory prediction method based on graph neural architecture search according to claim 4, characterized in that, By introducing a backpropagation operator to mix the weights calculated using softmax and updating them in the supernet, the final output features are obtained: In the formula, This is the final representation of the node; This represents the feature of the i-th intermediate layer; Represents a non-linear activation function; This represents the network weights shared by each node aggregator from candidate architectures in the search space; Indicate the input hidden features; This indicates a node aggregation operation; This indicates an inter-layer aggregation operation; This indicates a jump connection operation.

6. A heterogeneous multi-agent trajectory prediction system based on graph neural architecture search, the system being used to implement the method according to any one of claims 1-5, characterized in that, include: The module consists of a scene building module, a model building module, and a trajectory prediction module.

Images