A method and system for predicting target behavior based on heterogeneous graph attention networks

Abstract

This invention relates to the field of intelligent driving technology and discloses a target behavior prediction method and system based on heterogeneous graph attention networks. The method involves constructing and training a behavior prediction model, including: collecting real-time motion features of targets in traffic scenes, generating trajectory time-series samples, and extracting time-dependent features of target behavior using a temporal prediction network model based on long short-term memory networks; constructing a heterogeneous graph attention network model; inputting the heterogeneous graph of the traffic scene into the graph attention network to extract spatial interaction features; fusing the time-dependent features and spatial interaction features, processing the fused features through a fully connected layer to predict the future behavior of the target, and optimizing the behavior prediction model using a joint loss function. This invention comprehensively considers temporal and spatial factors, accurately predicts the behavior of targets outside the abnormal domain, and can significantly improve the accuracy and robustness of target behavior prediction.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a method and system for predicting target behavior based on heterogeneous graph attention networks. Background Technology

[0002] In the fields of intelligent transportation and autonomous driving technology, target behavior prediction is one of the key technologies for achieving safe and efficient driving. With the continuous development of autonomous driving systems, higher demands are placed on the vehicle's ability to predict target behavior in complex traffic environments, especially in predicting the behavior of out-of-domain targets. Out-of-domain targets refer to targets that suddenly appear around the vehicle and cannot be identified through conventional patterns; their behavior is highly uncertain, posing challenges to the decision-making and control of autonomous driving systems.

[0003] Traditional methods for predicting target behavior often rely on time series models or simple spatial modeling. However, these methods often struggle to simultaneously capture the temporal dependencies and spatial interactions of target behavior. With the rise of graph neural networks and attention mechanisms, researchers have gradually realized that single temporal or spatial modeling capabilities are insufficient to meet the needs of complex traffic environments. While models such as Long Short-Term Memory (LSTM) networks can capture temporal dependencies well, they are inadequate in handling complex spatial relationships among multiple traffic participants; graph neural networks, on the other hand, can effectively model spatial dependencies, but they have certain limitations in capturing dynamic temporal changes. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a target behavior prediction method and system based on heterogeneous graph attention networks. By comprehensively considering temporal and spatial factors, it accurately predicts the behavior of targets outside the abnormal domain. By fusing time series and spatial interaction information, it can significantly improve the accuracy and robustness of target behavior prediction, thereby providing more reliable decision support for autonomous driving systems, especially for target behavior prediction and decision optimization in complex traffic environments.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a target behavior prediction method based on heterogeneous graph attention networks, wherein the construction and training process of the behavior prediction model includes:

[0007] Real-time motion features of targets in traffic scenarios are collected to generate trajectory time series samples. Temporal prediction network models based on long short-term memory networks are used to extract the time-dependent features of target behavior.

[0008] A heterogeneous graph attention network model is constructed, which includes a traffic scene heterogeneous graph and a graph attention network. Nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features that cover the dynamic and static information of traffic participants. Edges represent the interaction relationships between traffic participants and have edge features that cover the changes in distance, relative speed, and relative angle between traffic participants. The traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features.

[0009] The model integrates time-dependent features with spatial interaction features, processes the integrated features through a fully connected layer to predict the future behavior of the target, and optimizes the behavior prediction model using a joint loss function.

[0010] In one embodiment, the step of collecting real-time motion features of targets in a traffic scene and generating trajectory time series samples includes:

[0011] Collect the target's real-time motion characteristics, including position, velocity, acceleration, and heading angle;

[0012] Based on the real-time motion characteristics of the target, combined with map and traffic scene data, an incremental structured optical flow algorithm is used to parse multi-frame image data. The generated sequence of the target's position coordinates at multiple consecutive times is taken as a spatial trajectory. The spatial trajectory is arranged in chronological order to form a trajectory time series, while the semantic information of the surrounding environment is labeled.

[0013] The trajectory time series is preprocessed, and the sliding window technique is used to divide the preprocessed trajectory time series into segments of fixed length to generate trajectory time series samples.

[0014] In one embodiment, the extraction of temporal dependency features of target behavior using a temporal prediction network model based on a long short-term memory network specifically includes:

[0015] The temporal prediction network model includes an input layer, multiple long short-term memory network units, a dropout layer, and a fully connected output layer. The long short-term memory network units use a gating mechanism and residual connections to extract the temporal series features of the trajectory time series samples as the temporal dependence features of the target behavior.

[0016] The time series prediction network model is trained based on trajectory time series samples and mean squared error loss (MSE). The mean squared error loss (MSE) is:

[0017] ;

[0018] This represents the total number of trajectory time series samples. This represents the true value of the i-th trajectory time series sample. This represents the predicted value of the i-th trajectory time series sample.

[0019] In one embodiment, the dynamic information includes the position, speed, and acceleration of traffic participants, and the static information includes the type of traffic participants and lane information; the traffic participants include targets as the object of prediction, as well as vehicles and pedestrians that interact with the targets;

[0020] The step of inputting the heterogeneous traffic scene graph into a graph attention network to extract spatial interaction features specifically includes:

[0021] A graph attention network is used to calculate the attention weights between nodes through a self-attention mechanism, and to aggregate neighbor node information to update node features; the final node features obtained after processing by a multi-layer graph attention network are the spatial interaction features.

[0022] Mean squared error is used as the loss function Training the graph attention network:

[0023] ;

[0024] Where m represents the prediction time step, This represents the predicted position of the traffic participant at time step j; This represents the actual location of the traffic participant at time step j. The node and edge features in the graph attention network are optimized using the backpropagation algorithm, and the attention weights are adjusted to minimize the error between the predicted and actual locations of the traffic participants.

[0025] In one embodiment, the fusion of temporal dependency features and spatial interaction features, the processing of the fused features through a fully connected layer to predict the future behavior of the target, and the optimization of the behavior prediction model using a joint loss function specifically include:

[0026] The future behavior of a target includes its position, velocity, and acceleration at future moments;

[0027] The joint loss function for:

[0028] ;

[0029] in, This represents the prediction error of the temporal dependency features of the time series prediction network. This represents the spatial interaction feature prediction error of the graph attention network. The fusion prediction error represents the fusion of time-dependent features and spatial interaction features, where α, β, and γ are weighting coefficients, and α+β+γ=1.

[0030] Secondly, this invention provides a target behavior prediction system based on heterogeneous graph attention networks. The construction and training of the behavior prediction model are achieved through the following modules:

[0031] The time feature extraction module collects real-time motion features of targets in traffic scenarios, generates trajectory time series samples, and uses a time-series prediction network model based on long short-term memory network to extract the time-dependent features of target behavior.

[0032] The spatial feature extraction module constructs a heterogeneous graph attention network model, including a traffic scene heterogeneous graph and a graph attention network. Nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features that cover the dynamic and static information of traffic participants. Edges represent the interaction relationships between traffic participants and have edge features that cover the changes in distance, relative speed, and relative angle between traffic participants. The traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features.

[0033] The fusion training module integrates temporal-dependent features with spatial interaction features, processes the fused features through a fully connected layer to predict the target's future behavior, and optimizes the behavior prediction model using a joint loss function.

[0034] In one embodiment, the step of collecting real-time motion features of targets in a traffic scene and generating trajectory time series samples includes:

[0035] Collect the target's real-time motion characteristics, including position, velocity, acceleration, and heading angle;

[0036] Based on the real-time motion characteristics of the target, combined with map and traffic scene data, an incremental structured optical flow algorithm is used to parse multi-frame image data. The generated sequence of the target's position coordinates at multiple consecutive times is taken as a spatial trajectory. The spatial trajectory is arranged in chronological order to form a trajectory time series, while the semantic information of the surrounding environment is labeled.

[0037] The trajectory time series is preprocessed, and the sliding window technique is used to divide the preprocessed trajectory time series into segments of fixed length to generate trajectory time series samples.

[0038] In one embodiment, the extraction of temporal dependency features of target behavior using a temporal prediction network model based on a long short-term memory network specifically includes:

[0039] The temporal prediction network model includes an input layer, multiple long short-term memory network units, a dropout layer, and a fully connected output layer. The long short-term memory network units use a gating mechanism and residual connections to extract the temporal series features of the trajectory time series samples as the temporal dependence features of the target behavior.

[0040] The time series prediction network model is trained based on trajectory time series samples and mean squared error loss (MSE). The mean squared error loss (MSE) is:

[0041] ;

[0042] This represents the total number of trajectory time series samples. This represents the true value of the i-th trajectory time series sample. This represents the predicted value of the i-th trajectory time series sample.

[0043] In one embodiment, the dynamic information includes the position, speed, and acceleration of traffic participants, and the static information includes the type of traffic participants and lane information; the traffic participants include targets as the object of prediction, as well as vehicles and pedestrians that interact with the targets;

[0044] The step of inputting the heterogeneous traffic scene graph into a graph attention network to extract spatial interaction features specifically includes:

[0045] A graph attention network is used to calculate the attention weights between nodes through a self-attention mechanism, and to aggregate neighbor node information to update node features; the final node features obtained after processing by a multi-layer graph attention network are the spatial interaction features.

[0046] Mean squared error is used as the loss function Training the graph attention network:

[0047] ;

[0048] Where m represents the prediction time step, This represents the predicted position of the traffic participant at time step j; This represents the actual location of the traffic participant at time step j. The node and edge features in the graph attention network are optimized using the backpropagation algorithm, and the attention weights are adjusted to minimize the error between the predicted and actual locations of the traffic participants.

[0049] In one embodiment, the fusion of temporal dependency features and spatial interaction features, the processing of the fused features through a fully connected layer to predict the future behavior of the target, and the optimization of the behavior prediction model using a joint loss function specifically include:

[0050] The future behavior of a target includes its position, velocity, and acceleration at future moments;

[0051] The joint loss function for:

[0052] ;

[0053] in, This represents the prediction error of the temporal dependency features of the time series prediction network. This represents the spatial interaction feature prediction error of the graph attention network. This represents the prediction error of the fused features, where α, β, and γ are weighting coefficients, and α+β+γ=1.

[0054] Compared with the prior art, the beneficial technical effects of the present invention are:

[0055] 1. Improved accuracy of target behavior prediction: This invention combines a temporal prediction network model based on Long Short-Term Memory (LSTM) with a Heterogeneous Graph Attention Network (HGAT) model, fully integrating temporal and spatial information to more accurately predict the behavior of traffic participants. The temporal prediction network model captures the temporal dependencies of target behavior, while the HGAT model analyzes the complex spatial interactions between traffic participants, thereby improving prediction accuracy, especially in complex traffic scenarios.

[0056] 2. Enhanced Model Recognition of Abnormal Behaviors: By introducing the HGAT model, this invention can effectively capture abnormal interaction patterns among traffic participants, especially abnormal out-of-domain target behaviors (such as sudden stopping, lateral movement, etc.). This spatial interaction modeling capability significantly improves the model's ability to predict irregular and sudden behaviors, thereby enhancing the adaptability of the autonomous driving system to abnormal traffic scenarios.

[0057] 3. Optimized use of computing resources: This invention fully considers computational efficiency in model design, optimizing the use of computing resources by integrating the advantages of LSTM and HGAT. Compared to traditional single models, ensemble models can better balance prediction accuracy and computational consumption, especially in real-time demanding autonomous driving scenarios, ensuring low latency and efficient processing.

[0058] 4. Enhanced Model Generalization Ability: This invention employs a heterogeneous graph attention network to model various traffic participants and their interactions. This structure endows the model with stronger generalization ability, enabling it to adapt to different traffic environments, including urban roads, highways, and complex intersection scenarios. This feature enhances the system's stability and adaptability in diverse scenarios.

[0059] 5. Enhanced Robustness of Abnormal Behavior Prediction: By combining time-series data and spatial relationships among traffic participants, this invention maintains high prediction accuracy and robustness under various environmental conditions. Whether facing high-density traffic or sudden events, the integrated model can effectively identify and predict target behavior, reducing error rates and improving the safety of autonomous driving systems.

[0060] 6. Reduced Data Acquisition Costs and Risks: Compared to traditional methods, this invention generates synthetic data by simulating traffic scenarios, avoiding the high costs and risks of on-site data collection in complex traffic environments, especially under highly uncertain or dangerous traffic conditions. Data generated through simulation not only fully simulates various traffic situations but also reduces environmental interference and limitations in data acquisition.

[0061] 7. Provides diverse training datasets: The model of this invention can generate datasets containing various traffic participants, abnormal behaviors, and complex interactions, providing richer and more diverse samples for training autonomous driving systems. This diverse dataset improves the system's adaptability in complex and dynamic traffic environments, enabling it to better cope with various scenarios and enhancing the system's robustness and safety.

[0062] 8. Driving the continuous advancement of autonomous driving technology: This invention not only improves prediction accuracy and real-time performance, but also promotes the development of autonomous driving technology through an innovative model architecture. Its technical solution provides new ideas for multi-objective behavior prediction and traffic interaction modeling, offering strong support for the reliable application of autonomous driving technology in complex scenarios.

[0063] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0065] Figure 2 This is a framework diagram of the time-series prediction network model in an embodiment of the present invention.

[0066] Figure 3 This is a framework diagram of the heterogeneous graph attention network model in an embodiment of the present invention. Detailed Implementation

[0067] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0068] like Figure 1 As shown, the target behavior prediction method based on heterogeneous graph attention network in this invention includes the following steps in the construction and training process of the behavior prediction model:

[0069] S1: Collect real-time motion features of targets in traffic scenarios, generate trajectory time series samples, and use a time-series prediction network model based on long short-term memory network to extract the time-dependent features of target behavior.

[0070] S2, Construct a heterogeneous graph attention network model, including a traffic scene heterogeneous graph and a graph attention network; Nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features covering the dynamic and static information of traffic participants, and edges represent the interaction relationships between traffic participants and have edge features covering the changes in distance, relative speed and relative angle between traffic participants; Input the traffic scene heterogeneous graph into the graph attention network to extract spatial interaction features.

[0071] S3 fuses temporal dependence features with spatial interaction features, processes the fused features through a fully connected layer to predict the future behavior of the target, and optimizes the behavior prediction model using a joint loss function.

[0072] In one embodiment, step S1 specifically includes the following steps:

[0073] Step S101: Use vehicle-mounted cameras and lidar to collect real-time motion characteristics of the target, including information such as position, speed, acceleration and heading angle, and record the data in time series format.

[0074] Step S102: Combining high-precision maps and traffic scene data, the incremental structured optical flow (SFM) algorithm is used to analyze the multi-frame image data collected by the vehicle camera. The generated sequence of the target's position coordinates at multiple consecutive times is used as a spatial trajectory. The spatial trajectory is arranged in chronological order to form a trajectory time series, while the semantic information of the surrounding environment is labeled.

[0075] First, the incremental structured optical flow algorithm processes consecutive frames of images and uses optical flow to estimate the temporal displacement of an object. The algorithm first extracts feature points from each frame and calculates the displacement of these feature points between different time frames. Then, it uses the optical flow equation (based on the assumption of image brightness invariance) to calculate the velocity field of each pixel, thereby estimating the object's trajectory. The basic formula of the optical flow equation is:

[0076] ;

[0077] in, It is the gradient of the image in the x-direction. It is the gradient of the image in the y-direction. It is the gradient of the image over time. and These are the optical flow components in the x and y directions, representing the displacement velocity of an object. To improve estimation accuracy, incremental structured optical flow algorithms typically employ a multi-scale approach, estimating optical flow at different scales to handle motion at varying scales in complex scenes. The estimation of the optical flow field at each scale depends on the estimation results of the previous scale.

[0078] Next, the incremental structured optical flow algorithm continuously updates the displacement information in the image sequence to progressively optimize the optical flow field and reduce errors. To ensure computational stability, the algorithm employs an iterative update strategy, gradually refining the displacement estimation to obtain a more accurate motion field. The incremental algorithm updates the optical flow field as follows:

[0079] ;

[0080] in, It is the optical flow field estimated at time k. The updated value is obtained through incremental estimation. To optimize the accuracy of the optical flow field, the algorithm typically uses weighted least squares to solve for the optical flow field.

[0081] This process takes into account the local neighborhood information of pixels, and the optimization process is described by the following least squares objective function:

[0082] ;

[0083] in, It is a pixel. Weights are assigned to reduce the impact of noise. By introducing weighted averaging and constraints, the incremental algorithm effectively reduces noise and improves the tracking ability for fast motion and detailed parts. Finally, by synthesizing the optical flow information of all frames, the algorithm can accurately estimate the complete motion trajectory of the object in three-dimensional space.

[0084] Step S103: Clean the collected trajectory time series, smooth the trajectory data using a method based on local weighted regression, remove noisy data, and fill in missing values ​​using interpolation to ensure the integrity and continuity of the data.

[0085] Step S104: Standardize the cleaned trajectory time series by using the Z-score normalization method to normalize features such as velocity and acceleration. The normalization formula is as follows:

[0086] ;

[0087] Where x is the original eigenvalue, μ is the mean, and σ is the standard deviation. These are the standardized feature values.

[0088] Step S105: Perform time window segmentation on the trajectory time series. Use the sliding window technique to divide the trajectory time series into segments of fixed length. Set the time window length to 5 seconds and the sliding step size to 1 second to generate trajectory time series samples for training.

[0089] Step S106: Design a temporal prediction network model based on Long Short-Term Memory (LSTM) network. The temporal prediction network model includes an input layer, multi-layer LSTM network units, a dropout layer, and a fully connected output layer. The LSTM network units use a gating mechanism (including input gate, forget gate, and output gate) to extract time-dependent features.

[0090] like Figure 2 As shown, the temporal prediction network model first receives target trajectory information from trajectory time-series samples. These samples come from multiple consecutive image frames, each containing real-time motion features of the target, such as position, velocity, and acceleration, which are then processed and input into the LSTM network. In the figure, the target's trajectory information is collected along the time axis, processed through time-series analysis, and then fed into the LSTM unit.

[0091] Next, the LSTM unit processes the input data step by step through its unique gating mechanism, including input gates, forget gates, and output gates. Specifically, the LSTM network calculates and generates the current hidden state and long-term memory based on the input data at each time step and the hidden state at the previous time step. A spatial edge LSTM embedding function φ is defined, whose inputs are the relative spatial features and the spatial embedding weight matrix. , is represented as:

[0092] ;

[0093] in, Represents a node All relative spatial distances between it and its neighboring nodes Represents a node The relative spatial characteristics at time t.

[0094] Spatial Edge LSTM extracts embedded input features and previous spatial hidden states from all relevant nodes. And use the normally initialized weight matrix. Transform them and output the hidden state vector. , is represented as:

[0095] ;

[0096] These states are passed to the next LSTM unit, gradually forming a feature representation of the time series data.

[0097] To prevent overfitting and improve the model's generalization ability, LSTM networks incorporate dropout layers between each layer. Dropout layers reduce the model's dependence on certain neurons by randomly discarding neurons, thereby enhancing the network's robustness and preventing overfitting during training.

[0098] Through a deep learning network with multiple LSTM units, the model progressively extracts the dynamic features of the target. The network continuously updates its memory state through recursive computation of multiple LSTM units, ultimately generating an output for predicting the target's future trajectory.

[0099] Finally, the residual connections introduced during training enhance the gradient propagation capability of deep LSTM networks, effectively avoiding the gradient vanishing problem during deep network training. This enables the network to more accurately capture the long-term dependencies of time-series data and make precise predictions in target behavior prediction.

[0100] Step S107: Use residual connections in the LSTM unit to extract features of the dynamic characteristics of the time series through a multi-layer network, thereby enhancing the gradient propagation capability of the multi-layer LSTM unit.

[0101] Step S108: Input the trajectory time series samples into the LSTM model, and define the mean squared error (MSE) as the loss function, the formula of which is:

[0102] ;

[0103] This represents the total number of trajectory time series samples. This represents the true value of the i-th trajectory time series sample. This represents the predicted value of the i-th trajectory time series sample.

[0104] Step S109: The Adam optimizer is used for model training. The first-order momentum parameter β1 = 0.9, the second-order momentum parameter β2 = 0.999, the initial learning rate α = 0.001, and the computational efficiency is improved by mini-batch stochastic gradient descent.

[0105] Step S110: Introduce an early stopping mechanism during training. When the validation set loss does not decrease significantly within 5 consecutive training cycles, terminate training and optimize the model parameters.

[0106] In one embodiment, step S2 specifically includes:

[0107] A heterogeneous graph attention network (HGAT) model was constructed and trained to analyze the spatial relationships among traffic participants. An attention mechanism was used to focus on key interactions and improve the ability to model spatial behavior.

[0108] like Figure 3 As shown, the heterogeneous graph attention network model consists of an input topology, a feature extraction module, and a graph attention network. In the topology of the heterogeneous graph in the traffic scenario, nodes represent different types of traffic participants, and edges represent the interaction relationships between participants. The features of the edges are obtained by calculating the Euclidean distance between nodes. Relative velocity The spatial relationships and influences among participants are quantified using factors such as relative angle changes. The feature extraction module focuses on the interactions between nodes and their neighbors through an attention mechanism. Node features are processed into features of different scales through convolution and upsampling operations before being input into the graph attention network. The attention mechanism dynamically assigns weights based on the relationships between nodes and their neighbors. In the graph attention network, at each time step t, a node calculates its attention weights using the features of its neighbors and the interaction features of its edges. The node features are aggregated in a weighted manner. Node feature representations are progressively updated in each layer of the Graph Attention Network (GAT). Through progressive information propagation and aggregation across multiple layers of GAT, node features can capture global spatial relationship information, which can be used for modeling and predicting the behavior of traffic participants.

[0109] Step S201: Construct a heterogeneous graph model of the traffic scene, defining nodes in the heterogeneous graph to represent different types of traffic participants, each node type having different attribute characteristics. Node characteristics include dynamic information of the traffic participants, such as location. ,speed acceleration In addition to static information, such as the types of traffic participants and lane information, the edges of the heterogeneous graph of the traffic scene represent the interaction relationships between nodes. The characteristics of the edges include the distance between nodes, relative speed, and relative angle changes.

[0110] Step S202: Assign features to each node by defining an appropriate feature space to describe the behavior of traffic participants. Node features should include time-series data, such as historical location, speed, acceleration, and other dynamic behavioral information, to capture the behavioral patterns of participants.

[0111] In this step, the characteristics of the edges in the graph are defined. For each pair of connected nodes... and Simultaneously calculate their relative spatial relationship: Euclidean distance. Relative velocity and relative angle changes These edge features provide quantitative information about the interaction between each pair of traffic participants.

[0112] Step S203: Define interaction features for the edges in the graph. Edge features characterize the influence or dependency between traffic participants by calculating the relative spatial relationships between nodes, such as Euclidean distance, relative speed, and relative angle changes. These edge features can describe the interaction patterns between participants.

[0113] Step S204: A Graph Attention Network (GAT) is used for information propagation within the graph structure. In GAT, the information transfer process between a node and its neighbors is achieved by calculating the weights between nodes through a self-attention mechanism, and then weighting and summing the information of neighboring nodes based on these weights to focus on important interactions. In each layer, information transfer between a node and its neighbors is achieved by calculating the attention weights between nodes. Specifically, nodes... From its neighboring nodes The process of receiving information is as follows:

[0114] ;

[0115] in, and These are nodes and Feature representation, It is the weight matrix of node features. It is the weight vector of the attention mechanism. Indicates feature splicing, It is a node From node The received attention weights. Represents a node The set of neighboring nodes, This indicates transpose.

[0116] The information aggregation process of nodes is as follows:

[0117] ;

[0118] Represents a node Updated feature representation, It is an activation function.

[0119] Step S205: Information is transmitted through a multi-layer graph attention network. Each layer calculates weighted interaction information between nodes using adaptive attention weights, and gradually aggregates global information through this mechanism. The output of each attention layer consists of neighbor node information and its own node information weighted by the attention mechanism.

[0120] Step S206: During training, target behavior data is used for training, with mean squared error (MSE) as the loss function. The representation of node features and edge features is optimized through backpropagation, and the weights in the attention mechanism are adjusted to minimize the error between the predicted trajectory and the actual trajectory.

[0121] Step S207: Train the Heterogeneous Graph Attention Network (HGAT) model using the Adam optimizer. Set the momentum parameter of the Adam optimizer. , Initial learning rate During training, the Mini-batch Stochastic Gradient Descent (Mini-batch SGD) method was used, and the Heterogeneous Graph Attention Network (HGAT) model was optimized using the backpropagation algorithm to ensure the efficiency and stability of the training process.

[0122] In one embodiment, step S3 specifically includes:

[0123] By integrating the temporal prediction network model and the heterogeneous graph attention network (HGAT) model, and combining temporal and spatial information, a comprehensive prediction is made, thereby improving the accuracy of the prediction.

[0124] Step S301: The temporal dependency features output by the temporal prediction network model and the spatial interaction features output by the heterogeneous graph attention network model are fused through a merging layer. The temporal prediction network model mainly extracts the temporal dynamic features of target behavior from time series data, capturing the temporal dependency of target behavior; while the heterogeneous graph attention network model learns the spatial relationships between targets through graph attention networks, extracting the interaction features between targets. The merging layer effectively integrates these two parts of features to generate a comprehensive feature vector containing both temporal and spatial information of target behavior.

[0125] Step S302: The fused features are further processed through a fully connected layer. The fully connected layer receives the fused features from the merging layer and maps them to the target prediction result through a series of nonlinear transformations, outputting the target's future behavior, such as position, velocity, and acceleration.

[0126] Step S303: Define a joint loss function to optimize the prediction results. The joint loss function includes the prediction error of temporal dependency features from the temporal prediction network, the prediction error of spatial interaction features from the heterogeneous graph attention network, and the prediction error of the fusion of temporal dependency features and spatial interaction features. The loss function uses mean squared error (MSE) to measure the difference between the predicted values ​​of the behavior prediction model and the true values, and weights the loss according to the task requirements to achieve a reasonable fusion of temporal and spatial features.

[0127] Step S304: The temporal prediction network model and the heterogeneous graph attention network model are pre-trained separately to enable them to independently learn temporal dependency features and spatial interaction features. Then, the outputs of the temporal prediction network model and the heterogeneous graph attention network model are optimized within an ensemble framework using a joint loss function. The parameters of the temporal prediction network model and the heterogeneous graph attention network model are adjusted using the backpropagation algorithm through the joint loss function, resulting in optimal fusion of temporal and spatial features within the model.

[0128] Step S305: Train using the Nadam optimizer and set the momentum parameters. , Initial learning rate The training process is accelerated using Mini-batch Stochastic Gradient Descent (SGD). During training, the parameters of the temporal prediction network model and the heterogeneous graph attention network model are progressively optimized. By updating the model parameters, prediction errors are reduced, and the model is ensured to converge to the global optimum.

[0129] Step S306: Monitor the training process using the validation set, adjust hyperparameters such as the learning rate in real time, and perform a final evaluation of the ensemble model using the test set. Evaluation metrics include mean bias error (ADE), final bias error (FDE), and prediction latency. Further adjust hyperparameters based on the evaluation results to optimize model performance.

[0130] In one embodiment, the method further includes: evaluating prediction accuracy and real-time performance through experimental verification in simulated traffic scenarios, and optimizing the behavior prediction model to reduce computational consumption and improve real-time processing capabilities.

[0131] In the simulation, Carla is used to generate various complex traffic scenarios, setting different traffic densities, weather conditions, road types, and emergencies to simulate the operation of the autonomous driving system under various situations. Based on the simulation environment, the performance of the model in complex scenarios is evaluated, the average prediction error (ADE) is calculated, and the performance of the behavior prediction model is detected.

[0132] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0133] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0134] Based on the description of the above method embodiments, the present invention also provides a system. The system may be a system that uses software (applications), modules, components, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary implementation hardware. Based on the same innovative concept, the systems in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem are similar, the specific system implementations in the embodiments of this specification can refer to the implementations of the foregoing methods, and repeated details will not be repeated. As used below, the terms "module" or "module group" refer to a combination of software and / or hardware capable of implementing a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0135] A target behavior prediction system based on heterogeneous graph attention networks is proposed, and the construction and training of the behavior prediction model are achieved through the following modules:

[0136] The time feature extraction module collects real-time motion features of targets in traffic scenarios, generates trajectory time series samples, and uses a time-series prediction network model based on long short-term memory network to extract the time-dependent features of target behavior.

[0137] The spatial feature extraction module constructs a heterogeneous graph attention network model, including a traffic scene heterogeneous graph and a graph attention network. Nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features that cover the dynamic and static information of traffic participants. Edges represent the interaction relationships between traffic participants and have edge features that cover the changes in distance, relative speed, and relative angle between traffic participants. The traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features.

[0138] The fusion training module integrates temporal-dependent features with spatial interaction features, processes the fused features through a fully connected layer to predict the target's future behavior, and optimizes the behavior prediction model using a joint loss function.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0141] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A target behavior prediction method based on a heterogeneous graph attention network, characterized in that, The construction and training process of the behavior prediction model adopted includes: Real-time motion features of the target in the traffic scene are collected, trajectory time series samples are generated, and a time series prediction network model based on a long short-term memory network is used to extract time-dependent features of the target behavior; A heterogeneous graph attention network model is constructed, including a traffic scene heterogeneous graph and a graph attention network; the nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features covering dynamic information and static information of the traffic participants, and the edges represent interaction relationships between the traffic participants and have edge features covering distance, relative speed and relative angle change between the traffic participants; the traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features; The time-dependent features and the spatial interaction features are fused, the fused features are processed through a fully connected layer to predict the future behavior of the target, and a joint loss function is used to optimize the behavior prediction model.

2. The target behavior prediction method based on the heterogeneous graph attention network according to claim 1, characterized in that, The collection of real-time motion features of the target in the traffic scene and the generation of trajectory time series samples include: Real-time motion features of the target are collected, including position, speed, acceleration and heading angle; Based on the real-time motion features of the target, map and traffic scene data are combined, and an incremental structure light flow algorithm is used to analyze multiple frames of image data; the generated sequence of position coordinates of the target at consecutive multiple time points is taken as a spatial trajectory, and the spatial trajectory is arranged in time sequence to form a trajectory time series, while the semantic information of the surrounding environment is labeled; The trajectory time series is preprocessed, and the preprocessed trajectory time series is segmented into fixed-length segments using a sliding window technique to generate trajectory time series samples.

3. The target behavior prediction method based on the heterogeneous graph attention network according to claim 1, characterized in that, The time series prediction network model based on the long short-term memory network is used to extract the time-dependent features of the target behavior, specifically including: The time series prediction network model includes an input layer, multiple long short-term memory network units, a dropout layer and a fully connected output layer; the long short-term memory network unit uses a gating mechanism and a residual connection to extract time series features of the trajectory time series sample as the time-dependent features of the target behavior; The trajectory time series sample and the mean square error loss MSE are used to train the time series prediction network model, and the mean square error loss MSE is: ； denotes the total number of trajectory time series samples, denotes the true value of the i-th trajectory time series sample, denotes the predicted value of the i-th trajectory time series sample.

4. The target behavior prediction method based on the heterogeneous graph attention network according to claim 1, characterized in that, The dynamic information includes the position, speed and acceleration of the traffic participant, and the static information includes the type and lane information of the traffic participant; the traffic participant includes the target as the prediction object, and vehicles and pedestrians that have interaction relationships with the target; The graph attention network is used to calculate the attention weights between nodes through a self-attention mechanism, aggregate neighbor node information to update node features; the final node features obtained after processing through multiple layers of graph attention network are the spatial interaction features; The time-dependent features and the spatial interaction features are fused, the fused features are processed through a fully connected layer to predict the future behavior of the target, and a joint loss function is used to optimize the behavior prediction model, specifically including: Using mean squared error as loss function Training the graph attention network: ； where m represents the prediction time step, represents the predicted position of the traffic participant at the jth time step; represents the actual position of the traffic participant at the jth time step; the node features and edge features in the graph attention network are optimized through a backpropagation algorithm, the attention weights are adjusted, and the error between the predicted position and the actual position of the traffic participant is minimized.

5. The target behavior prediction method based on the heterogeneous graph attention network according to claim 1, characterized in that, ​ The future behavior of the target includes the position, speed and acceleration of the target at future time points; The joint loss function is: ； wherein, represents a temporal dependency feature prediction error of the temporal prediction network, represents a spatial interaction feature prediction error of the graph attention network, represents a fusion prediction error of the temporal dependency feature and the spatial interaction feature, and α, β, γ are weight coefficients, and α + β + γ = 1. 6.A target behavior prediction system based on a heterogeneous graph attention network, characterized in that, The construction and training of the adopted behavior prediction model are achieved by the following modules: A time feature extraction module collects real-time motion features of the target in the traffic scene, generates trajectory time series samples, and extracts time-dependent features of the target behavior by using a time series prediction network model based on a long short-term memory network; A spatial feature extraction module constructs a heterogeneous graph attention network model, including a traffic scene heterogeneous graph and a graph attention network; the nodes in the traffic scene heterogeneous graph represent different types of traffic participants and have node features covering dynamic information and static information of the traffic participants, and the edges represent interaction relationships between the traffic participants and have edge features covering distance, relative speed and relative angle change between the traffic participants; The traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features; A fusion training module fuses the time-dependent features and the spatial interaction features, processes the fused features through a fully connected layer to predict the future behavior of the target, and optimizes the behavior prediction model by using a joint loss function.

7. The target behavior prediction system based on the heterogeneous graph attention network according to claim 6, characterized in that, The collection of real-time motion features of the target in the traffic scene and the generation of trajectory time series samples include: Collecting real-time motion features of the target, including position, speed, acceleration and heading angle; Based on the real-time motion features of the target, combining map and traffic scene data, using an incremental structured light flow algorithm to analyze multiple image data, the generated sequence of position coordinates of the target at consecutive time points is taken as a spatial trajectory, and the spatial trajectory is arranged in time sequence to form a trajectory time series, while the semantic information of the surrounding environment is labeled; The trajectory time series is preprocessed, and the preprocessed trajectory time series is segmented into fixed-length segments by using a sliding window technique to generate trajectory time series samples.

8. The target behavior prediction system based on the heterogeneous graph attention network according to claim 6, wherein, The time series prediction network model includes an input layer, multiple long short-term memory network units, a dropout layer and a fully connected output layer; the long short-term memory network unit extracts time series features of the trajectory time series sample by using a gating mechanism and a residual connection, as the time-dependent features of the target behavior; The trajectory time series sample and the mean square error loss MSE are used to train the time series prediction network model, and the mean square error loss MSE is: The dynamic information includes the position, speed and acceleration of the traffic participant, and the static information includes the type and lane information of the traffic participant; the traffic participant includes the target as the prediction object, and vehicles and pedestrians that have interaction relationships with the target; ； denotes the total number of trajectory time series samples, denotes the true value of the i-th trajectory time series sample, denotes the predicted value of the i-th trajectory time series sample. 9.The target behavior prediction system based on heterogeneous graph attention network according to claim 6, wherein, The traffic scene heterogeneous graph is input into the graph attention network to extract spatial interaction features, specifically including: The graph attention network is used to calculate the attention weights between nodes by using a self-attention mechanism, aggregate neighbor node information to update node features, and obtain the final node features after multiple layers of graph attention network processing, which are the spatial interaction features; ​ Using mean squared error as loss function Training the graph attention network: ； where m represents the prediction time step, represents the predicted position of the traffic participant at the jth time step; represents the actual position of the traffic participant at the jth time step; the node features and edge features in the graph attention network are optimized through a backpropagation algorithm, the attention weights are adjusted, and the error between the predicted position and the actual position of the traffic participant is minimized.

10. The target behavior prediction system based on the heterogeneous graph attention network according to claim 6, wherein, The time-dependent feature is fused with the space interaction feature, the fused feature is processed through a full connection layer to predict the future behavior of the target, and a joint loss function is used to optimize the behavior prediction model, specifically including: The future behavior of the target includes the position, speed and acceleration of the target at the future moment. The joint loss function is: ； wherein, represents a time-dependent feature prediction error of the temporal prediction network, represents a spatial interaction feature prediction error of the graph attention network, represents a prediction error of the fused feature, and a, b, g are weight coefficients, and a+b+g = 1.

Images