A vehicle lane-changing planning method and system based on graph neural networks and multi-agent systems

Abstract

This invention provides a vehicle lane-changing planning method and system based on graph neural networks and multi-agent systems, relating to the field of autonomous driving technology. The method involves: acquiring a traffic scene graph structure with multiple agents; training a graph neural network fusion model using a multi-agent reinforcement learning algorithm and corresponding reward functions; updating the parameters of the graph neural network fusion model by minimizing the policy gradient loss based on the advantage function and the value function loss, thus obtaining a trained graph neural network fusion model; analyzing the traffic scene graph structure using the trained graph neural network fusion model to obtain multi-agent decision-making strategies; fusing the decision priors of the multi-agent decision-making strategies and dynamically interacting with the environment to obtain the vehicle lane-changing planning result, thus completing the vehicle lane-changing planning. This invention solves the problems of low accuracy and poor robustness in multi-agent trajectory tracking in complex scenarios.

Description

Technical Field

[0001] This specification relates to the field of autonomous driving technology, and in particular to a vehicle lane-changing planning method and system based on graph neural networks and multi-agent systems. Background Technology

[0002] Lane-changing decisions and control for autonomous vehicles are a key challenge in intelligent transportation systems, significantly impacting driving safety and traffic efficiency. During lane changes, autonomous vehicles must interact in real-time with surrounding human-driven vehicles in a high-speed dynamic environment, handling complex uncertainties to achieve safe, efficient, and smooth trajectory changes. However, traditional decision-making and control methods have many limitations in addressing these challenges. Traditional non-data-driven methods mainly fall into two categories: rule-driven and model-driven. Rule-driven methods employ manually designed heuristic rules, such as maintaining a safe following distance and choosing the right time to change lanes. Early low-complexity lane-changing algorithms followed these rules to achieve lane changes. Model-driven methods, on the other hand, are based on vehicle dynamics models and optimization solutions to achieve trajectory planning and control. For example, some studies pre-set virtual reference trajectories for each vehicle and consider the trajectories of other vehicles to plan safe lane-changing paths; other works model the lane-changing planning problem as a longitudinally and laterally coupled predictive control optimization, solving for the optimal trajectory through quadratic programming (QP). These MPC (Model Predictive Control) or optimization methods can provide high trajectory tracking accuracy in structured environments. However, due to the complexity and variability of real-world traffic scenarios, artificial rules and optimization objective functions are insufficient to fully characterize all possible situations. When encountering uncertain driving behaviors that are not modeled, traditional methods often struggle to make timely and effective optimal decisions. Especially in mixed traffic scenarios, human driver behavior exhibits randomness and variability, making it difficult to guarantee robustness and universality of control strategies that rely on fixed rules or models. Furthermore, traditional planning-control schemes typically require replanning the trajectory and solving the optimization problem in each decision cycle, resulting in significant computational overhead and limited real-time performance. Summary of the Invention

[0003] To address the aforementioned shortcomings in existing technologies, the vehicle lane-changing planning method and system based on graph neural networks and multi-agent systems provided by this invention solves the problems of low trajectory tracking accuracy and poor robustness of multi-agent systems in complex scenarios.

[0004] To achieve the aforementioned objectives, the present invention employs the following technical solution: a vehicle lane-changing planning method based on graph neural networks and multi-agent systems, comprising:

[0005] S1: Obtain the traffic scene graph structure with multiple agents;

[0006] S2: Using a multi-agent reinforcement learning algorithm and the corresponding reward function, train the graph neural network fusion model. Update the parameters of the graph neural network fusion model by minimizing the policy gradient loss and value function loss based on the advantage function, and obtain the trained graph neural network fusion model.

[0007] S3: Using a trained graph neural network fusion model, analyze the graph structure of the traffic scene to obtain multi-agent decision-making strategies;

[0008] S4: Perform decision prior fusion on the multi-agent decision-making strategy, dynamically interact with the environment, obtain the vehicle lane-changing planning result, and complete the vehicle lane-changing planning.

[0009] The beneficial effects of the present invention are as follows: The present invention provides a vehicle lane-changing planning method based on graph neural network and multi-agent, which uses multi-agent reinforcement learning algorithm and corresponding reward function to train graph neural network fusion model, updates the parameters of graph neural network fusion model by minimizing policy gradient loss based on advantage function and value function loss, and obtains trained graph neural network fusion model, uses trained graph neural network fusion model to analyze traffic scene graph structure and obtain multi-agent decision strategy; performs decision prior fusion on the multi-agent decision strategy, interacts dynamically with the environment, obtains vehicle lane-changing planning result, and completes vehicle lane-changing planning. (1) The executable control quantity is directly output by the policy network, reducing the multi-level coupling of "rule base / trajectory optimization + tracking control", reducing engineering complexity and delay propagation error, and facilitating vehicle deployment and maintenance. (2) Graph neural network explicitly encodes multi-vehicle interaction and topological relationship, and can adaptively adjust strategy in dynamic mixed traffic in combination with multi-agent reinforcement learning, solving the problem of insufficient robustness of traditional methods to "unmodeled human driving behavior". (3) The training phase uses global information to learn the collaborative strategy, while the execution phase relies only on the observations available to the vehicle, which ensures both collaborative capability and meets the real-world constraints of limited vehicle resources and communication. (4) The reward function intrinsically balances safety, efficiency, comfort, and collaboration, avoiding unnecessary lane changes, sudden acceleration and deceleration, and other poor driving behaviors, thereby improving the overall traffic quality. (5) The forward computation complexity of the strategy is linearly related to the vehicle size and has no iterative optimization overhead; it can still meet the millisecond-level decision cycle requirements of the vehicle in high-density traffic. (6) In the joint simulation, compared with MPC / rule baseline, the collision rate is significantly reduced, the lane change success rate is improved, the average speed is increased, and the jerk is lower, demonstrating the stability advantage of the method in complex scenarios.

[0010] Further, S1 includes:

[0011] Based on the state observed by multiple agents, the initial feature vectors of the multiple agents are extracted;

[0012] Node features are constructed using the initial feature vector and neighbor information;

[0013] Edge features are constructed using the relative position and relative velocity between two nodes;

[0014] Based on the node features and edge features, a traffic scene graph structure is constructed.

[0015] Using "relative position / speed" as edge features and "vehicle and neighboring vehicle states" as node features, a sparse graph is formed. The computational burden increases approximately linearly with the local neighborhood, making it easy to scale to large-scale traffic flows. The graph representation is insensitive to vehicle numbers and order, naturally adapting to topological changes such as varying numbers of vehicles and ramp entry / exit, improving generalization ability. Using "relative speed / relative distance" to characterize potential conflicts and lane-changing gaps is closer to the driving decision-making mechanism than pure rule or gridded input, improving the analyzability of lane-changing timing judgment. Neighborhood aggregation mitigates the impact of single-frame noise and missing measurements, making it more robust to sensor occlusion / short-term frame drops.

[0016] Furthermore, the graph neural network fusion model includes:

[0017] The input feature layer is used to extract features from the graph structure to obtain the vehicle's own and relative information features;

[0018] A linear embedding layer is used to unify and pre-normalize the dimensionality of the vehicle's own and relative information features to obtain normalized features.

[0019] The first attention convolutional layer is used to perform a one-hop interaction on the normalized features using four-head attention to obtain the first interaction result.

[0020] The second attention convolutional layer is used to fuse two-hop neighbor information based on normalized features to obtain the second interaction result;

[0021] Residual convolutional layers are used to enable normalized features as needed based on the simulation scale, resulting in enabled data.

[0022] The readout-stitching layer is used to process the first interaction result, the second interaction result, and the activation data using a multilayer perceptron to obtain the policy input data;

[0023] The actor-critic network layer, consisting of an actor network and a critic network, is used to perform fully connected processing on the policy input data to obtain vehicle lane-changing planning instructions.

[0024] The four-headed GAT performs a first-hop and second-hop approach, focusing on key neighboring vehicles while also acquiring prior knowledge of "neighbors' neighbors," thus anticipating potential conflict chains. It unifies the dimensions and weights key information, improving numerical stability and representation efficiency during message passing. In high-density / deep propagation scenarios, it mitigates oversmoothing and gradient degradation, stabilizing training and enhancing deep representation. By compressing local topological embeddings into compact policy inputs, continuous action outputs result in smoother horizontal and vertical control, outperforming coarse-grained switching of discrete actions.

[0025] Furthermore, the expression for the first interaction result is:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, Indicates the first Layer nodes The hidden state is a representation of the original features of a node after multi-hop neighborhood interactions. Represents a non-linear activation function. Indicates the first The layer is the total message vector obtained by aggregating all neighbors. Indicates the first Layer training bias vectors, Indicates the first From the neighbor Incoming one-sided message, Represents a node Let j and k represent the set of neighboring nodes in the set. Index in Indicates attention weights, Indicates the first A layer-shared linear mapping matrix is ​​used for feature transformation. Represents a node In the hidden state of the previous layer, express function, This represents the training attention vector, used to score the concatenated vector, where T denotes the vector transpose. This represents the natural exponential function.

[0031] Attention weights are dynamically allocated to information channels based on the scene, amplifying key neighboring vehicles and suppressing redundant nodes to avoid information dilution caused by average aggregation. Visualizing attention weights helps to verify "strategy focus points" offline, facilitating safety reviews and parameter tuning. The combination of splicing scoring and sharing linear mapping matrices balances expressiveness and parameter sharing, resulting in more stable training and avoiding overfitting and drift caused by graph depth.

[0032] Further, S2 includes:

[0033] Obtain the environmental state of the traffic scene graph structure; whereby the environmental state includes the observation data of each agent;

[0034] Using the aforementioned graph neural network fusion model, the environmental state of the traffic scene graph structure is analyzed to obtain the agent's action vector:

[0035] ;

[0036] in, Represents the action vector of the agent. Indicates the desired lateral acceleration. This represents the desired longitudinal acceleration;

[0037] The environment after each agent executes the corresponding action vector is taken as the environment state at the next moment, and the instantaneous reward of the graph neural network fusion model is obtained by using the reward function.

[0038] A multi-agent reinforcement learning algorithm is used to train each agent in a centralized manner and execute them in a distributed manner. A critic network is used to evaluate the overall value of the state and actions of all agents, and the critic network parameters and global state value are obtained.

[0039] The actor network is used to obtain the output action of each agent based on its own state distribution. Based on the immediate reward, the critic network parameters and the global state value, the parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss and value function loss based on the advantage function, and the trained graph neural network fusion model is obtained.

[0040] Directly outputting the desired lateral / longitudinal acceleration reduces discontinuities and high-frequency jitter in lane-changing trajectories, improving ride comfort. Padding / masking unifies the network dimension for a variable number of vehicles, improving sample utilization and generalization. A global critic evaluates the joint value, alleviating the "environment non-stationarity" problem in multi-agent learning and accelerating convergence. In joint simulations, lane-changing success rates are higher, average speed is improved, and fuel consumption is lower.

[0041] Furthermore, the immediate rewards include driving safety, driving efficiency, passenger comfort, and cooperation factors, and the expression for the immediate rewards is:

[0042] ;

[0043] in, Indicates vehicle At any moment The immediate rewards obtained can be used as optimization targets for RL (Research and Development). , , , and This represents the weighting coefficient for each reward / penalty component, used to measure the importance of different indicators. This indicates a reward for driving efficiency. This indicates a cooperation reward; the closer to the target lane / the higher the reward upon successful lane change, and it is used to guide vehicles to complete lane changes within a specified distance. This indicates a negative reward for driving safety, suggesting that potential collisions or sudden braking should be mitigated in advance. This indicates a penalty for changing lanes, the negative consequences of excessive or frequent lane changes. This indicates acceleration penalty and ride comfort index.

[0044] The weighting coefficients for safety, efficiency, comfort, and collaboration can be quickly tuned according to the application scenario, supporting switching between conservative and aggressive driving styles and scenario migration. It penalizes frequent lane changes and high acceleration, significantly reducing unnecessary lane changes and limiting rapid changes in acceleration, deceleration, and steering. This significantly reduces vehicle swaying and the incidence of sudden braking, resulting in a more stable, smoother, and more comfortable ride. The example demonstrates enhanced safety (fewer collisions), higher efficiency (higher speed and arrival rate), smoother ride, and better fuel / electricity savings.

[0045] Furthermore, the expression for the policy gradient loss is:

[0046] ;

[0047] ;

[0048] in, Indicates to The gradient operator is used for optimization. This represents the policy objective function, which maximizes the value of the object. Indicating compliance strategy The expected trajectory Representing the log-likelihood facilitates gradient calculation. Represents intelligent agents At any moment The action, Represents intelligent agents The observations are obtained through graph attention convolution. The advantage function represents the quality of an action. Represents the action value function. Indicates the global / centralized state of the environment. This represents the state-value function, reducing variance.

[0049] A dominant baseline significantly reduces policy gradient variance, resulting in more stable training and higher sample efficiency. Centralized commentators provide more accurate joint value assessments, alleviating the difficulty of credit allocation for multi-vehicle cooperative behavior and encouraging beneficial collaborative strategies such as yielding / overtaking. Stable terms such as entropy regularization can be naturally superimposed, preventing premature convergence and facilitating exploration.

[0050] Furthermore, the expression for the value function loss is:

[0051] ;

[0052] in, This represents the value function loss, which measures the error in the critic's estimation of the state's value. This represents the state value estimate output by the value network. Indicates time The environmental conditions, Indicates the target return, for Estimates of real returns This indicates the critic's parameters.

[0053] Explicit constraints on value error reduce critic bias, prevent misleading the strategy, and result in more stable overall training. More accurate state values ​​make advantage estimation more reliable, policy update directions are "cleaner," and oscillations and regressions are reduced.

[0054] Further, S4 includes:

[0055] The decision-making strategies of the multi-agent agents are fused with prior decisions. When a collision is detected that the policy output may lead to a collision, a collision avoidance action is triggered.

[0056] Based on the current observed state of the agents, the multi-agent decision-making strategy is updated using a trained graph neural network fusion model, and the system interacts dynamically with the environment to obtain the vehicle lane-changing planning result and complete the vehicle lane-changing planning.

[0057] By overlaying a priori modules for collision avoidance / constraint triggering at the strategy execution layer, a dual-protection framework of "learning strategy + rule fallback" is formed to meet the requirements for functional safety implementation. When scenarios outside the training distribution or extreme situations occur, the safety prior can intervene immediately to limit / brake, significantly reducing the risk of rare catastrophic events. After safety is triggered, the strategy can be adaptively corrected in the next cycle based on updated observations, ensuring that the lane-changing process is not interrupted or can be gracefully abandoned when necessary, demonstrating strong stability. The prior and reward work together to encourage a collaborative mode of "yielding—merging—recovery," reducing conflicts caused by parallel competition.

[0058] This invention provides a vehicle lane-changing planning system based on graph neural networks and multi-agent systems, comprising:

[0059] The perception module is used to acquire state information of the intelligent agent and surrounding traffic participants, and to construct a traffic scene graph structure.

[0060] The decision module is used to extract and encode features from the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and use multi-agent reinforcement learning decision strategies to output multi-agent decision strategies.

[0061] The vehicle control module is used to perform decision prior fusion on the decision-making strategies of the multi-agent system, dynamically interact with the environment based on the current observed state of the agents, obtain the vehicle lane-changing planning result, and complete the vehicle lane-changing planning. Attached Figure Description

[0062] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0063] Figure 1 This is a schematic diagram of a vehicle lane-changing planning system based on graph neural networks and multi-agent systems, as shown in some embodiments of this specification.

[0064] Figure 2 This is an exemplary flowchart of a vehicle lane-changing planning method based on graph neural networks and multi-agent systems, as shown in some embodiments of this specification.

[0065] Figure 3 This is an exemplary schematic diagram of an intelligent vehicle lane replanning control framework according to some embodiments of this specification;

[0066] Figure 4 This is an exemplary schematic diagram of the training process of a graph neural network fusion model according to some embodiments of this specification;

[0067] Figure 5 This is an exemplary schematic diagram illustrating a traffic flow-vehicle dynamics co-simulation framework and a road scene according to some embodiments of this specification. Detailed Implementation

[0068] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0069] Example 1

[0070] Figure 1 This is a schematic diagram of a vehicle lane-changing planning system based on graph neural networks and multi-agent systems, as shown in some embodiments of this specification.

[0071] In some embodiments, the vehicle lane change planning system based on graph neural networks and multi-agent systems may include a perception module, a decision-making module, and a vehicle control module.

[0072] The perception module is used to acquire state information of the intelligent agent and surrounding traffic participants, and to construct a traffic scene graph structure.

[0073] The decision module is used to extract and encode features from the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and output the multi-agent decision strategy using multi-agent reinforcement learning.

[0074] The vehicle control module is used to perform decision prior fusion on the decision-making strategies of the multi-agent system, dynamically interact with the environment based on the current observed state of the agents, obtain the vehicle lane-changing planning result, and complete the vehicle lane-changing planning.

[0075] In some embodiments, such as Figure 3The diagram illustrates the perception-decision-control closed loop of this invention. From left to right, the sequence is as follows: onboard sensors & communication (camera, millimeter-wave radar, lidar, GPS / IMU, V2X); perception module (time synchronization, calibration, and multi-source fusion to obtain the vehicle / neighbor vehicle state); graph neural network processing (organizing the state into a traffic scene graph G=(V,E), where nodes represent vehicles and edges represent interaction / communication relationships, and performing topological encoding); multi-agent reinforcement learning decision-making (outputting candidate actions: hold / left lane change / right lane change and target longitudinal acceleration / velocity); decision module (overlaying safety constraints and prior rules to filter executable instructions); vehicle control module (trajectory generation and tracking, executing steering and driving / braking), ultimately executed by the intelligent vehicle. This process operates in a fixed decision-making cycle in an online closed loop, achieving safe, smooth, and efficient lane changing in dynamic traffic flow. Each intelligent vehicle acquires state information about itself and surrounding traffic participants through onboard sensors (cameras, radar, etc.) and vehicle-to-everything (V2X) communication, including its own speed, acceleration, position, lane, and relative distance and speed to neighboring vehicles. The perception module processes this information to construct a traffic scene graph G=(V,E), where the set of nodes V represents the vehicles in the road network, and the set of edges E represents the interaction relationships between vehicles (such as neighbor relationships or communication connections). Subsequently, the decision module uses a graph neural network to extract and encode features from the scene graph, mapping the multi-vehicle interaction state into a compact low-dimensional representation, and inputs this into a multi-agent reinforcement learning decision strategy to output lane-changing decisions and control actions for each intelligent vehicle. Specifically, for each intelligent vehicle, the decision module outputs a set of control commands based on its currently observed state (including neighbor information fused by the GNN), including lateral lane-changing intentions and longitudinal speed adjustments. Finally, the vehicle control module executes corresponding lane-changing and trajectory tracking control according to the decision commands: for example, when the decision is to change lanes, the control module generates steering and acceleration commands to smoothly change lanes towards the target lane; when the decision is to maintain lane position, the control module maintains the current lane while cruising. The entire process operates in real-time closed loop during each decision cycle, forming a "perception-decision-control" cycle: intelligent vehicles continuously perceive changes in the environment and adaptively replan lane-changing actions based on learning strategies, thereby achieving safe and efficient lane-changing maneuvers in dynamic traffic flow.

[0076] In some embodiments, a vehicle lane-changing planning system based on graph neural networks and multi-agent systems can be used to execute a vehicle lane-changing planning method based on graph neural networks and multi-agent systems, including: S1: acquiring a traffic scene graph structure with multiple agents; S2: training a graph neural network fusion model using a multi-agent reinforcement learning algorithm and a corresponding reward function, updating the parameters of the graph neural network fusion model by minimizing the policy gradient loss based on the advantage function and the value function loss, and obtaining a trained graph neural network fusion model; S3: analyzing the traffic scene graph structure using the trained graph neural network fusion model to obtain a multi-agent decision strategy; S4: fusing the decision priors of the multi-agent decision strategy, dynamically interacting with the environment, obtaining the vehicle lane-changing planning result, and completing the vehicle lane-changing planning.

[0077] In some embodiments of this specification, the processor utilizes a vehicle lane-changing planning system based on graph neural networks and multi-agent systems to execute a vehicle lane-changing planning method based on graph neural networks and multi-agent systems. By directly outputting control variables through a policy network, the system structure is significantly simplified. Furthermore, leveraging the self-learning capabilities of reinforcement learning and the powerful representational capabilities of graph networks, it can automatically adjust decisions for different traffic scenarios, achieving adaptive lane-changing replanning control.

[0078] Example 2

[0079] Figure 2 This is an exemplary flowchart illustrating a vehicle lane-changing planning method based on graph neural networks and multi-agent systems, according to some embodiments of this specification. Figure 2 As shown, the process includes the following steps. In some embodiments, the process may be executed by a processor.

[0080] S1: Obtain the traffic scene graph structure with multiple agents.

[0081] A traffic scene graph structure is a structural graph that reflects vehicle information and the relationships between adjacent vehicles in a traffic scene. For example, a traffic scene graph structure can include node features and edge features.

[0082] In some embodiments, the processor can model traffic scenes using graph neural networks (GNNs) to capture dynamic relationships between vehicles and serve as a perceptual front end for reinforcement learning strategies. The research abstracts traffic scenes as graphs... Each node Represents a vehicle on the road (including autonomous vehicles and manned vehicles around it), each edge Indicates vehicle and vehicles The interaction relationships between them. The principle of edge construction can be based on neighborhood radius or lane-changing relevance: for example, when vehicles... Located in the vehicle When within a certain distance ahead or in an adjacent lane, the study suggests and Vehicles with direct interactions are connected as adjacent nodes in the graph; however, no edge connections are established for distant vehicles beyond the perception / communication range. In this way, the graph structure naturally represents the neighbor relationships of intelligent vehicles in local traffic flows, which helps to limit the scope of information propagation and reduce model complexity.

[0083] In some embodiments, the processor can extract initial feature vectors of multiple agents based on the states observed by the multiple agents; construct node features using the initial feature vectors and neighbor information; construct edge features using the relative positions and relative velocities between two nodes; and construct a traffic scene graph structure based on the node features and the edge features.

[0084] The state observed by an agent is the vehicle state information observed by the agent. For example, the state observed by an agent can include the state information of its own vehicle and surrounding traffic participants, including its own speed, acceleration, position in the lane, and the relative distance and speed of nearby vehicles.

[0085] The initial feature vector is a vector composed of features related to the vehicle's own state and attributes observed by the agent. For example, the initial feature vector may include information about the vehicle's own state and attributes, such as longitudinal speed. acceleration The position of the vehicle in the road coordinate system Current lane number Vehicle type indicator (autonomous vehicle or manned vehicle), etc.

[0086] In some embodiments, the processor may incorporate some neighbor information into node features, such as encoding the distance and speed difference relative to the vehicle into the features of neighbor nodes, and using relative position and speed as edge features during graph construction. Alternatively, the feature descriptions of adjacent nodes can be incorporated.

[0087] Node features are the characteristics of each node in the traffic scene graph structure. For example, node features may include the initial feature vector, neighbor information, distance and speed difference relative to the vehicle itself, etc.

[0088] Edge features are the characteristics of each edge in a traffic scene graph structure. For example, edge features can include relative position and velocity.

[0089] S2: Using a multi-agent reinforcement learning algorithm and the corresponding reward function, train the graph neural network fusion model. Update the parameters of the graph neural network fusion model by minimizing the policy gradient loss and value function loss based on the advantage function, and obtain the trained graph neural network fusion model.

[0090] Graph neural network fusion models are neural network models used to analyze the graph structure of traffic scenarios and obtain vehicle lane-changing planning instructions. For example, a graph neural network fusion model can include a graph neural network model and an actor-critic network model.

[0091] In some embodiments, the graph neural network fusion model includes: an input feature layer, a linear embedding layer, a first graph attention convolutional layer, a second graph attention convolutional layer, a residual convolutional layer, a readout-sewup layer, and an actor / critic network layer.

[0092] In some embodiments, default parameter settings may include All hidden layers use ReLU, and Dropout=0.1.

[0093] The input feature layer is used to extract features from the graph structure to obtain vehicle-specific and relative information features.

[0094] Vehicle-specific and relative information features are the vehicle-specific state and attribute-related features in the traffic scene graph structure.

[0095] A linear embedding layer is used to unify and pre-normalize the dimensions of the vehicle's own and relative information features to obtain normalized features.

[0096] Normalized features are the normalized features of the vehicle itself and its relative information.

[0097] In some embodiments, the expression for the normalized feature can be:

[0098] ;

[0099] in, Indicates normalized features, Represents a non-linear activation function. Represents matrix-vector product. Represents a constant.

[0100] In some embodiments, the linear embedding layer performs scale unification and feature reweighting to provide a high-dimensional representation for subsequent message passing.

[0101] The first image shows an attention convolutional layer, which uses four-head attention to perform a one-hop interaction on the normalized features to obtain the first interaction result.

[0102] The first interaction result is the result of normalized feature one-hop neighbor information fusion.

[0103] In some embodiments, the expression for the first interaction result can be:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] in, Indicates the first Layer nodes The hidden state is a representation of the original features of a node after multi-hop neighborhood interactions. Represents a non-linear activation function. Indicates the first The layer is the total message vector obtained by aggregating all neighbors. Indicates the first Layer training bias vectors, Indicates the first From the neighbor Incoming one-sided message, Represents a node Let j and k represent the set of neighboring nodes in the set. Index in Indicates attention weights, Indicates the first A layer-shared linear mapping matrix is ​​used for feature transformation. Represents a node In the hidden state of the previous layer, express function, This represents the training attention vector, used to score the concatenated vector, where T denotes the vector transpose. This represents the natural exponential function.

[0109] In some embodiments, the multi-head attention of the graph attention convolutional layer enables nodes to adaptively focus on key neighboring vehicles; after the second layer, the information has covered the two-hop range.

[0110] The second attention convolutional layer is used to fuse two-hop neighbor information based on normalized features to obtain the second interaction result.

[0111] The second interaction result is the result of normalized feature two-hop neighbor information fusion.

[0112] Residual convolutional layers are used to enable normalized features as needed, depending on the simulation scale, to obtain enabled data.

[0113] The enabled data is the data that is enabled as needed, depending on the scale of the simulation.

[0114] In some embodiments, if the vehicle density in the scene is high or the graph depth is insufficient, a residual GAT layer can be added to alleviate oversmoothing and stabilize the gradient; whether to enable or not is determined by [the relevant authority / entity]. Dynamic judgment; This indicates the number of vehicles currently included in the graph.

[0115] In some embodiments, the expression for enabling data is:

[0116] ;

[0117] in, Represents the 3rd level node The hidden representation, obtained by one-hop / two-hop attention convolution plus activation function, is used as the input to the final readout-concatenation layer. Represents a non-linear activation function. This represents the aggregated message vector at layer 3, obtained by summing neighbor messages according to their attention weights and then performing a linear mapping. Represents the second-level node The hidden representation, as a residual or skip term, is related to... Add and then activation.

[0118] The readout-stitching layer is used to process the first interaction result, the second interaction result, and the activation data using a multilayer perceptron to obtain the policy input data.

[0119] The policy input data is the observation data of the multi-agent policy network.

[0120] In some embodiments, the processor can represent the final node. With neighboring regions and After splicing, it is mapped by two layers of MLP to Local synthesis representation , as an observation of a multi-agent policy network.

[0121] In some embodiments, the expression for the policy input data can be:

[0122] ;

[0123] in, This represents the final input feature vector of the policy network. This refers to a multilayer perceptron, used for dimensionality compression and nonlinear mapping. Represents a node After the first The final hidden representation after layer graph attention convolution, Representing neighboring nodes The final hidden representation.

[0124] The actor-critic network layer, consisting of an actor network and a critic network, is used to perform fully connected processing on the policy input data to obtain vehicle lane-changing planning instructions.

[0125] The vehicle lane change planning command is a command that controls the vehicle to plan its lane change.

[0126] In some embodiments, the processor can enable Continuous motion is obtained by using three fully connected layers (128-256-128). Or state-value ,in For lateral control actions, Longitudinal control actions serve as lane-changing planning instructions for vehicles.

[0127] In some embodiments, graph neural networks update the hidden states of nodes through a message-passing mechanism, thereby extracting higher-level interaction features layer by layer. Let... Indicates the first Nodes in a layered GNN The hidden state vector (at level 0, it can be set to...) (Assuming initial characteristics), a typical message passing process can be described as follows:

[0128] ;

[0129] ;

[0130] in, Indicates a one-sided message, neighbor In the Vectors passed by the layer This represents a message function used to generate neighbors. Send to News , Represents a node In the The hidden state vector of the layer, Representing neighboring nodes The hidden state vector, express The neighboring car, Represents a node The set of adjacent nodes, Represents a node In the The layer's updated state is hidden. This represents the update function, which is used to update the hidden state of a node after summarizing neighbor messages. and Typically, it is parameterized by a trainable neural network (such as a multilayer perceptron, MLP).

[0131] In some embodiments, a simplified approach is adopted using graph convolution: the message is a linear transformation of the neighbor hidden vectors, summed with certain weights, i.e.:

[0132] ;

[0133] in, Indicates the first Layer nodes The hidden state, Represents a non-linear activation function. Indicates the first The layer's weight matrix shares information from its neighbors and itself. Indicates the first Layer nodes The hidden state, Indicates the first Layer nodes The hidden state. This formula is equivalent to the simple case of the classic GCN model, which assumes equal neighbor contributions and uses degree normalization. In practical applications, research can also use Graph Attention Networks (GAT) to adaptively assign neighbor weights, or introduce edge features to influence message passing. Regardless of the specific implementation, after multiple GNN iterations, each node obtains a vector. This representation encodes the vehicle. The state and its interactions with neighbors in the local traffic map, such as the influence of a slower vehicle ahead on its speed and the constraints imposed by vehicles to its side and rear on its lane-changing space, are considered. These high-level features will serve as input to the reinforcement learning decision network, thus providing a graph-structure-aware state representation for subsequent decisions.

[0134] In some embodiments, the GNN is jointly trained by embedding it into a reinforcement learning policy network, without applying a separate supervised loss to the GNN. The parameter optimization of the GNN is indirectly driven by the overall goal of reinforcement learning, namely, that the output of the GNN should contribute to the correct estimation of the policy's value or Q-value. In other words, the role of the GNN is information fusion: combining the original states of multiple vehicles... Mapped to joint state features This allows the reinforcement learning agent to more accurately determine the benefits of each action. In the implementation of this research, the GNN and the subsequent decision network constitute an end-to-end trainable policy model, which is jointly trained through gradient descent during the reinforcement learning process.

[0135] In some embodiments, the processor can acquire the environmental state of the traffic scene graph structure; wherein the environmental state includes the observation data of each agent; using the graph neural network fusion model, the environmental state of the traffic scene graph structure is analyzed to obtain the action vectors of the agents; the environment after each agent executes the corresponding action vector is taken as the environmental state at the next moment, and the immediate reward of the graph neural network fusion model is obtained using a reward function; a multi-agent reinforcement learning algorithm is used to centrally train and distribute the execution of each agent, and a critic network is used to evaluate the overall value of the states and actions of all agents to obtain the critic network parameters and global state value; an actor network is used to acquire the output action of each agent based on its own state distribution, and based on the immediate reward, critic network parameters, and global state value, the parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss and value function loss based on the advantage function to obtain the trained graph neural network fusion model.

[0136] The environmental state of a traffic scene graph structure reflects the state information of the vehicles themselves and the observation data. For example, the environmental state can include information such as the vehicle's position and speed, as well as the observation data of each agent.

[0137] In some embodiments, the processor can formalize the problem as a Markov game. Consider a set of intelligent vehicles. (i.e., multiple agents), and several non-agent vehicles (such as manned vehicles, which can be considered part of the environment). At each discrete decision-making time... The entire environment can be described as a state. It includes information such as the location and speed of all vehicles; each intelligent agent You can obtain your own observations The state is composed of the agent's own state and the states of neighboring vehicles fused by a GNN. Due to the inclusion of vehicle-to-everything (V2X) communication and environmental perception, the study assumes that the agent can obtain a fairly complete local environmental state, that is, it can be approximated as globally observable or partially observable but sufficient to meet decision-making requirements. According to its strategy From observation Mapping an action In the lane-changing control settings studied, the action space can be designed as continuous or discrete. To fully leverage the advantages of reinforcement learning for continuous control, this study adopts a continuous action space: the agent... Action vector It consists of two components, lateral and longitudinal, corresponding to lane-changing steering control and longitudinal acceleration / deceleration control, respectively. This can be interpreted as a desired lateral acceleration or steering wheel angle command (controlling the lane change magnitude and rate by limiting its amplitude). The desired longitudinal acceleration (positive acceleration, negative deceleration) is used to adjust vehicle speed. Through continuous action output, the agent can finely control the vehicle to perform lane changes and follow other vehicles, with both working together to achieve smooth trajectory tracking. In contrast, a discrete action space (such as combinations of lane change / no lane change, acceleration / deceleration) could be used, but that would transform the problem into a hybrid decision-making process, potentially requiring hierarchical processing. This paper chooses continuous control to unify the decisions regarding lane changes and vehicle speed adjustment.

[0138] An agent's action vector is the vector of the action command that the agent is currently executing.

[0139] In some embodiments, the expression for the agent's action vector can be:

[0140] ;

[0141] in, Represents the action vector of the agent. Indicates the desired lateral acceleration. This represents the desired longitudinal acceleration.

[0142] In some embodiments, when all agents simultaneously perform their respective actions, the environment state... The next moment evolves through system dynamics and vehicle interaction. In this process, each agent will receive an immediate reward. This is used to provide feedback on the quality of the action.

[0143] Immediate rewards are reward functions used to balance the importance of driving efficiency, goal achievement, and various penalties. For example, immediate rewards can include driving safety, driving efficiency, passenger comfort, and cooperation factors.

[0144] Driving safety refers to the significant negative reward given if the intelligent vehicle collides with any other vehicle or the distance falls below a safety threshold. This encourages intelligent agents to avoid dangerous situations. Under normal circumstances where no collision occurs, small negative rewards can be provided based on indicators such as minimum following distance to incentivize vehicles to maintain a safe following distance.

[0145] Driving efficiency is the ability to incentivize vehicles to travel at higher speeds and complete tasks such as lane changes and overtaking as quickly as possible. Research defines speed rewards. This is the ratio of vehicle speed to the road speed limit (between 0 and 1), which encourages vehicles to drive close to the speed limit. In addition, vehicles with a specific lane-changing intention (such as about to exit) can be rewarded for achieving their goal in time (e.g., merging into the target lane before the specified distance).

[0146] Passenger comfort is a measure to avoid frequent and aggressive maneuvers, and researches include penalizing excessive acceleration and lane-changing behaviors. This involves introducing lane-changing penalties. (A certain number of points will be deducted for each lane change) and acceleration penalty (Points are deducted when acceleration changes are too large) to encourage strategies to stay in the lane as much as possible when not necessary, reducing unnecessary lane changes and sudden braking and acceleration. Comfort indicators are also reflected in the smoothness of lateral acceleration and the limitation of longitudinal acceleration. The study indirectly ensures this by limiting the range of motion and adding a lateral smoothing term (such as the negative norm of lateral acceleration change) to the reward.

[0147] In multi-agent systems, collaboration, along with other factors, encourages positive interactions between intelligent vehicles. When a vehicle yields to help another vehicle safely merge, a positive reward can be provided to the yielding vehicle to reflect the benefits of cooperation. This collaborative mechanism can be achieved by sharing partial rewards or designing localized rewards.

[0148] In some embodiments, the processor employs a local multi-objective reward function, where the reward for each agent is composed of weighted factors such as safety, efficiency, and comfort. Safety and cooperation-related events (such as collisions) affect the rewards of all relevant vehicles, prompting agents to learn to cooperate and avoid collectively disadvantageous situations.

[0149] In some embodiments, the expression for instant reward is:

[0150] ;

[0151] in, Indicates vehicle At any moment The immediate rewards obtained can be used as optimization targets for RL (Research and Development). , , , and This represents the weighting coefficient for each reward / penalty component, used to measure the importance of different indicators. This indicates a reward for driving efficiency. This indicates a cooperation reward; the closer to the target lane / the higher the reward upon successful lane change, and it is used to guide vehicles to complete lane changes within a specified distance. This indicates a negative reward for driving safety, suggesting that potential collisions or sudden braking should be mitigated in advance. This indicates a penalty for changing lanes, the negative consequences of excessive or frequent lane changes. This indicates acceleration penalty and ride comfort index.

[0152] In some embodiments, all agents share a joint reinforcement learning policy network, whose input can be a global state or their respective locally labeled states, while the research chooses to utilize the embedded information of each node after processing by a graph neural network. As inputs to each agent, this is equivalent to a distributed strategy with shared parameters. The study uses a multi-agent A2C (Advantage Actor-Critic) algorithm as the learning mechanism: during the centralized training phase, a global "critic" network evaluates the overall value using the states and actions of all agents, while each agent outputs actions based on its own state distribution using a shared "actor" network. Specifically, the study connects the output of the GNN to a multilayer perceptron, with parameters... Represents the policy of each agent And a commentator's network parameters Estimate global state value During training, the policy gradient loss and value function loss are updated by minimizing the advantage function-based policy gradient loss. .

[0153] In some embodiments, the expression for policy gradient loss can be:

[0154] ;

[0155] ;

[0156] in, Indicates to The gradient operator is used for optimization. This represents the policy objective function, which maximizes the value of the object. Indicating compliance strategy The expected trajectory Representing the log-likelihood facilitates gradient calculation. Represents intelligent agents At any moment The action, Represents intelligent agents The observations are obtained through graph attention convolution. The advantage function represents the quality of an action. Represents the action value function. Indicates the global / centralized state of the environment. This represents the state-value function, reducing variance.

[0157] In some embodiments, the expression for the value function loss can be:

[0158] ;

[0159] in, This represents the value function loss, which measures the error in the critic's estimation of the state's value. This represents the state value estimate output by the value network. Indicates time The environmental conditions, Indicates the target return, for Estimates of real returns This indicates the critic's parameters.

[0160] Through centralized training, agents can coordinately learn their respective policies to maximize a certain joint reward (e.g., the sum of rewards from all agents). During execution, each intelligent vehicle selects actions solely based on its own observations and local policies, without the need for central coordination, thus achieving decentralized autonomous control. Notably, the study introduces mechanisms such as experience replay during training to improve sample efficiency and policy exploration. Furthermore, to handle training with a variable number of vehicles, the study sets a maximum number of vehicles in the environment. The network output of invalid vehicles is filtered out using a mask vector. This fixes the input and output dimensions of the GNN-Actor network (by zero-padding to...). This ensures successful training under varying numbers of vehicles. In practice, this graph-based representation and masking process makes the policy robust to vehicle additions and subtractions, achieving, to some extent, zero-shot generalization to traffic flows of different sizes.

[0161] In some embodiments, such as Figure 4 As shown, the centralized training-distributed execution (CTDE) process for graph neural network fusion is presented. The input consists of "global state + agent local observations". First, the GNN extracts features from the traffic scene graph to obtain the node embeddings of each vehicle. Subsequently, candidate actions are generated using an MLP and a greedy exploration strategy. To adapt to a variable number of vehicles, missing positions are first zero-padded to unify the dimension, and invalid vehicle entries are masked using a mask mechanism. On the policy side, all vehicles share the same Actor network, and each agent independently selects actions based on its own observations, achieving distributed execution. On the value evaluation side, the global Critic network uses global information to estimate the joint value and calculates the advantage function accordingly. During training, combined with experience replay, the policy gradient loss and value function loss are minimized respectively. Joint updates are performed; the optimization objective is to maximize joint rewards, with immediate rewards weighted by factors such as driving safety, driving efficiency, passenger comfort, and cooperation. After repeated interactions and parameter updates, the model converges; during the execution phase, it relies solely on the current observations and GNN encoding, with the Actor outputting stable lane-changing and longitudinal control actions, completing the end-to-end perception-decision closed loop.

[0162] S3: Using a trained graph neural network fusion model, analyze the graph structure of the traffic scene to obtain multi-agent decision-making strategies.

[0163] Multi-agent decision-making strategy is a decision-making strategy that controls and plans the movement of all agents.

[0164] In some embodiments, the processor can embed a trained GNN fusion module into the onboard decision-making system of each intelligent vehicle, enabling it to transform perceived neighbor vehicle information into high-level feature embeddings in real time. This is equivalent to equipping the vehicle with an "interactive perception front-end" that can dynamically update the strategy input based on the current traffic map.

[0165] S4: Perform decision prior fusion on the multi-agent decision-making strategy, dynamically interact with the environment, obtain the vehicle lane-changing planning result, and complete the vehicle lane-changing planning.

[0166] The lane-changing planning result is the planning and control result of lane changing for multi-agent vehicles.

[0167] In some embodiments, during policy execution, the study retains necessary safety rules as underlying constraints to prevent accidental unsafe actions by the RL policy. For example, a safety monitoring module is set up to trigger collision avoidance actions (such as emergency braking or lane changing restrictions) when the policy output is detected as potentially causing a collision. These rules essentially integrate human experience as decision-making priors with the RL policy, ensuring that the decision outcome meets basic safety requirements. Furthermore, the study utilizes reward functions designed during training to implicitly reflect priors; for example, high collision penalties and lane-changing penalties essentially embed priors such as "avoiding collisions and minimizing lane changes" into the policy optimization objective.

[0168] In some embodiments, the strategy can respond to changes in the environment and interactive relationships. For example, when a vehicle in an adjacent lane suddenly decelerates, that vehicle node transmits information to its own node via the GNN, enabling the vehicle's strategy to perceive the slowing down of the lane ahead and decide whether to change lanes to overtake. Similarly, when two autonomous vehicles are about to simultaneously change lanes into the same lane, through vehicle-to-vehicle communication and GNN interaction, they can anticipate each other's intentions, allowing one vehicle to voluntarily wait to avoid competition. It can be said that the GNN+MARL strategy itself achieves implicit dynamic interactive coordination: during joint training, the agent learns to adjust its own strategy based on the state of other vehicles to maximize long-term rewards; this behavior manifests as high interactive adaptability during execution. For example, according to literature reports, multi-vehicle RL models based on GCN fusion significantly reduce lane-changing conflicts; when yielding is required, the intelligent vehicle will proactively slow down to give way, thus avoiding the deadlock and conflicts that may occur based on fixed rules.

[0169] In some embodiments, the continuous actions output by the strategy need to be mapped to control signals executable by the vehicle. In the method studied, and These can be directly considered as the desired lateral and longitudinal accelerations. If the vehicle control uses a low-level PID closed-loop system, then... As a set value, it is fed into the cruise controller for tracking. The acceleration command is converted into an equivalent steering wheel angle or yaw rate command for the steering controller to track. If an open control approach is used, the acceleration command is converted into throttle, brake, and steering angle commands via a model and sent to the vehicle for execution. In this implementation, since the study focuses on controlling the vehicle in a high-fidelity simulation environment (CarSim / Carla), the vehicle's dynamics are accurately simulated by the simulator; therefore, the strategy output can be directly used as the control input. Simultaneously, the study performs smoothing filtering on the motion (e.g., averaging the acceleration over consecutive frames) to avoid high-frequency jitter, thereby ensuring the smoothness of the trajectory.

[0170] In some embodiments, the processor can perform decision prior fusion on the multi-agent decision-making strategy, and trigger a collision avoidance action when it detects that the strategy output may lead to a collision; based on the current observed state of the agents, the processor uses a trained graph neural network fusion model to update the multi-agent decision-making strategy, dynamically interacts with the environment, obtains the vehicle lane-changing planning result, and completes the vehicle lane-changing planning.

[0171] In some embodiments, such as Figure 5 As shown, to verify the effectiveness of the lane-change replanning control method based on GNN and MARL, a simulation experiment of a highly dynamic mixed traffic scenario was designed, and the proposed method was compared with the traditional MPC control benchmark. The experiment was conducted on two simulation platforms: one was a joint platform of vehicle dynamics (CarSim) simulation and traffic flow (SUMO) simulation, where CAV represents the assisting agent and HDV represents the non-cooperative agent. The figure shows a test road with a three-lane straight section and a right-side ramp, used to simulate high-fidelity vehicle motion and large-scale traffic flow; the other was the open-source autonomous driving simulator Carla combined with ROS bridging, used to verify the algorithm's performance in a realistic 3D environment. The results from the two platforms were consistent, and the following description is mainly based on the former. Figure 5The diagram illustrates a co-simulation scenario and platform used to verify the effectiveness of the method in this invention: The background traffic flow is generated by the SUMO traffic flow engine and runs synchronously with the CarSim vehicle dynamics environment, forming a "CarSim+SUMO" co-simulation platform; the test road is a three-lane main road with a right-side ramp merging in, and speed limit control and traffic density can be configured according to the scenario. In the diagram, cooperative connected vehicles (CAVs) and human-driven vehicles (HDVs) are mixed. The CAVs run the centralized training and decentralized execution strategy fused by the GNN, while the HDVs are generated by the traffic flow model as non-cooperative entities; the system advances according to simulation steps, exchanges vehicle status and decision information in real time, and records indicators such as lane change success rate, collision rate, average vehicle speed, and ride comfort for comparison and evaluation with the traditional MPC baseline.

[0172] In some embodiments, key experimental metrics include: Lane change success rate: the proportion of vehicles intending to change / merge lanes that successfully complete the required lane change maneuver. A failure is considered if a vehicle misses an exit or fails to merge into the target lane in time. Collision rate: the number of collisions per kilometer of vehicle travel. Reflects safety performance. Average speed improvement: the percentage increase in average AV speed relative to the initial state or relative to the average traffic speed after adopting the strategy, measuring traffic efficiency. Passenger comfort: measured by changes in vehicle acceleration and lateral acceleration. Trajectory tracking error: for vehicles performing lane changes, the study records the deviation between their actual trajectory and the ideal reference trajectory. For example, for the MPC scheme, the reference trajectory is a polynomial programming curve; for the RL scheme, the study uses the centerline of the target lane after the lane change as a reference to calculate the change in the lateral offset of the vehicle's center of mass relative to the lane center over time. The maximum offset and steady-state offset are used to quantify the trajectory tracking error. In addition, the study also statistically analyzes secondary metrics such as the average time per lane change for intelligent vehicles and average fuel consumption (estimated from speed and acceleration) to analyze passenger experience and energy consumption performance.

[0173] In some embodiments, as shown in Table 1, the experimental results show that the proposed GNN+MARL method outperforms the traditional MPC scheme in all indicators, especially in complex scenarios with high density and mixed driving styles; where the improvement is equal to (GNN+MARL−MPC) / MPC; a negative value indicates a decrease (better).

[0174] Table 1 Comparative Experiment

[0175]

[0176] In terms of safety: the researched method resulted in almost zero collisions in all simulations: only one minor scratch occurred in 100 experiments (the AV made a slight contact with a neighboring vehicle while changing lanes in extreme congestion, without substantial damage), a collision rate far lower than the MPC scheme's 0.72 collisions / km. Conversely, the MPC scheme experienced multiple collisions or near-misses due to high-density and aggressive driving scenarios. This demonstrates that the reinforcement learning strategy effectively learned safe avoidance through training, enabling it to delay or abandon lane changes in confined spaces to avoid accidents; while traditional MPC, due to the rigidity of its pre-planned trajectory, is prone to uncontrollable situations when the environment suddenly changes (such as a vehicle in the target lane braking suddenly).

[0177] In terms of efficiency, the GNN+MARL strategy significantly improves traffic flow efficiency. The average speed of AVs using this strategy is about 15% to 20% higher than that of the MPC scheme (the improvement is smaller in low to medium traffic flow, but more significant in high traffic flow due to the RL strategy's ability to find gaps for overtaking). Specifically, in a medium traffic flow scenario with aggressive behavior, the average speed of AVs controlled by the RL strategy reaches 33.5 m / s, while the MPC scheme achieves 28.1 m / s, an improvement of about 19%. In high traffic flow scenarios, even with overall traffic congestion, the RL strategy still improves convoy efficiency through cooperation. For example, multiple AVs can form a "relay overtaking" mechanism: the AV in front accelerates after overtaking, helping the following vehicles to complete the overtaking in sequence, reducing the frequent braking caused by individual vehicles changing lanes independently. In contrast, in traditional schemes, AVs often act independently, and overtaking can easily trigger a chain reaction that leads to a decrease in overall speed.

[0178] Trajectory Smoothness and Comfort: Although the RL strategy does not explicitly plan a polynomial trajectory, the learned strategy still ensures a smooth trajectory transition because comfort factors are considered in the reward. Statistics show that the average peak lateral acceleration of the RL strategy vehicle during lane changes is 2.1 m / s², lower than the 2.5 m / s² of the MPC scheme; the average lateral jerk is reduced by about 30%. Passenger subjective comfort is equivalent to reducing sudden changes in direction and abrupt acceleration / deceleration. Regarding trajectory deviation, the final steady state of lane-changing vehicles can return to the vicinity of the target lane center. The average maximum lateral deviation (relative to the lane center) of the RL strategy is 0.12 m, while that of the MPC scheme is 0.15 m. The two are similar in error range, but the deviation change of the RL strategy is smoother, without obvious oscillations. Especially in the case of interfering vehicles (such as a slow vehicle suddenly appearing in front of the target lane, requiring mid-course trajectory adjustment), the RL strategy shows good robustness. The vehicle can actively slow down and wait for the opportunity to complete the lane change, avoiding the drastic control adjustments that occur in the MPC scheme due to forcibly changing lanes according to the predetermined trajectory.

[0179] Cooperative Behavior: Through multi-agent training, AVs learned some implicit cooperative strategies. Simulations observed that when two AVs approach simultaneously merging into the same lane, the AV behind will spontaneously decelerate and accelerate to merge only after the preceding vehicle has completed its lane change, avoiding parallel competition. This courteous behavior did not occur in the MPC scheme: in the MPC scheme, two vehicles sometimes become parallel due to simultaneous lane changes, even triggering the safety mechanism of one vehicle to interrupt the lane change. Furthermore, at ramp merging entrances, vehicles in the RL scheme will proactively move to adjacent lanes to make room for merging vehicles, improving overall traffic efficiency and safety. These cooperative effects have also been reported in the literature: MARL-based vehicles can cooperatively adapt to the uncertain behavior of human vehicles, jointly improving queuing capacity.

[0180] This method achieves excellent results in safety (zero collisions), efficiency (higher vehicle speeds and traffic flow), and comfort (lower acceleration and deceleration impacts). In contrast, while traditional MPC schemes can maintain high trajectory tracking accuracy, they lack global coordination and adaptability in complex scenarios, exhibiting significant disadvantages. This fully validates the value of introducing graph neural networks and multi-agent reinforcement learning for lane-changing control.

[0181] To further analyze the effectiveness of the strategy, the study dissects the results under different scenarios and parameters. First, as traffic flow density increases, the performance gap between the two methods widens: at low density, both methods are almost collision-free and have similar efficiency, but at high density, the traditional method experiences frequent collisions and congestion, while the RL strategy remains robust. This indicates that the studied strategy has better scalability and can still function normally in more complex environments. Second, the impact of different HDV driving styles on the results shows that aggressive human driving increases accident risk and strategy difficulty, but the RL agent can partially adapt to this unpredictability through training (collisions increase slightly but are far less than in the MPC scheme), demonstrating the robustness of the learned strategy. Third, the study examines the impact of reward weights on policy behavior: by assigning safety weights... and comfort weight Increasing the efficiency weight will result in more conservative lane-changing behavior (collisions approach zero but efficiency is slightly reduced); conversely, increasing the efficiency weight will result in more conservative lane-changing behavior. This leads to more aggressive overtaking strategies (increased average speed but more emergency situations). This illustrates the balancing role of multi-objective rewards in decision-making. The default weight selection in this study achieves a balance between safety and efficiency, but in practical applications, it can be adjusted according to needs.

[0182] Finally, from a computational performance perspective, since policy execution only requires forward computation of a single neural network (including a GNN and several fully connected layers), it can run in real time on an onboard computing unit (the study tested on the NVIDIA AGX platform, and the decision time per cycle was <). In contrast, MPC requires solving QP optimizations at each step, and the computation time increases with the number of vehicles, making it difficult to meet real-time requirements in high-density scenarios. Therefore, the method researched also has advantages in terms of real-time performance and scalability.

[0183] In some embodiments of this specification, a vehicle lane-changing planning method based on graph neural networks and multi-agent systems is provided. This method utilizes a multi-agent reinforcement learning algorithm and a corresponding reward function to train a graph neural network fusion model. The parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss based on the advantage function and the value function loss, resulting in a trained graph neural network fusion model. Using this trained model, the traffic scene graph structure is analyzed to obtain a multi-agent decision strategy. The decision strategy is then fused with prior decisions and dynamically interacted with the environment to obtain the vehicle lane-changing planning result, thus completing the vehicle lane-changing planning. By directly outputting control variables through the policy network, the system structure is significantly simplified. Furthermore, relying on the self-learning capability of reinforcement learning and the powerful representational capability of graph networks, the method can automatically adjust decisions for different traffic scenarios, achieving adaptive lane-changing replanning control.

Claims

1. A vehicle lane-changing planning method based on graph neural networks and multi-agent systems, characterized in that, include: S1: Obtain the traffic scene graph structure with multiple agents; S2: Using a multi-agent reinforcement learning algorithm and corresponding reward function, train the graph neural network fusion model, and update the parameters of the graph neural network fusion model by minimizing the policy gradient loss and value function loss based on the advantage function, to obtain the trained graph neural network fusion model; S2 includes: Obtain the environmental state of the traffic scene graph structure; whereby the environmental state includes the observation data of each agent; Using the aforementioned graph neural network fusion model, the environmental state of the traffic scene graph structure is analyzed to obtain the agent's action vector: ； in, Represents the action vector of the agent. Indicates the desired lateral acceleration. This represents the desired longitudinal acceleration; The environment after each agent executes the corresponding action vector is taken as the environment state at the next moment, and the instantaneous reward of the graph neural network fusion model is obtained by using the reward function. The immediate rewards include driving safety, driving efficiency, passenger comfort, and cooperation factors. The expression for the immediate rewards is: ； in, Indicates vehicle At any moment Instant rewards received , , , and This represents the weighting coefficient for each reward / penalty component, used to measure the importance of different indicators. This indicates a reward for driving efficiency. This indicates a cooperation reward; the closer to the target lane / the higher the reward upon successful lane change, and it is used to guide vehicles to complete lane changes within a specified distance. This indicates a negative reward for safe driving. This indicates a penalty for changing lanes, the negative consequences of excessive or frequent lane changes. Indicates acceleration penalty and ride comfort index; A multi-agent reinforcement learning algorithm is used to train each agent in a centralized manner and execute them in a distributed manner. A critic network is used to evaluate the overall value of the state and actions of all agents, and the critic network parameters and global state value are obtained. The output actions of each agent based on its own state distribution are obtained by using an actor network. Based on the immediate reward, the critic network parameters and the global state value, the parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss and value function loss based on the advantage function, and the trained graph neural network fusion model is obtained. S3: Using a trained graph neural network fusion model, analyze the graph structure of the traffic scene to obtain multi-agent decision-making strategies; S4: The multi-agent decision-making strategy is fused with prior decisions, dynamically interacted with the environment, and the vehicle lane-changing planning result is obtained, thus completing the vehicle lane-changing planning; including: The decision-making strategies of the multi-agent agents are fused with prior decisions. When a collision is detected that the policy output will lead to a collision, a collision avoidance action is triggered. Based on the current observed state of the agents, the multi-agent decision-making strategy is updated using a trained graph neural network fusion model, and the system interacts dynamically with the environment to obtain the vehicle lane-changing planning result and complete the vehicle lane-changing planning.

2. The vehicle lane-changing planning method based on graph neural networks and multi-agent systems according to claim 1, characterized in that, S1 includes: Based on the state observed by multiple agents, the initial feature vectors of the multiple agents are extracted; Node features are constructed using the initial feature vector and neighbor information; Edge features are constructed using the relative position and relative velocity between two nodes; Based on the node features and edge features, a traffic scene graph structure is constructed.

3. The vehicle lane-changing planning method based on graph neural networks and multi-agent systems according to claim 1, characterized in that, The graph neural network fusion model includes: The input feature layer is used to extract features from the graph structure to obtain the vehicle's own and relative information features; A linear embedding layer is used to unify and pre-normalize the dimensionality of the vehicle's own and relative information features to obtain normalized features. The first attention convolutional layer is used to perform a one-hop interaction on the normalized features using four-head attention to obtain the first interaction result. The second attention convolutional layer is used to fuse two-hop neighbor information based on normalized features to obtain the second interaction result; Residual convolutional layers are used to enable normalized features as needed based on the simulation scale, resulting in enabled data. The readout-stitching layer is used to process the first interaction result, the second interaction result, and the activation data using a multilayer perceptron to obtain the policy input data; The actor-critic network layer, consisting of an actor network and a critic network, is used to perform fully connected processing on the policy input data to obtain vehicle lane-changing planning instructions.

4. The vehicle lane-changing planning method based on graph neural networks and multi-agent systems according to claim 3, characterized in that, The expression for the first interaction result is: ； ； ； ； in, Indicates the first Layer nodes The hidden state is a representation of the original features of a node after multi-hop neighborhood interactions. Represents a non-linear activation function. Indicates the first The layer is the total message vector obtained by aggregating all neighbors. Indicates the first Layer training bias vectors, Indicates the first From the neighbor Incoming one-sided message, Represents a node Let j and k represent the set of neighboring nodes in the set. Index in Indicates attention weights. Indicates the first A layer-shared linear mapping matrix is ​​used for feature transformation. Represents a node In the hidden state of the previous layer, express function, This represents the training attention vector, used to score the concatenated vector, where T represents the vector transpose. This represents the natural exponential function.

5. The vehicle lane-changing planning method based on graph neural networks and multi-agent systems according to claim 1, characterized in that, The expression for the policy gradient loss is: ； ； in, Indicates to The gradient operator is used for optimization. This represents the policy objective function, which maximizes the value of the object. Indicating compliance strategy The expected trajectory Representing the log-likelihood facilitates gradient calculation. Represents intelligent agents At any moment The action, Represents intelligent agents The observations are obtained through graph attention convolution. The advantage function represents the quality of an action. Represents the action value function. Indicates the global / centralized state of the environment. This represents the state-value function and reduces variance.

6. The vehicle lane-changing planning method based on graph neural networks and multi-agent systems according to claim 1, characterized in that, The expression for the value function loss is: ； in, This represents the value function loss, which measures the error in the critic's estimation of the state's value. This represents the state value estimate output by the value network. Indicates time The environmental conditions, Indicates the target return, for Estimates of true returns This indicates the critic's parameters.

7. A vehicle lane-changing planning system based on graph neural networks and multi-agent systems, used to execute the vehicle lane-changing planning method based on graph neural networks and multi-agent systems as described in any one of claims 1 to 6, characterized in that, include: The perception module is used to acquire state information of the intelligent agent and surrounding traffic participants, and to construct a traffic scene graph structure. The decision module is used to extract and encode features from the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and use multi-agent reinforcement learning decision strategies to output multi-agent decision strategies. The vehicle control module is used to perform decision prior fusion on the decision-making strategies of the multi-agent system, dynamically interact with the environment based on the current observed state of the agents, obtain the vehicle lane-changing planning result, and complete the vehicle lane-changing planning.

Images