Vehicle lane changing planning method and system based on graph neural network and multiple agents

Abstract

The invention provides a vehicle lane changing planning method and system based on a graph neural network and multiple agents, and relates to the technical field of unmanned driving, and the method comprises the steps: obtaining a traffic scene graph structure with multiple agents; using a multi-agent reinforcement learning algorithm and a corresponding reward function to train the graph neural network fusion model, and updating parameters of the graph neural network fusion model by minimizing dominant function-based strategy gradient loss and value function loss to obtain a trained graph neural network fusion model; analyzing the traffic scene graph structure by using the trained graph neural network fusion model to obtain a multi-agent decision strategy; and performing decision-making prior fusion on the multi-agent decision-making strategy, performing dynamic interaction with the environment, obtaining a vehicle lane changing planning result, and completing the vehicle lane changing planning. According to the method, the problems of low multi-agent trajectory tracking precision and poor robustness in a complex scene are solved.

Description

Technical Field

[0001] This specification relates to the field of unmanned driving technology, and in particular to a vehicle lane change planning method and system based on graph neural networks and multi-agents. Background Art

[0002] Lane-changing decision-making and control for autonomous vehicles is a key challenge in intelligent transportation systems, significantly impacting driving safety and traffic efficiency. During lane changes, autonomous vehicles must interact in real time with surrounding human-driven vehicles in a high-speed, dynamic environment, managing complex uncertainties to achieve safe, efficient, and smooth trajectory changes. However, traditional decision-making and control methods face numerous limitations in addressing these challenges. Traditional non-data-driven approaches primarily fall into two categories: rule-driven and model-driven. Rule-driven approaches employ manually designed heuristic rules, such as maintaining a safe distance and choosing the right lane to change. Early low-complexity lane-changing algorithms have implemented such rules to achieve lane changes. Model-driven approaches utilize vehicle dynamics models and optimization solutions to achieve trajectory planning and control. For example, some studies pre-set a virtual reference trajectory for each vehicle and consider the trajectories of other vehicles to plan a safe lane-changing path. Other work models the lane-changing planning problem as a coupled longitudinal and lateral predictive control optimization, solving for the optimal trajectory via quadratic programming (QP). These MPC (model predictive control) or optimization methods offer high trajectory tracking accuracy in structured environments. However, due to the complexity and variability of real-world traffic scenarios, artificial rules and optimization objective functions cannot fully capture all possible scenarios. When faced with unmodeled, uncertain driving behaviors, traditional methods often struggle to make timely and effective optimal decisions. Especially in mixed traffic scenarios, where human driver behavior is random and heterogeneous, control strategies that rely on fixed rules or models struggle to ensure robustness and versatility. Furthermore, traditional planning-control schemes typically require replanning trajectories and solving optimization problems at each decision cycle, resulting in high computational overhead and limited real-time performance. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the vehicle lane change planning method and system based on graph neural network and multi-agent provided by the present invention solves the problems of low multi-agent trajectory tracking accuracy and poor robustness in complex scenarios.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a vehicle lane change planning method based on graph neural network and multi-agent, comprising: S1: Obtaining a traffic scene graph structure with multiple agents; S2: Use the multi-agent reinforcement learning algorithm and the corresponding reward function to train the graph neural network fusion model. Update the parameters of the graph neural network fusion model by minimizing the policy gradient loss based on the advantage function and the value function loss to obtain the trained graph neural network fusion model. S3: Use the trained graph neural network fusion model to analyze the traffic scene graph structure and obtain the multi-agent decision-making strategy; S4: Perform decision prior fusion on the multi-agent decision strategy, dynamically interact with the environment, obtain the vehicle lane change planning result, and complete the vehicle lane change planning.

[0005] The beneficial effects of the present invention are as follows: the present invention provides a vehicle lane change planning method based on graph neural network and multi-agent, using a multi-agent reinforcement learning algorithm and a corresponding reward function to train the graph neural network fusion model, and by minimizing the policy gradient loss and value function loss based on the advantage function to update the parameters of the graph neural network fusion model, a trained graph neural network fusion model is obtained, and the trained graph neural network fusion model is used to analyze the traffic scene graph structure and obtain a multi-agent decision strategy; the multi-agent decision strategy is subjected to decision prior fusion, and dynamically interacts with the environment to obtain a vehicle lane change planning result, thereby completing the vehicle lane change planning. (1) The executable control quantity is directly output from the policy network, reducing the multi-level coupling of "rule base / trajectory optimization + tracking control", reducing engineering complexity and delay propagation error, and facilitating real vehicle deployment and maintenance. (2) The graph neural network explicitly encodes the multi-vehicle interaction and topological relationship, and can adaptively adjust the strategy in dynamic mixed traffic in conjunction with multi-agent reinforcement learning, solving the problem that traditional methods are insufficiently robust to "unmodeled human driving behavior". (3) The collaborative strategy is learned using global information during the training phase, and only relies on the available observations of the vehicle itself during the execution phase, ensuring both collaborative capabilities and meeting the realistic constraints of limited vehicle-side resources and communication. (4) The reward function inherently balances safety, efficiency, comfort, and collaboration, avoiding unnecessary lane changes, sudden acceleration and deceleration, and other undesirable driving patterns, thereby improving overall traffic quality. (5) The forward computational complexity of the strategy is linearly related to the vehicle size, and there is no iterative optimization overhead; it can still meet the millisecond-level decision cycle requirements of the vehicle side in high-density traffic. (6) In the joint simulation, compared with the MPC / rule baseline, the collision rate is significantly reduced, the lane change success rate is improved, the average speed is increased, and the acceleration is lower, demonstrating the stability advantage of the method in complex scenarios.

[0006] Furthermore, the S1 includes: Extract the initial feature vector of the multi-agent based on the state observed by the multi-agent; Constructing node features using the initial feature vector and neighbor information; Use the relative position and relative velocity between two nodes to construct edge features; A traffic scene graph structure is constructed based on the node features and the edge features.

[0007] Using "relative position / speed" as edge features and "vehicle and neighboring vehicle status" as node features, a sparse graph is formed. The computational burden grows approximately linearly with the local neighborhood, making it easy to scale to large traffic flows. The graph representation is insensitive to vehicle number and sequence, naturally adapting to topological changes such as variable vehicle numbers and on- and off-ramps, improving generalization. Using "relative speed / relative distance" to characterize potential conflicts and merging gaps is more closely aligned with driving decision-making mechanisms than pure rules or rasterized inputs, improving the analyzability of lane change timing judgments. Neighborhood aggregation mitigates the impact of single-frame noise and missed measurements, and is more robust to sensor occlusion and short-term frame loss.

[0008] Furthermore, the graph neural network fusion model includes: The input feature layer is used to extract features from the graph structure to obtain the vehicle's own and relative information features; A linear embedding layer is used to unify the dimensions of the vehicle itself and relative information features and pre-normalize them to obtain normalized features; The first image attention convolution layer is used to use four-head attention to perform one-hop interaction on the normalized features to obtain the first interaction result; The second graph attention convolution layer is used to perform two-hop neighbor information fusion based on normalized features to obtain the second interaction result; The residual convolution layer is used to enable normalized features as needed depending on the simulation scale to obtain enabled data; A readout-splicing layer is used to process the first interaction result, the second interaction result, and the activation data using a multi-layer perceptron to obtain policy input data; The actor-critic network layer, including the actor network and the critic network, is used to fully connect the policy input data to obtain the vehicle lane change planning instructions.

[0009] The four-head GAT uses one hop first and two hops later, focusing on key neighboring vehicles while also obtaining priors about "neighbors' neighbors" to provide early insight into potential conflict chains. It unifies dimensions and weights key information, improving numerical stability and representation efficiency during the message passing phase. It mitigates oversmoothing and gradient degradation in high-density / deep propagation scenarios, stabilizing training and improving deep representation. It compresses local topological embeddings into compact policy inputs, and its continuous action outputs enable smoother lateral and longitudinal control, outperforming coarse-grained switches with discrete actions.

[0010] Furthermore, the expression of the first interaction result is: ; ; ; ; in, Indicates the Layer Node The hidden state is the representation of the original features of the node after multi-hop neighborhood interaction. represents a nonlinear activation function, Indicates the The total message vector obtained by aggregating all neighbors at the layer, Indicates the Layer training bias vector, Indicates the Layer from neighbors Incoming unilateral messages, Representation node The neighbor node set, j, k represent the neighbor nodes in the set The index in represents the attention weight, Indicates the Layer-shared linear mapping matrix for feature transformation, Representation node In the hidden state of the previous layer, express function, Represents the trained attention vector, which is used to score the concatenated vector. T represents the vector transpose. represents the natural exponential function.

[0011] Attention weights dynamically allocate information channels based on the scenario, amplifying key neighboring vehicles and suppressing redundant nodes to avoid information dilution caused by averaging. Attention weight visualization facilitates offline verification of "strategy focus points," facilitating security reviews and parameter adjustments. The combination of concatenated scoring and shared linear mapping matrices balances expressiveness and parameter sharing, resulting in more stable training and avoiding overfitting and drift caused by graph depth.

[0012] Furthermore, the S2 includes: Obtaining the environmental state of the traffic scene graph structure; wherein the environmental state includes the observation data of each intelligent agent; Using the graph neural network fusion model, we analyze the environmental state of the traffic scene graph structure and obtain the action vector of the agent: ; in, represents the action vector of the agent, represents the expected lateral acceleration, represents the expected longitudinal acceleration; The environment after each agent performs the corresponding action vector is taken as the environment state at the next time, and the immediate reward of the graph neural network fusion model is obtained by using the reward function; The multi-agent reinforcement learning algorithm is used for centralized training and decentralized execution of each agent, the critic network is used to evaluate the overall value of the state and action of all agents, and the critic network parameters and global state value are obtained; The actor network is used to obtain the output action of each agent based on the state distribution of the agent, and based on the immediate reward, critic network parameters and global state value, the parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss and value function loss based on the advantage function, to obtain the trained graph neural network fusion model.

[0013] Directly output the expected lateral / longitudinal acceleration, reduce the lane-changing trajectory discontinuity and high-frequency jitter, and improve the ride comfort. The variable number of vehicles is unified in the network dimension by padding / masking, improving the sample utilization rate and generalization. The global critic evaluates the joint value, alleviates the "non-stationary environment" problem of multi-agent learning, and accelerates the convergence. In joint simulation, the lane-changing success rate is higher, the average speed is improved, and the fuel consumption is lower.

[0014] Further, the immediate reward includes driving safety, driving efficiency, passenger comfort and cooperation factors, and the expression of the immediate reward is: ; Wherein, represents the vehicle at time The immediate reward obtained can be used as an RL optimization target, 、 、 、 and represents the weighting coefficient of each reward / penalty component, which is used to measure the importance of different indicators, represents the driving efficiency reward, represents the cooperation reward, which is higher when it is closer to the target lane / lane-changing completion, and is used to guide the vehicle to complete lane-changing within a specified distance, represents the driving safety negative reward, which can suppress possible collisions or sudden braking in advance, represents the lane-changing penalty, which is the negative cost of excessive or frequent lane-changing, represents the acceleration penalty, which is the ride comfort indicator.

[0015] The weighting factors for safety, efficiency, comfort, and collaboration can be quickly adjusted based on the application scenario, supporting conservative / aggressive style switching and scenario migration. Penalties for frequent lane changes and high acceleration significantly reduce unnecessary lane changes, and rapid acceleration, deceleration, and steering are limited, significantly reducing vehicle sway and sudden braking, resulting in a more stable, smoother, and more comfortable ride. The implementation demonstrates improved safety (fewer collisions), increased efficiency (higher speed and arrival rate), smoother ride, and greater fuel and power efficiency.

[0016] Furthermore, the expression of the policy gradient loss is: ; ; in, Express The gradient operator for optimization, represents the policy objective function, maximization object, Obey the strategy The trajectory expectation, represents the log-likelihood, which is convenient for gradient calculation, Representing an agent At the moment action, Representing an agent The observation of is obtained by graph attention convolution, represents the advantage function, which measures the quality of an action. represents the action-value function, Represents the global / centralized state of the environment, Represents the state-value function and reduces variance.

[0017] The dominant baseline significantly reduces policy gradient variance, resulting in more stable training and higher sample efficiency. A centralized critic provides more accurate joint value assessments, alleviating the difficulty of credit allocation in multi-vehicle coordinated behavior and encouraging beneficial collaborative strategies such as yielding and relaying. Stabilizing factors such as entropy regularization can be naturally added to prevent premature maturation and promote exploration.

[0018] Furthermore, the expression of the value function loss is: ; in, represents the value function loss, which measures the error of the critic’s estimation of the state value, represents the state value estimate output by the value network, Indicates time environmental conditions, Indicates the target return, Estimates of real returns, represents the critic parameters.

[0019] Explicit constraints on value errors reduce critic bias, avoid misleading policy, and make overall training more stable. More accurate state values ​​make advantage estimates more reliable, policy updates more "clean," and reduce oscillations and regressions.

[0020] Furthermore, the S4 includes: Performing decision prior fusion on the multi-agent decision strategy, and triggering a collision avoidance action when it is detected that the strategy output may cause a collision; Based on the current observed state of the agent, the trained graph neural network fusion model is used to update the multi-agent decision-making strategy, dynamically interact with the environment, obtain the vehicle lane change planning results, and complete the vehicle lane change planning.

[0021] The collision avoidance / constraint triggering prior module is superimposed on the policy execution layer, forming a dual-security framework of "learning strategy + rule-based support" to meet the requirements of functional safety implementation. When scenarios outside the training distribution or extreme conditions occur, the safety prior can immediately intervene in limiting / braking, significantly reducing the risk of rare catastrophic events. After the safety trigger, the policy can be adaptively modified in the next cycle based on updated observations, ensuring uninterrupted lane changes or graceful abandonment when necessary, with strong stability. The prior and rewards jointly encourage a collaborative "courtesy-merge-recovery" model, reducing conflicts caused by parallel competition.

[0022] The present invention provides a vehicle lane change planning system based on graph neural network and multi-agent, comprising: The perception module is used to obtain the status information of the intelligent agent and surrounding traffic participants and construct the traffic scene graph structure; The decision module is used to extract and encode features from the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and output a multi-agent decision strategy using multi-agent reinforcement learning. The vehicle control module is used to perform decision-making prior fusion on the multi-agent decision-making strategy, dynamically interact with the environment according to the current observed state of the agent, obtain the vehicle lane change planning result, and complete the vehicle lane change planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein: Figure 1 is a module diagram of a vehicle lane change planning system based on a graph neural network and multi-agent according to some embodiments of this specification; Figure 2is an exemplary flow chart of a vehicle lane change planning method based on a graph neural network and multi-agent according to some embodiments of this specification; Figure 3 is an exemplary schematic diagram of a smart vehicle lane change re-planning control framework according to some embodiments of this specification; Figure 4 is an exemplary schematic diagram of a graph neural network fusion model training process according to some embodiments of this specification; Figure 5 This is an exemplary schematic diagram illustrating a traffic flow-vehicle dynamics joint simulation framework and a road scenario according to some embodiments of this specification. DETAILED DESCRIPTION

[0024] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0025] Example 1 Figure 1 This is a module diagram of a vehicle lane change planning system based on graph neural network and multi-agent according to some embodiments of this specification.

[0026] In some embodiments, the vehicle lane change planning system based on graph neural network and multi-agent may include a perception module, a decision module and a vehicle control module.

[0027] The perception module is used to obtain the status information of the intelligent agent and surrounding traffic participants and construct the traffic scene graph structure.

[0028] The decision module is used to extract and encode features of the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and output a multi-agent decision strategy using multi-agent reinforcement learning.

[0029] The vehicle control module is used to perform decision-making prior fusion on the multi-agent decision-making strategy, dynamically interact with the environment according to the current observed state of the agent, obtain the vehicle lane change planning result, and complete the vehicle lane change planning.

[0030] In some embodiments, as Figure 3The figure shows the perception-decision-control closed loop of the present invention. From left to right, it consists of: on-board sensors and communications (cameras, millimeter-wave radar, lidar, GPS / IMU, V2X); perception module (time synchronization, calibration, and multi-source fusion to determine the vehicle and neighboring vehicle states); graph neural network processing (organizing the states into a traffic scene graph G = (V, E), with nodes representing vehicles and edges representing interactions / communication relationships, and topological encoding); multi-agent reinforcement learning decision-making (outputting candidate actions: hold / left / right lane change and target longitudinal acceleration / speed); decision module (superimposing safety constraints and prior rules to filter executable instructions); vehicle control module (trajectory generation and tracking, steering, driving / braking), and ultimately execution by the intelligent vehicle. This process operates in an online closed loop with a fixed decision cycle, achieving safe, smooth, and efficient lane changes in dynamic traffic flows. Each intelligent vehicle acquires state information about itself and surrounding traffic participants through onboard sensors (cameras, radar, etc.) and vehicle-to-vehicle communication. This information includes its own speed, acceleration, lane position, and the relative distance and speed of neighboring vehicles. The perception module processes this information and constructs a traffic scene graph G = (V, E), where the nodes V represent each vehicle in the road network, and the edges E represent the interactions between vehicles (such as neighboring vehicle relationships or communication connections). Subsequently, the decision module uses a graph neural network to extract and encode features from the scene graph, mapping the multi-vehicle interaction states into a compact low-dimensional representation. This representation is then fed into a multi-agent reinforcement learning decision strategy to output lane change decisions and control actions for each intelligent vehicle. Specifically, for each intelligent vehicle, the decision module outputs a set of control instructions based on its current observed state (including neighboring vehicle information fused via the GNN), including lateral lane change intentions and longitudinal speed adjustments. Finally, the vehicle control module executes the corresponding lane change and trajectory tracking control based on the decision instructions. For example, when the decision is to change lanes, the control module generates steering and acceleration commands to smoothly move the vehicle toward the target lane; when the decision is to maintain lanes, the control module maintains the current lane. The entire process operates in a real-time closed-loop during each decision cycle, forming a "perception-decision-control" cycle: the intelligent vehicle continuously perceives environmental changes and adaptively replans lane-changing actions based on learning strategies, thereby achieving safe and efficient lane-changing maneuvers in dynamic traffic flows.

[0031] In some embodiments, a vehicle lane change planning system based on graph neural networks and multi-agents can be used to execute a vehicle lane change planning method based on graph neural networks and multi-agents, including: S1: obtaining a traffic scene graph structure with multi-agents; S2: using a multi-agent reinforcement learning algorithm and a corresponding reward function to train the graph neural network fusion model, and updating the parameters of the graph neural network fusion model by minimizing the policy gradient loss and value function loss based on the advantage function to obtain a trained graph neural network fusion model; S3: using the trained graph neural network fusion model to analyze the traffic scene graph structure to obtain a multi-agent decision-making strategy; S4: performing decision prior fusion on the multi-agent decision-making strategy, dynamically interacting with the environment, obtaining a vehicle lane change planning result, and completing the vehicle lane change planning.

[0032] In some embodiments of this specification, a processor utilizes a graph neural network and multi-agent-based vehicle lane change planning system to execute a vehicle lane change planning method. Directly outputting control variables through the policy network significantly simplifies the system architecture. Furthermore, leveraging the self-learning capabilities of reinforcement learning and the powerful representation capabilities of graph networks, this system can automatically adjust decisions for different traffic scenarios, achieving adaptive lane change replanning control.

[0033] Example 2 Figure 2 This is an exemplary flow chart of a vehicle lane change planning method based on graph neural network and multi-agent according to some embodiments of this specification. Figure 2 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0034] S1: Obtain a traffic scene graph structure with multiple agents.

[0035] The traffic scene graph structure is a structural graph that reflects the vehicle information and the relationship between adjacent vehicles in the traffic scene. For example, the traffic scene graph structure can include node features and edge features.

[0036] In some embodiments, the processor can use a graph neural network (GNN) to model traffic scenes to capture the dynamic relationship between vehicles and serve as the perception front end of the reinforcement learning strategy. , where each node Represents a vehicle on the road (including the autonomous vehicle and surrounding manned vehicles), each edge Indicates vehicle and vehicles The construction principle of the edge can be based on the neighborhood radius or lane change correlation: for example, when the vehicle Located in the vehicle When the vehicle is within a certain distance ahead or in an adjacent lane, the research suggests that and Direct interactions exist between vehicles, connecting them as adjacent nodes in the graph; no edge connections are established for distant vehicles that are beyond the perception / communication range. In this way, the graph structure naturally expresses the neighborhood relationships of intelligent vehicles in the local traffic flow, helping to limit the scope of information dissemination and reduce model complexity.

[0037] In some embodiments, the processor can extract the initial feature vector of the multi-agent based on the state observed by the multi-agent; use the initial feature vector and neighbor information to construct node features; use the relative position and relative speed between two nodes to construct edge features; and construct a traffic scene graph structure based on the node features and the edge features.

[0038] The state observed by the agent is the vehicle state information observed by the agent. For example, the state observed by the agent can include the state information of the vehicle and surrounding traffic participants, including its own speed, acceleration, position, lane, and the relative distance and speed of neighboring vehicles.

[0039] The initial feature vector is a vector of features related to the state and attributes of the vehicle itself in the state observed by the agent. For example, the initial feature vector can include information about the vehicle's state and attributes, such as the longitudinal velocity , acceleration , the vehicle's position in the road coordinate system , Current lane number , vehicle type indicator (autonomous vehicle or manned vehicle), etc.

[0040] In some embodiments, the processor can incorporate some neighbor information into node features, for example, encoding the distance and speed difference relative to the vehicle into the features of the neighbor node, and using the relative position and speed as edge features when composing the map. , or merged into the feature description of adjacent nodes.

[0041] Node features are the characteristics of each node in the traffic scene graph structure. For example, node features can include the initial feature vector, neighbor information, distance and speed difference relative to the vehicle, etc.

[0042] Edge features are the features of each edge in the traffic scene graph structure. For example, edge features can include relative position and speed.

[0043] S2: Use the multi-agent reinforcement learning algorithm and the corresponding reward function to train the graph neural network fusion model, and update the parameters of the graph neural network fusion model by minimizing the policy gradient loss and value function loss based on the advantage function to obtain the trained graph neural network fusion model.

[0044] A graph neural network fusion model is a neural network model used to analyze the structure of a traffic scene graph and derive vehicle lane change planning instructions. For example, a graph neural network fusion model can include a graph neural network model and an actor-critic network model.

[0045] In some embodiments, the graph neural network fusion model includes: an input feature layer, a linear embedding layer, a first graph attention convolution layer, a second graph attention convolution layer, a residual convolution layer, a readout-concatenation layer, and an actor-critic network layer.

[0046] In some embodiments, the default parameter settings may include , ReLU is used in all hidden layers, and Dropout=0.1.

[0047] The input feature layer is used to extract features from the graph structure and obtain the vehicle’s own and relative information features.

[0048] The vehicle's own and relative information features are the features related to the vehicle's own state and attributes in the traffic scene graph structure.

[0049] The linear embedding layer is used to unify the dimensions of the vehicle itself and the relative information features and pre-normalize them to obtain normalized features.

[0050] Normalized features are the normalized features of the vehicle itself and relative information.

[0051] In some embodiments, the expression of the normalized feature may be: ; in, represents the normalized features, represents a nonlinear activation function, represents the matrix-vector product, Represents a constant.

[0052] In some embodiments, the linear embedding layer performs scale unification and feature reweighting to provide high-dimensional expression for subsequent message passing.

[0053] The first image attention convolution layer is used to use four-head attention to perform one-hop interaction on the normalized features to obtain the first interaction result.

[0054] The first interaction result is the normalized feature one-hop neighbor information fusion result.

[0055] In some embodiments, the expression of the first interaction result may be: ; ; ; ; wherein, denotes the hidden state of the layer node, is the representation of the node original features after the interaction of the multi-hop neighborhood, denotes a nonlinear activation function, denotes the total message vector aggregated by all neighbors of the layer, denotes the training bias vector of the layer, denotes the unilateral message passed in from the neighbor of the layer node, denotes the set of neighbor nodes of the node , j, k denote the indices of the neighbor nodes in the set , denotes the attention weight, denotes the linear mapping matrix shared by the layer, used for feature transformation, denotes the hidden state of the node in the last layer, denotes function, denotes the trained attention vector used to score the concatenated vector, and T denotes vector transposition, denotes the natural exponential function.

[0056] In some embodiments, the multi-head attention of the graph attention convolution layer enables the node to adaptively focus on key neighbors; after the second layer, the information has covered a two-hop range.

[0057] The second graph attention convolution layer is used to fuse two-hop neighbor information based on normalized features to obtain a second interaction result.

[0058] The second interaction result is the normalized feature two-hop neighbor information fusion result.

[0059] The residual convolution layer is used to enable the normalized feature as needed according to the simulation scale to obtain enabled data.

[0060] The enabled data is the data enabled as needed according to the simulation scale.

[0061] In some embodiments, if the vehicle density in the scene is high or the graph depth is insufficient, a residual GAT layer can be added to alleviate oversmoothing and stabilize the gradient; whether to enable or not is dynamically determined. ; denotes the number of vehicles currently included in the graph.

[0062] In some embodiments, the expression of the enabled data is: ; in, Indicates a layer 3 node The hidden representation of is obtained by one-hop / two-hop attention convolution + activation function and used as the input of the final readout-concatenation layer. represents a nonlinear activation function, Represents the third-layer aggregate message vector, which is obtained by linearly mapping the neighbor messages after summing them according to the attention weights. Represents a layer 2 node The hidden representation of , as a residual or skip term, is After adding activation.

[0063] The readout-splicing layer is used to process the first interaction result, the second interaction result and the enabling data using a multi-layer perceptron to obtain policy input data.

[0064] Policy input data is the observation data of the multi-agent policy network.

[0065] In some embodiments, the processor may represent the final node With neighbors and After concatenation, it is mapped to two layers of MLP , the local synthesis expression , as observations of the multi-agent policy network.

[0066] In some embodiments, the expression of the policy input data may be: ; in, represents the final input feature vector of the policy network, Represents a multi-layer perceptron, used for dimension compression and nonlinear mapping, Representation node After the The final hidden representation after the layer graph attention convolution, Represents neighbor nodes The final hidden representation of .

[0067] The actor-critic network layer, including the actor network and the critic network, is used to fully connect the policy input data to obtain the vehicle lane change planning instructions.

[0068] The vehicle lane change planning instruction is an instruction to control the vehicle to perform lane change planning.

[0069] In some embodiments, the processor may cause Through three-layer full connection (128-256-128), continuous action is obtained or state-value ,in To control the lateral movement, Longitudinal control actions serve as vehicle lane change planning instructions.

[0070] In some embodiments, the graph neural network updates the hidden state of the node through the message passing mechanism, thereby extracting more advanced interaction features layer by layer. Indicates the Nodes in layer GNN The hidden state vector of the 0th layer can be is the initial feature), the typical message passing process can be described as: ; ; in, Indicates unilateral message, neighbor In the The vector passed by the layer, Represents a message function used to generate neighbors Send to News , Representation node In the The hidden state vector of the layer, Represents neighbor nodes The hidden state vector of express The neighboring car, Representation node The set of adjacent nodes, Representation node In the The updated hidden state of the layer, Represents the update function, which is used to update the node's own hidden state after summarizing neighbor messages. and It is usually parameterized by a trainable neural network such as a multilayer perceptron (MLP).

[0071] In some embodiments, a simplification is performed using graph convolution: the message is the sum of the neighbor hidden vectors after linear transformation according to a certain weight, that is: ; in, Indicates the Layer Node The hidden state of represents a nonlinear activation function, Indicates the The weight matrix of the layer, sharing the information conversion between neighbors and itself, Indicates the Layer Node The hidden state of Indicates the Layer Node The hidden state of . This formula is equivalent to the simple case of the classic GCN model assuming equal contribution of neighbors and using degree normalization. In practical applications, researchers can also use graph attention networks (GAT) to adaptively assign neighbor weights or introduce edge features to affect message passing. Regardless of the specific implementation, after the multi-layer GNN iteration, each node obtains the vector This representation encodes the vehicle The state and its neighbor interaction characteristics in the local traffic graph, such as the impact of slow vehicles in front on its speed, the constraints on lane change space imposed by vehicles on the sides and rear, etc. These high-level features will serve as input to the reinforcement learning decision network, providing a graph-structured state representation for subsequent decision-making.

[0072] In some embodiments, the GNN is embedded in the reinforcement learning policy network for joint training, without applying a separate supervisory loss to the GNN. The parameter optimization of the GNN is indirectly driven by the overall goal of reinforcement learning, that is, the output of the GNN must help to correctly estimate the value or Q value of the policy. In other words, the role of the GNN is information fusion: the original state of multiple vehicles is combined into a single state. Mapping to joint state features , so that the reinforcement learning agent can more accurately judge the benefits of each action. In the research implementation, GNN and the subsequent decision network form an end-to-end trainable policy model, which is jointly trained through gradient descent during the reinforcement learning process.

[0073] In some embodiments, the processor can obtain the environmental state of the traffic scene graph structure; wherein the environmental state includes the observation data of each intelligent agent; using the graph neural network fusion model, the environmental state of the traffic scene graph structure is analyzed to obtain the action vector of the intelligent agent; the environment after each intelligent agent executes the corresponding action vector is used as the environmental state at the next moment, and the reward function is used to obtain the immediate reward of the graph neural network fusion model; using the multi-agent reinforcement learning algorithm to centrally train and disperse the execution of each intelligent agent, using the critic network to evaluate the overall value of the state and action of all intelligent agents, and obtain the critic network parameters and global state value; using the actor network to obtain the output action of each intelligent agent based on its own state distribution, based on the immediate reward, critic network parameters and global state value, by minimizing the policy gradient loss based on the advantage function and the value function loss, the parameters of the graph neural network fusion model are updated to obtain a trained graph neural network fusion model.

[0074] The environment state of the traffic scene graph structure is the state information reflecting the vehicle’s own information and observation data. For example, the environment state can include information such as the vehicle’s position and speed as well as the observation data of each agent.

[0075] In some embodiments, the processor may formulate the problem as a Markov Game. Consider a group of intelligent vehicles (i.e., multiple agents), and several non-agent vehicles (such as manned vehicles, which can be considered as part of the environment). At each discrete decision moment , the entire environment can be described as a state , including the location, speed and other information of all vehicles; each agent You can get your own observations , which is composed of the state of itself and neighboring vehicles after GNN fusion. Due to the addition of vehicle-to-vehicle communication and environmental perception, the study assumes that the intelligent agent can obtain a fairly complete local environmental state, that is, it can be approximated as globally observable or partially observable but sufficient to meet decision-making needs. According to its strategy From observation Mapping an action In the lane-changing control setting of the study, the action space can be designed to be continuous or discrete. In order to give full play to the advantages of reinforcement learning for continuous control, the study adopts a continuous action space: the agent Motion vector It consists of two components, the horizontal and the longitudinal, which correspond to lane change steering control and longitudinal acceleration and deceleration control respectively. can be interpreted as the desired lateral acceleration or steering wheel angle command (controlling the lane change amplitude and rate by limiting its amplitude), represents the desired longitudinal acceleration (positive values ​​for acceleration, negative values ​​for deceleration) used to adjust vehicle speed. By outputting continuous actions, the agent can finely control the vehicle's lane changes and following, which work together to achieve smooth trajectory tracking. By contrast, a discrete action space (such as a combination of lane change / no lane change, acceleration / deceleration) could also be used, but this would transform the problem into a mixed decision-making process, potentially requiring a hierarchical approach. This paper chooses continuous control to unify the decisions for lane changes and speed adjustment.

[0076] The action vector of an agent is the vector of action instructions currently executed by the agent.

[0077] In some embodiments, the expression of the agent's action vector can be: ; in, represents the action vector of the agent, represents the expected lateral acceleration, represents the desired longitudinal acceleration.

[0078] In some embodiments, when all agents perform their actions simultaneously, the state of the environment Evolving to the next moment through system dynamics and vehicle interaction In this process, each agent will receive an immediate reward , used to provide feedback on the quality of the action.

[0079] The immediate reward is a reward function that balances the importance of driving efficiency, goal completion, and various penalty factors. For example, the immediate reward can include driving safety, driving efficiency, passenger comfort, and cooperation factors.

[0080] Driving safety means that if the smart car collides with any other vehicle or the distance between them falls below the safety threshold, a huge negative reward will be given. , encouraging the agent to avoid dangerous situations. In normal situations where no collision occurs, a small negative reward can be provided based on indicators such as the minimum vehicle distance to encourage vehicles to maintain a safe distance.

[0081] Driving efficiency is to encourage vehicles to travel at higher speeds and complete tasks such as lane changes and overtaking as quickly as possible. Research defines speed rewards , which is the ratio of vehicle speed to the road speed limit (between 0 and 1), thus encouraging vehicles to drive close to the speed limit. Furthermore, vehicles with specific lane-changing intentions (such as approaching an exit) can be given a goal achievement reward if they complete the goal in a timely manner (for example, merging into the target lane within a specified distance).

[0082] Passenger comfort (Comfort) is to avoid frequent and violent operations, and to study the penalties imposed on excessive acceleration and lane changing. (A certain number of points will be deducted for each lane change) and acceleration penalty (Points are deducted for excessive acceleration changes) to encourage the strategy to maintain lanes when not necessary, reducing unnecessary lane changes and sudden braking and acceleration. Comfort metrics are also reflected in the smoothness of lateral acceleration and the limits on longitudinal acceleration. This is indirectly ensured by limiting the amplitude of movement and adding a lateral smoothness term to the reward (such as the negative norm of the lateral acceleration change).

[0083] Collaboration and other factors are researched to encourage positive interactions between intelligent vehicles in multi-agent systems. When a vehicle yields to help another vehicle merge safely, a positive reward can be provided to the yielding vehicle to reflect the benefits of cooperation. This collaborative mechanism can be achieved through the sharing of partial rewards or the design of local rewards.

[0084] In some embodiments, the processor adopts a local multi-objective reward function, and the reward of each agent is composed of the weighted components of the above-mentioned safety, efficiency, comfort, etc., among which safety and cooperation-related events (such as collisions) will affect the rewards of all related vehicles, prompting the agents to learn to cooperate to avoid collective disadvantageous situations.

[0085] In some embodiments, the expression for the instant reward is: ; in, Indicates vehicle At the moment The immediate reward obtained can be used as the RL optimization target, 、 、 、 and Represents the weighted coefficient of each reward / penalty component, which is used to measure the importance of different indicators. represents the driving efficiency reward, Indicates the collaboration reward. The closer to the target lane / lane change completion, the higher the reward. It is used to guide the vehicle to complete the lane change within the specified distance. Indicates negative rewards for driving safety, which can help prevent possible collisions or sudden braking. represents the lane-changing penalty, the negative cost of excessive or frequent lane changing, Represents acceleration penalty, a ride comfort index.

[0086] In some embodiments, all agents share a joint reinforcement learning policy network, whose input can be the global state or the state with their own local labels, and the study chooses to use the embedding of each node after the graph neural network processing As their respective inputs, this is equivalent to a parameter-sharing distributed strategy. The study uses a multi-agent A2C (Advantage Actor-Critic) algorithm as a learning mechanism: during the centralized training phase, a global "critic" network uses the states and actions of all agents to evaluate the overall value, while each agent uses a shared "actor" network to output actions based on its own state distribution. Specifically, the study connects the output of the GNN to a multi-layer perceptron with parameters Represents the strategy of each agent , and a critic network parameter Estimating the value of global state During training, we update θ by minimizing the policy gradient loss and value function loss based on the advantage function. .

[0087] In some embodiments, the expression of policy gradient loss can be: ; ; in, Express The gradient operator for optimization, represents the policy objective function, maximization object, Obey the strategy The trajectory expectation, represents the log-likelihood, which is convenient for gradient calculation, Representing an agent At the moment action, Representing an agent The observation of is obtained by graph attention convolution, represents the advantage function, which measures the quality of an action. represents the action-value function, Represents the global / centralized state of the environment, Represents the state-value function and reduces variance.

[0088] In some embodiments, the value function loss may be expressed as: ; in, represents the value function loss, which measures the error of the critic’s estimation of the state value, represents the state value estimate output by the value network, Indicates time environmental conditions, Indicates the target return, Estimates of real returns, represents the critic parameters.

[0089] Through centralized training, agents can coordinately learn their own strategies to maximize a certain joint reward (such as the sum of all agent rewards). During execution, each intelligent vehicle selects actions based solely on its own observations and local strategies, without the need for central coordination, thus achieving decentralized autonomous control. It is worth noting that the study introduced mechanisms such as experience replay during the training process to improve sample efficiency and strategy exploration. In addition, to handle training with a variable number of vehicles, the study set a maximum number of vehicles in the environment. Use the mask vector to filter the network output of invalid vehicles. In this way, the input and output dimensions of the GNN-Actor network are fixed (by padding zero to ), ensuring smooth training with varying numbers of vehicles. In practice, this graph-based representation and mask processing makes the strategy robust to the addition or removal of vehicles, achieving a certain degree of zero-shot generalization to traffic flows of varying sizes.

[0090] In some embodiments, as Figure 4As shown in the figure, the centralized training-decentralized execution (CTDE) process of graph neural network fusion is given. The input is "global state + local observation of the agent". First, the GNN extracts features from the traffic scene graph to obtain the node embedding of each vehicle. Then, candidate actions are generated through MLP and greedy exploration strategy. To adapt to the variable number of vehicles, the missing positions are first filled with zeros to unify the dimensions, and the Mask mechanism is used to block invalid vehicle entries. On the policy side, all vehicles share the same Actor network, and each agent independently selects actions based on its own observations to achieve decentralized execution. On the value evaluation side, the global Critic network uses global information to estimate the joint value and calculate the advantage function accordingly. During training, combined with experience replay, the policy gradient loss and value function loss are minimized respectively. Joint updates; the optimization goal is to maximize the joint reward, with the immediate reward being a weighted combination of driving safety, driving efficiency, ride comfort, and collaboration factors. After repeated interactions and parameter updates, the model converges. During the execution phase, the actor outputs stable lane change and longitudinal control actions, relying solely on current observations and GNN encoding, completing the end-to-end perception-decision loop.

[0091] S3: Use the trained graph neural network fusion model to analyze the traffic scene graph structure and obtain the multi-agent decision-making strategy.

[0092] The multi-agent decision-making strategy is a decision-making strategy that controls and plans the movements of all agents.

[0093] In some embodiments, the processor can embed a trained GNN fusion module into each smart car's onboard decision-making system, enabling it to convert perceived neighboring vehicle information into high-level feature embeddings in real time. This is equivalent to equipping the vehicle with an "interactive perception front end" that can dynamically update policy inputs based on the current traffic map.

[0094] S4: Perform decision prior fusion on the multi-agent decision strategy, dynamically interact with the environment, obtain the vehicle lane change planning result, and complete the vehicle lane change planning.

[0095] The vehicle lane-changing planning result is the planning and control result of the lane-changing of multi-agent vehicles.

[0096] In some embodiments, during policy execution, the research retains necessary safety rules as underlying constraints to prevent the RL policy from occasionally taking unsafe actions. For example, a safety monitoring module is implemented to trigger collision avoidance actions (such as emergency braking or lane change restrictions) when it detects that the policy output may cause a collision. These rules are equivalent to integrating human experience as decision priors with the RL policy, ensuring that the decision results meet basic safety requirements. Furthermore, the research utilizes reward functions designed during training to implicitly reflect priors. For example, high collision penalties and lane change penalties actually embed priors such as "avoiding collisions and reducing lane changes" into the policy optimization objective.

[0097] In some embodiments, policies can respond to interactions as the environment changes. For example, when a vehicle in an adjacent lane suddenly decelerates, that vehicle's node transmits information to the host vehicle's node via a GNN, allowing the host vehicle's policy to detect the slowdown in the lane ahead and decide whether to change lanes and overtake. Another example is when two autonomous vehicles nearly simultaneously change lanes. Through inter-vehicle communication and GNN interaction, they can anticipate each other's intentions, allowing one vehicle to proactively wait for a moment to avoid competition. It can be said that the GNN+MARL policy inherently implements implicit dynamic interactive coordination: during joint training, the agent learns to adjust its own policy based on the states of other vehicles to maximize long-term rewards. This behavior manifests as a high degree of interactive adaptability during execution. For example, a multi-vehicle RL model based on GCN fusion has been reported to significantly reduce lane change conflicts. When required to yield, the intelligent vehicle will proactively slow down and yield, thus avoiding the deadlocks and conflicts that can occur with fixed rules.

[0098] In some embodiments, the continuous actions output by the strategy need to be mapped into control signals that can be executed by the vehicle. and It can be directly regarded as the desired lateral acceleration and longitudinal acceleration. If the vehicle control adopts the underlying PID closed loop, then As the set value to be tracked by the cruise controller, This is converted into an equivalent steering wheel angle or yaw rate command for the steering controller to track. If open control is employed, the model converts acceleration commands into throttle, brake, and steering angles, which are then sent to the vehicle for execution. In this implementation, since the vehicle is controlled within a high-fidelity simulation environment (CarSim / Carla), and the vehicle dynamics are accurately simulated by the simulator, the policy output can be used directly as the control input. Furthermore, smoothing filtering is applied to the actions (e.g., averaging the acceleration across consecutive frames) to avoid high-frequency jitter and thus ensure trajectory smoothness.

[0099] In some embodiments, the processor can perform decision prior fusion on the multi-agent decision-making strategy, and trigger a collision avoidance action when it is detected that the strategy output may cause a collision; based on the current observed state of the agent, the multi-agent decision-making strategy is updated using the trained graph neural network fusion model, and dynamic interaction is performed with the environment to obtain the vehicle lane change planning result and complete the vehicle lane change planning.

[0100] In some embodiments, as Figure 5 To validate the effectiveness of the lane change re-planning control method based on GNNs and MARL, a simulation experiment was designed for a highly dynamic mixed traffic scenario and compared with a traditional MPC control benchmark. The experiment was conducted on two simulation platforms: one is a combined vehicle dynamics (CarSim) simulation and traffic flow (SUMO) simulation, where CAV stands for cooperative agent and HDV stands for non-cooperative agent. The figure shows a test road with a three-lane straight section and a right-hand ramp, which is used to simulate high-fidelity vehicle motion and large-scale traffic flow. The other is the open source autonomous driving simulator Carla, combined with a ROS bridge, which is used to verify the algorithm's performance in a realistic 3D environment. The results from both platforms are consistent, and the following description is based primarily on the former. Figure 5 Figure 3 illustrates the joint simulation scenario and platform used to verify the effectiveness of the method presented in the present invention. Background traffic is generated by the SUMO traffic flow engine and runs synchronously with the CarSim vehicle dynamics environment, forming a "CarSim+SUMO" joint simulation platform. The test road is a three-lane mainline with a right-side ramp. Speed ​​limits and traffic density can be configured according to the scenario. The figure shows a mixed traffic flow of collaborative connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs). The CAVs use the centralized training and decentralized execution strategy of the GNN fusion, while the HDVs are generated by the traffic flow model as non-collaborative entities. The system advances in simulation steps, exchanging vehicle status and decision information in real time, and recording indicators such as lane change success rate, collision rate, average speed, and ride comfort for comparative evaluation against a traditional MPC baseline.

[0101] In some embodiments, key experimental metrics include: Lane Change Success Rate: The percentage of vehicles attempting to change / merge successfully completing the required lane change maneuver. A vehicle is considered to have failed if it misses an exit or fails to merge into the target lane in time. Crash Rate: The number of collisions per kilometer traveled. This reflects safety performance. Average Speed ​​Improvement: The percentage increase in the average AV speed after implementing the strategy relative to the initial state or the traffic average speed, a measure of traffic efficiency. Passenger Comfort: This is measured by changes in vehicle acceleration and lateral acceleration. Trajectory Tracking Error: For vehicles executing lane changes, the study records the deviation of their actual trajectory from an ideal reference trajectory. For example, for the MPC solution, the reference trajectory is a polynomial programming curve; for the RL solution, the study uses the target lane centerline after the lane-changing vehicle completes the change as a reference, and calculates the time-varying lateral offset of the vehicle's center of mass relative to the lane centerline. Trajectory tracking error is quantified using maximum offset and steady-state offset. Additionally, the study calculates secondary metrics such as the average lane change time and average fuel consumption (estimated by speed and acceleration) for intelligent vehicles to analyze passenger experience and energy efficiency.

[0102] In some embodiments, as shown in Table 1, experimental results show that the proposed GNN+MARL method outperforms the traditional MPC solution in all indicators, especially in complex scenarios with high density and mixed driving styles. The improvement is (GNN+MARL−MPC) / MPC; negative values ​​indicate a decrease (better performance).

[0103] Table 1 Comparative experiment

[0104] Safety: The research method achieved near-zero collisions across all simulations: only one minor scrape occurred in 100 experiments (the AV lightly touched a neighboring vehicle while merging in extremely congested traffic, resulting in no substantial damage). This collision rate is significantly lower than the MPC solution's 0.72 collisions per kilometer. In contrast, the MPC solution experienced multiple collisions or sudden braking incidents in high-density, aggressive driving scenarios. This demonstrates that the reinforcement learning strategy effectively learned safe avoidance through training, delaying or abandoning lane changes in confined spaces to avoid accidents. Traditional MPC, however, suffers from the rigidity of its pre-planned trajectories, making it susceptible to uncontrolled situations when the environment changes suddenly (e.g., a vehicle in the target lane suddenly braking).

[0105] Efficiency: The GNN+MARL strategy significantly improves traffic flow efficiency. The average speed of AVs using this strategy is approximately 15% to 20% higher than that of the MPC solution (the improvement is smaller in low and medium traffic, but more pronounced in high traffic due to the RL strategy's ability to find gaps for overtaking). Specifically, in a medium traffic and aggressive behavior scenario, the average speed of AVs controlled by the RL strategy reached 33.5 m / s, compared to 28.1 m / s for the MPC solution, an improvement of approximately 19%. In high-traffic scenarios, even in overall traffic congestion, the RL strategy improves fleet efficiency through cooperation. For example, multiple AVs can form a "relay overtaking" pattern: the leading AV accelerates after overtaking, pulling the following vehicles along, completing the overtaking process in sequence, reducing the frequent braking caused by individual vehicles changing lanes independently. In contrast, AVs in traditional solutions often act independently, which can easily trigger a chain reaction when overtaking, leading to a decrease in overall speed.

[0106] Trajectory Smoothness and Comfort: Although the RL policy does not explicitly plan a polynomial trajectory, the learned policy still ensures smooth trajectory transitions due to the inclusion of comfort factors in the reward. Statistics show that the RL policy vehicle's average peak lateral acceleration during lane changes is 2.1 m / s², lower than the 2.5 m / s² of the MPC solution, and the average lateral jerk is reduced by approximately 30%. Passenger subjective comfort is equivalent to a reduction in abrupt changes in direction and sudden acceleration and deceleration. Regarding trajectory deviation, all lane-changing vehicles ultimately return to the target lane center. The RL policy's maximum lateral deviation (relative to the lane center) averages 0.12 m, while the MPC solution's is 0.15 m. While the two approaches have similar error ranges, the RL policy exhibits smoother deviations with less noticeable oscillation. The RL policy demonstrates excellent robustness, particularly in the presence of interfering vehicles (e.g., a slow vehicle suddenly appears ahead of the target lane, necessitating a mid-lane trajectory adjustment). The vehicle can proactively slow down and wait for an opportunity before completing the merge, avoiding the drastic control adjustments that occur in the MPC solution due to the forced lane change trajectory.

[0107] Collaborative behavior: Through multi-agent training, AVs have learned some implicit cooperative strategies. Simulation observations show that when two AVs are about to merge into the same lane at the same time, the AV at the rear will spontaneously slow down, and then accelerate to merge after the vehicle in front completes the merge, to avoid parallel competition. This kind of courteous behavior has never appeared in the MPC scheme: under the MPC scheme, sometimes the two vehicles will be stuck in parallel due to simultaneous lane changes, and even trigger the safety mechanism of one of the vehicles to interrupt the lane change. For example, at the entrance of the ramp, the RL scheme's in-lane vehicles will change to the adjacent lane in advance to make room for the merging vehicles, improving overall traffic efficiency and safety. These collaborative effects have also been reported in the literature: MARL-based vehicles can collaboratively adapt to the uncertain behavior of human vehicles and jointly improve the queue traffic capacity.

[0108] This approach achieves excellent results in safety (zero collisions), efficiency (higher speeds and traffic flow), and comfort (lower acceleration and deceleration shock). In contrast, while traditional MPC solutions maintain high trajectory tracking accuracy, they lack global coordination and adaptability in complex scenarios, demonstrating significant disadvantages. This demonstrates the value of incorporating graph neural networks and multi-agent reinforcement learning into lane-changing control.

[0109] To further analyze the effectiveness of the strategy, the study analyzed the results for different scenarios and parameters. First, as the traffic flow density increases, the performance gap between the two methods widens: at low density, both are almost collision-free and have similar efficiency, but at high density, traditional methods collide and jam frequently, while the RL strategy remains robust. This shows that the strategy studied is more scalable and can still work properly in more complex environments. Secondly, the impact of different HDV driving styles on the results shows that aggressive human driving increases the risk of accidents and the difficulty of the strategy, but the RL agent can partially adapt to this unpredictability through training (although the collisions increase slightly, they are far less than the MPC scheme), demonstrating the robustness of the learning strategy. Thirdly, the study examined the impact of reward weights on policy behavior: the safety weights were increased. and comfort weight Raising the weight will result in more conservative lane-changing behavior (collision approaches zero but efficiency is slightly reduced); on the contrary, increasing the efficiency weight This results in a more aggressive overtaking strategy (increased average speed but more critical situations). This demonstrates the balancing effect of multiple objective rewards in decision-making. The default weights used in this study strike a balance between safety and efficiency, but these can be adjusted in practice.

[0110] Finally, from the perspective of computing performance, since strategy execution only requires forward calculation of a neural network (including GNN and several fully connected layers), it can run in real time on the on-board computing unit (the study was tested on the NVIDIA AGX platform, and the decision time per cycle was < ms). In contrast, MPC requires solving the QP optimization at each step, and the computation time increases with the number of vehicles, making it difficult to meet real-time requirements in high-density scenarios. Therefore, the proposed method also has advantages in terms of real-time and scalability.

[0111] In some embodiments of the present specification, a vehicle lane change planning method based on a graph neural network and multi-agents is provided. The graph neural network fusion model is trained using a multi-agent reinforcement learning algorithm and a corresponding reward function. The parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss and value function loss based on the advantage function to obtain a trained graph neural network fusion model. The trained graph neural network fusion model is used to analyze the traffic scene graph structure to obtain a multi-agent decision-making strategy. The multi-agent decision-making strategy is subjected to decision prior fusion and dynamically interacts with the environment to obtain a vehicle lane change planning result, thereby completing the vehicle lane change planning. The control quantity is directly output by the policy network, which significantly simplifies the system structure. At the same time, relying on the self-learning ability of reinforcement learning and the powerful representation ability of the graph network, it can automatically adjust the decision for different traffic scenarios to achieve adaptive lane change re-planning control.

Claims

1. A vehicle lane change planning method based on graph neural network and multi-agent, characterized in that: include: S1: Obtaining a traffic scene graph structure with multiple agents; S2: Use the multi-agent reinforcement learning algorithm and the corresponding reward function to train the graph neural network fusion model. Update the parameters of the graph neural network fusion model by minimizing the policy gradient loss based on the advantage function and the value function loss to obtain the trained graph neural network fusion model. S3: Use the trained graph neural network fusion model to analyze the traffic scene graph structure and obtain the multi-agent decision-making strategy; S4: Perform decision prior fusion on the multi-agent decision strategy, dynamically interact with the environment, obtain the vehicle lane change planning result, and complete the vehicle lane change planning.

2. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: Said S1 comprises: Extract the initial feature vector of the multi-agent based on the state observed by the multi-agent; Constructing node features using the initial feature vector and neighbor information; Use the relative position and relative velocity between two nodes to construct edge features; A traffic scene graph structure is constructed based on the node features and the edge features.

3. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: The graph neural network fusion model includes: The input feature layer is used to extract features from the graph structure to obtain the vehicle's own and relative information features; A linear embedding layer is used to unify the dimensions of the vehicle itself and relative information features and pre-normalize them to obtain normalized features; The first image attention convolution layer is used to use four-head attention to perform one-hop interaction on the normalized features to obtain the first interaction result; The second graph attention convolution layer is used to perform two-hop neighbor information fusion based on normalized features to obtain the second interaction result; The residual convolution layer is used to enable normalized features as needed depending on the simulation scale to obtain enabled data; A readout-splicing layer is used to process the first interaction result, the second interaction result, and the activation data using a multi-layer perceptron to obtain policy input data; The actor-critic network layer, including the actor network and the critic network, is used to fully connect the policy input data to obtain the vehicle lane change planning instructions.

4. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 3 is characterized in that: The expression of the first interaction result is: ； ； ； ； in, Indicates the Layer Node The hidden state is the representation of the original features of the node after multi-hop neighborhood interaction. represents a nonlinear activation function, Indicates the The total message vector obtained by aggregating all neighbors at the layer, Indicates the Layer training bias vector, Indicates the Layer from neighbors Incoming unilateral messages, Representation node The neighbor node set, j, k represent the neighbor nodes in the set The index in represents the attention weight, Indicates the Layer-shared linear mapping matrix for feature transformation, Representation node In the hidden state of the previous layer, express function, Represents the trained attention vector, which is used to score the concatenated vector. T represents the vector transpose. represents the natural exponential function.

5. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: The S2 includes: Obtaining the environmental state of the traffic scene graph structure; wherein the environmental state includes the observation data of each intelligent agent; Using the graph neural network fusion model, we analyze the environmental state of the traffic scene graph structure and obtain the action vector of the agent: ； in, represents the action vector of the agent, represents the expected lateral acceleration, represents the expected longitudinal acceleration; The environment after each agent executes the corresponding action vector is used as the environment state at the next moment, and the reward function is used to obtain the immediate reward of the graph neural network fusion model; A multi-agent reinforcement learning algorithm is used to centrally train and decentralizedly execute each agent. A critic network is used to evaluate the overall value of the states and actions of all agents, obtaining the critic network parameters and global state value. The actor network is used to obtain the output action of each agent based on its own state distribution. Based on the immediate reward, critic network parameters and global state value, the parameters of the graph neural network fusion model are updated by minimizing the policy gradient loss based on the advantage function and the value function loss to obtain a trained graph neural network fusion model.

6. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 5 is characterized in that: The instant reward includes driving safety, driving efficiency, passenger comfort and cooperation factors. The expression of the instant reward is: ； in, Indicates vehicle At the moment Instant rewards received, 、 、 、 and Represents the weighted coefficient of each reward / penalty component, which is used to measure the importance of different indicators. represents the driving efficiency reward, Indicates the collaboration reward. The closer to the target lane / lane change completion, the higher the reward. It is used to guide the vehicle to complete the lane change within the specified distance. Indicates negative reward for driving safety, represents the lane-changing penalty, the negative cost of excessive or frequent lane changing, Represents acceleration penalty, a ride comfort index.

7. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: The expression of the policy gradient loss is: ； ； in, Express The gradient operator for optimization, represents the policy objective function, maximization object, Obey the strategy The trajectory expectation, represents the log-likelihood, which is convenient for gradient calculation, Representing an agent At the moment action, Representing an agent The observation of is obtained by graph attention convolution, represents the advantage function, which measures the quality of an action. represents the action-value function, Represents the global / centralized state of the environment, Represents the state-value function and reduces variance.

8. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: The expression of the value function loss is: ； in, represents the value function loss, which measures the error of the critic’s estimation of the state value, represents the state value estimate output by the value network, Indicates time environmental conditions, Indicates the target return, Estimates of real returns, represents the critic parameters.

9. The vehicle lane change planning method based on graph neural network and multi-agent according to claim 1 is characterized in that: The S4 includes: Performing decision prior fusion on the multi-agent decision strategy, and triggering a collision avoidance action when it is detected that the strategy output will cause a collision; Based on the current observed state of the agent, the trained graph neural network fusion model is used to update the multi-agent decision-making strategy, dynamically interact with the environment, obtain the vehicle lane change planning results, and complete the vehicle lane change planning.

10. A vehicle lane change planning system based on graph neural network and multi-agent, used to execute the vehicle lane change planning method based on graph neural network and multi-agent according to any one of claims 1 to 9, characterized in that: include: The perception module is used to obtain the status information of the intelligent agent and surrounding traffic participants and construct the traffic scene graph structure; The decision module is used to extract and encode features from the scene graph using a graph neural network fusion model, map the multi-vehicle interaction state into a compact low-dimensional representation, and output a multi-agent decision strategy using multi-agent reinforcement learning. The vehicle control module is used to perform decision-making prior fusion on the multi-agent decision-making strategy, dynamically interact with the environment according to the current observed state of the agent, obtain the vehicle lane change planning result, and complete the vehicle lane change planning.

Images