A Deep Reinforcement Learning-Based CAV Lane Changing Decision Optimization Method

By optimizing CAV lane-changing decisions through a spatiotemporal attention mechanism and a deep deterministic policy gradient algorithm, and combining driving style perception and hybrid fault-tolerant control, the problem of decision-making accuracy and collaborative efficiency of CAV in complex traffic scenarios is solved, achieving a balance between safety, efficiency and comfort.

CN121224740BActive Publication Date: 2026-11-17CHONGQING UNIV OF POSTS & TELECOMM
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202511512373.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-10-22
Publication Date
2026-11-17
Estimated Expiration
2045-10-22

AI Technical Summary

Technical Problem

Existing CAV lane-changing decision-making methods struggle to effectively integrate multi-source heterogeneous information in complex traffic scenarios, resulting in insufficient decision-making accuracy, low system coordination efficiency, and difficulty in achieving a good balance between safety, efficiency, and comfort.

Method used

A spatiotemporal attention mechanism is adopted to encode and dynamically weight and fuse the vehicle's micro-motion state and lane-level macro-traffic parameters. The decision instructions are optimized by combining a deep deterministic policy gradient algorithm and a driving style perception mechanism. The results are then converted into vehicle control quantities through hybrid fault-tolerant control, realizing closed-loop decision control under vehicle-road-cloud collaboration.

Benefits of technology

It improves the accuracy and environmental adaptability of lane-changing decisions, enhances multi-vehicle coordination efficiency, and optimizes traffic efficiency, driving safety, and passenger comfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121224740B_ABST
    Figure CN121224740B_ABST
Patent Text Reader

Abstract

This invention proposes a CAV lane-changing decision optimization method based on deep reinforcement learning, comprising: collecting vehicle micro-motion state, lane-level macro-traffic parameters, and global traffic event information; using a spatiotemporal attention mechanism to encode and dynamically weight and fuse the vehicle micro-motion state and lane-level macro-traffic parameters to obtain a fused environmental state representation; processing the fused environmental state representation using a deep deterministic policy gradient algorithm to generate preliminary decision instructions; constructing a driving style perception mechanism, dynamically adjusting the reward function weights based on this mechanism, and co-optimizing the preliminary decision instructions through the optimized reward function to obtain optimized decision instructions adapted to individual driving styles; using a hybrid fault-tolerant strategy to convert the optimized decision instructions into lateral and longitudinal control variables; controlling the vehicle to change lanes based on the lateral and longitudinal control variables to complete closed-loop decision control under vehicle-road-cloud collaboration; this invention can effectively improve the accuracy and environmental adaptability of lane-changing decisions and enhance the efficiency of multi-vehicle collaboration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a CAV lane-changing decision optimization method based on deep reinforcement learning. Background Technology

[0002] With the deep integration of intelligent transportation systems and autonomous driving technology, connected autonomous vehicles (CAVs) have become a key vehicle for improving road traffic efficiency and ensuring driving safety. In complex and dynamic traffic scenarios, lane-changing decisions are one of the most challenging driving behaviors, and their level of intelligence directly determines the applicability and reliability of CAVs in real-world road environments.

[0003] In existing technologies, traditional CAV lane-changing decision-making methods mainly rely on rule-based strategies, optimal control theory, or traditional machine learning algorithms. While rule-based methods are simple to implement, they lack the necessary flexibility and struggle to adapt to dynamically changing traffic conditions. Optimization control-based methods are overly dependent on the accuracy of the system model and often fail to meet real-time decision-making requirements due to computational complexity. Traditional machine learning methods, on the other hand, have significant shortcomings in automatic feature extraction and long-term decision sequence optimization. These methods generally fail to effectively integrate multi-source heterogeneous information in the vehicle-road-cloud collaborative environment, leading to problems such as insufficient decision-making accuracy and low system collaboration efficiency in complex scenarios.

[0004] In recent years, deep reinforcement learning (DRL) technology has provided a new approach to solving the sequential decision-making problem in autonomous driving. However, existing DRL-based lane-changing decision-making methods for CAVs still suffer from systemic defects, including incomplete perception of the spatiotemporal characteristics of the environment, a lack of adaptability of decision-making strategies to multi-objective and personalized needs, and insufficient reliability assurance in system execution. These technical bottlenecks make it difficult for existing methods to achieve a good balance between safety, efficiency, and comfort when dealing with highly dynamic and highly interactive real-world road environments, severely restricting the large-scale application of CAVs. Breakthroughs through innovative algorithm architectures and system designs are urgently needed. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a CAV lane-changing decision optimization method based on deep reinforcement learning. This method includes: collecting vehicle micro-motion states, lane-level macro-traffic parameters, and global traffic event information; employing a spatiotemporal attention mechanism to encode and dynamically weight the vehicle micro-motion states and lane-level macro-traffic parameters to obtain a fused environmental state representation; processing the fused environmental state representation using a deep deterministic policy gradient algorithm to generate preliminary decision instructions; constructing a driving style perception mechanism, dynamically adjusting the reward function weights based on this mechanism, and co-optimizing the preliminary decision instructions using the optimized reward function to obtain optimized decision instructions adapted to individual driving styles; employing a hybrid fault-tolerant strategy to convert the optimized decision instructions into lateral and longitudinal control variables; and controlling the vehicle to change lanes based on the lateral and longitudinal control variables to complete closed-loop decision control under vehicle-road-cloud collaboration.

[0006] The beneficial effects of this invention are:

[0007] This invention constructs an environmental perception method that integrates spatiotemporal attention mechanisms, achieving dynamic weighted fusion and deep feature extraction of macro and micro traffic information under vehicle-road-cloud collaboration. By designing a deep reinforcement learning decision-making method embedding a driving style perception mechanism, it realizes dynamic weight adjustment and personalized strategy optimization of multi-objective reward functions. Furthermore, by employing a hybrid fault-tolerant control method, it ensures reliable execution of decision commands under complex operating conditions. This invention effectively improves the accuracy and environmental adaptability of lane-changing decisions, enhances multi-vehicle collaboration efficiency, and achieves synergistic optimization in key performance indicators such as traffic efficiency, driving safety, and passenger comfort. Attached Figure Description

[0008] Figure 1 This is a flowchart of the CAV lane-changing decision optimization in an embodiment of the present invention;

[0009] Figure 2 This is a schematic diagram of the spatiotemporal attention mechanism for CAV lane-changing decision-making in an embodiment of the present invention.

[0010] Figure 3 This is a flowchart of the deep reinforcement learning training mechanism for CAV lane-changing decision-making in an embodiment of the present invention.

[0011] Figure 4 This is a schematic diagram illustrating the dynamic adjustment of driving style perception and reward function in CAV lane-changing decision-making in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] A deep reinforcement learning-based CAV lane-changing decision optimization method includes: collecting vehicle micro-motion state, lane-level macro-traffic parameters, and global traffic event information; using a spatiotemporal attention mechanism to encode and dynamically weight the vehicle micro-motion state and lane-level macro-traffic parameters to obtain a fused environmental state representation; processing the fused environmental state representation using a deep deterministic policy gradient algorithm to generate preliminary decision instructions; constructing a driving style perception mechanism, dynamically adjusting the reward function weights based on this mechanism, and co-optimizing the preliminary decision instructions through the optimized reward function to obtain optimized decision instructions adapted to individual driving styles; using a hybrid fault-tolerant strategy to convert the optimized decision instructions into lateral and longitudinal control variables; and controlling the vehicle to change lanes based on the lateral and longitudinal control variables to complete closed-loop decision control under vehicle-road-cloud collaboration.

[0014] A specific implementation of a CAV lane-changing decision optimization method based on deep reinforcement learning, such as... Figure 1 As shown, the method includes:

[0015] S1. The vehicle unit, roadside unit and cloud control platform work together to collect multi-source heterogeneous information consisting of vehicle micro motion state, lane-level macro traffic parameters and global traffic events. The raw data is preprocessed by time alignment, coordinate unification and noise filtering to form standardized multi-source information input.

[0016] The onboard unit utilizes a multi-sensor fusion system integrating LiDAR, millimeter-wave radar, and vision sensors to acquire real-time microscopic motion state parameters of the vehicle and surrounding vehicles, including high-precision 3D coordinates, instantaneous speed, acceleration, and heading angle data. A federated Kalman filter algorithm is employed to fuse and estimate the state of the multi-source sensor data, generating a feature vector that accurately characterizes the vehicle's dynamics. Simultaneously, the roadside unit estimates lane-level traffic parameters in real-time based on the Lighthill-Whitham-Richards macroscopic traffic flow model, covering lane flow rate, density, and spatial average speed. It also integrates traffic event warning information identified by roadside sensing devices and vehicle cooperation tendency tags obtained through V2X communication. The cloud control platform is responsible for issuing global traffic situation information, including full-domain traffic event identifiers, cooperative lane-changing suggestions, and system-level scheduling commands.

[0017] In this embodiment, a systematic preprocessing process is performed on multi-source heterogeneous data: First, a hardware time synchronization mechanism is used to ensure accurate alignment of timestamps across data sources. Then, coordinate transformation is used to unify all spatial data to a standard geographic coordinate system. Next, an adaptive filtering algorithm is used to suppress noise and remove outliers from the original sensor data. Finally, through feature-level splicing and normalization, a standardized state input with consistent data structure and semantic norms is generated, which establishes a high-quality information foundation for subsequent spatiotemporal feature fusion and deep reinforcement learning decision-making.

[0018] S2. A spatiotemporal attention mechanism is used to perform deep feature fusion on the preprocessed multi-source information. A hierarchical attention mechanism is used to effectively integrate macro and micro information and generate a unified environmental state representation vector.

[0019] In this embodiment, the dynamic perception principle and effect of the spatiotemporal attention mechanism are as follows: Figure 2 As shown, in a typical lane-changing scenario caused by road construction, the spatiotemporal attention mechanism can intelligently assign differentiated attention weights to surrounding vehicles: a high attention weight is assigned to vehicle A, which cuts into its lane to avoid construction, because its behavior poses a direct risk; a medium attention weight is assigned to vehicle B in front of it in its lane; and a low attention weight is assigned to vehicle C, which is far away. This mechanism effectively improves the efficiency and robustness of environmental perception by focusing on key interacting vehicles.

[0020] The processing steps of the spatiotemporal attention mechanism are as follows:

[0021] Step 1: Encode lane topology and macroscopic traffic flow parameters using a graph neural network to extract spatial embedding features characterizing the road network structure. First, construct a lane-level topology graph based on high-precision map data, discretizing each lane's centerline into a series of nodes at 10-meter intervals. Each node is described using a 6-dimensional feature vector, containing normalized lane curvature, slope, speed limit, and a 3-dimensional one-hot encoding representing the connectivity. Edge features are described using 2-dimensional vectors, including normalized steering angle and node spacing. The GraphSAGE algorithm is used for feature propagation, specifically including three sub-steps: neighbor sampling, feature aggregation, and feature transformation. In the neighbor sampling stage, for each target node, directly connected nodes within a 50-meter radius are selected as candidate neighbors, ensuring a maximum of 8 neighbor nodes through random sampling. In the feature aggregation stage, a mean aggregator is used to calculate the arithmetic mean of the neighbor node features:

[0022]

[0023] in, This represents the aggregated feature vector of all neighboring nodes of the target node v. Indicates the number of neighboring nodes. This represents the feature vector of neighbor node u at the (k-1)th layer. This represents the summation of the feature vectors of all neighboring nodes. During the feature transformation stage, the aggregated neighbor features are concatenated with the target node's own features, and then a linear transformation is performed through a fully connected layer using the ReLU activation function.

[0024]

[0025] in, This represents the feature vector output by the target node v at the k-th layer. Indicates the first The learnable weight matrix of the layer, This represents a vector concatenation operation. Indicates that the target node v is at the th The feature vector of the layer, This represents the aggregated feature vector of neighboring nodes. This represents the modified linear unit activation function. After two layers of graph convolution, the output is an 8-dimensional lane-level embedding feature vector, where the weight matrix... , .

[0026] Step 2: A bidirectional long short-term memory network with an attention mechanism is used to process the vehicle's historical trajectory sequence to capture key temporal driving patterns. A two-layer bidirectional LSTM network is used to process the vehicle's historical trajectory sequence. The input consists of 30 time-step data samplesd at 10Hz over the past 3 seconds. Each time-step input vector contains 15-dimensional normalized features, including the relative position, velocity, and acceleration of the vehicle and surrounding vehicles. The number of hidden units in the network is set to 64. The forward propagation calculation process is as follows: Backpropagation is: The bidirectional output is spliced ​​into a 128-dimensional hidden state. Building upon this, a temporal attention mechanism is constructed. First, the encoder's hidden state is mapped to key-value pairs using three independent fully connected layers:

[0027]

[0028] in, This represents the key vector at the i-th time step. This represents the value vector at the i-th time step. and These represent the learnable weight matrices for keys and values, respectively. This represents the encoder's hidden state at time step i. The decoder state is mapped to a query vector:

[0029]

[0030] in, This represents the query vector for the j-th decoding step. This represents the learnable weight matrix of the query. The decoder is in the first The hidden states at each time step. The specific process for calculating the attention score is as follows: First, calculate the dot product similarity between the query vector and all key vectors:

[0031]

[0032] in, This represents the unnormalized attention score between the j-th decoding step and the i-th encoding step. This represents the transpose of the query vector. This represents the key vector at the i-th time step. Then, a scaling factor is applied. After adjusting the numerical range, the attention weights are finally obtained by normalization using the softmax function.

[0033]

[0034] in, This represents the normalized attention weight of the j-th decoding step to the i-th encoding step. Represents an exponential function. This represents the summation over all 30 coded time steps. The weighted summation process is expressed as:

[0035]

[0036] in, This represents the context vector for the j-th decoding step. Indicates attention weights, This represents the value vector at the i-th time step. This context vector, concatenated with the decoder state, is passed through a fully connected layer to output 128-dimensional micro-features.

[0037] Step 3: At the fusion layer, a spatiotemporal attention mechanism is used to capture long-term dependencies in the time dimension using multi-head self-attention, and to quantify the interaction strength between vehicles in the spatial dimension using a graph attention network. This achieves dynamic weighting and deep fusion of macro and micro features, outputting a unified state vector with strong discriminative power. The 128-dimensional micro features are concatenated with the 8-dimensional macro features to form a 136-dimensional joint feature vector. The time dimension processing employs a 4-head self-attention mechanism, with each head having a 16-dimensional key-value space. The specific calculation process includes generating a query matrix, key matrix, and value matrix from the input features through linear transformation.

[0038]

[0039] in, , , These represent the query, key, and value matrices of the h-th attention head, respectively. Represents the input feature matrix. , , Let represent the learnable projection matrices of the h-th attention head query, key, and value, respectively. Attention is calculated using a scaled dot product:

[0040]

[0041] in, Let QhKhT represent the output of the h-th attention head, where QhKhT represents the product of the query matrix and the transpose of the key matrix, 16 represents the scaling factor, MM represents the causal mask matrix, and VhVh represents the value matrix. A vehicle interaction graph is constructed in the spatial dimension, and a graph attention network is used to calculate the influence weights between nodes, specifically including feature transformations.

[0042]

[0043] in, This represents the feature vector after the transformation of node i. Represents the learnable weight matrix. This represents the original feature vector of node i. Calculate the attention coefficient:

[0044]

[0045] in, This represents the unnormalized attention coefficient between node i and node j. This represents a learnable attention vector. This represents a vector concatenation operation. This represents the modified linear unit activation function with leakage. Normalized attention weights:

[0046]

[0047] in This represents the summation over all neighboring nodes of node i. Feature aggregation:

[0048]

[0049] in, This represents the feature vector after aggregation of node i. This represents the sigmoid activation function. This represents summing the values ​​of all neighboring nodes of node i. Indicates attention weights, This represents the transformed feature vector of neighbor node j. Finally, the fused features are dimensionality-reduced through two fully connected layers (dimensions 136→68→33 respectively), with each layer followed by a LayerNorm layer and a ReLU activation function, outputting a 33-dimensional environment state representation vector.

[0050] S3. The fused state representation vector is input into a deep deterministic policy gradient algorithm network. An optimized lane-change control command is generated through an Actor-Critic network architecture, and personalized decision-making is achieved by combining this with a driving style perception mechanism. The overall process is as follows: Figure 3 As shown.

[0051] The decision-making algorithm network is based on the Deep Deterministic Policy Gradient (DDPG) algorithm, constructing an Actor-Critic network architecture. It processes a 33-dimensional unified state representation vector generated by spatiotemporal attention mechanisms, outputting continuous control commands (including lateral displacement and longitudinal acceleration commands) to optimize lane-changing decisions. The Actor network, acting as the policy network, is responsible for generating deterministic actions based on the current environmental state; the Critic network, acting as the value network, evaluates the long-term expected reward of the Actor's output actions, guiding policy updates through temporal differential errors. To enhance the personalization and adaptability of the decision-making, a driving style perception mechanism is embedded in the reward function design, the principle of which is as follows: Figure 4 As shown, Figure 4 (a) shows the differences in preferences for efficiency, safety, and comfort among the three driving styles. Figure 4 (b) qualitatively demonstrates the adjustment strategy for the corresponding reward weights, enabling the decision-making to adapt to different driving styles. Based on the clustering analysis results of historical driving behavior data, the multi-objective reward weights are dynamically adjusted to ensure that the decision-making strategy can fit the preferences of different driving styles (such as aggressive, moderate, and conservative). During training, a priority experience replay mechanism is adopted, and sampling priority is allocated according to the temporal difference error of the samples to accelerate model convergence. Ornstein-Uhlenbeck process noise is introduced to enhance exploration efficiency and ensure the robustness and generalization ability of the strategy in complex traffic environments.

[0052] In this embodiment, an Actor-Critic network architecture is constructed. The Actor network takes the environmental state representation as input and outputs deterministic actions through a deep feedforward neural network containing multiple fully connected layers. The deterministic actions include normalized lateral displacement commands and longitudinal acceleration commands. The Critic network takes the concatenated vector of the environmental state representation and the actions as input and outputs an estimate of the state-action value function. A multi-objective reward function integrating traffic efficiency, driving safety, and ride comfort is constructed, and multi-objective collaborative optimization is achieved through weighted summation. A priority experience replay mechanism is adopted, and sampling priority is assigned to training samples according to the temporal difference error. Ornstein-Uhlenbeck process noise is introduced to enhance exploration efficiency. The Actor network and Critic network are co-trained by minimizing the temporal difference error to generate optimized lane-changing commands.

[0053] The decision-making algorithm's processing steps are as follows:

[0054] Step 1: Construct the Actor-Critic network architecture and design the training mechanism. The decision algorithm adopts the Deep Deterministic Policy Gradient (DDPG) algorithm, constructing a dual-network architecture containing an Actor policy network and a Critic value network to process a 33-dimensional unified environment state representation vector. The Actor policy network adopts a deep feedforward neural network structure, including an input layer, three fully connected hidden layers, and an output layer. Its forward propagation process is as follows:

[0055]

[0056] in, The input state vector, to This is the weight matrix. to For bias vectors, to The activation value of the hidden layer. The output action vector contains normalized lateral displacement and longitudinal acceleration commands. The Critic value network takes a concatenated vector of state and action as input and outputs an estimate of the state-action value function.

[0057]

[0058] in, This represents a vector concatenation operation. For long-term expected returns.

[0059] Step 2: Design a multi-objective reward function and style-aware mechanism. The reward function integrates traffic efficiency, driving safety, and passenger comfort objectives, achieving multi-objective optimization through weighted summation:

[0060]

[0061] in, These are the weighting coefficients. The calculation functions for each reward item are provided. Efficiency rewards are calculated based on the speed increase after lane changes and the traffic flow optimization effect.

[0062]

[0063] in, , These represent the speeds before and after the lane change. Speed ​​limits for roads, For the target lane density, For maximum density, , This is an adjustment factor. The safety bonus is calculated using a time-to-collision (TTC) versus safe distance model:

[0064]

[0065] in, Let τ be the collision time between the vehicle and surrounding vehicle j, and τ be a time constant. As a safety threshold, This is an indicator function. The driving style perception mechanism achieves personalized decision-making by dynamically adjusting reward weights.

[0066]

[0067] in, As the benchmark weight, The adjustment coefficient is obtained from the preset mapping table based on the real-time recognition of driving style.

[0068] Step 3: Implement training optimization and personalized decision-making. Network training employs a priority experience replay mechanism, with sample priority calculated based on temporal difference error:

[0069]

[0070] in, For timing difference error, To prevent small constants with zero priority, the Actor network policy gradient is calculated using the deterministic policy gradient theorem:

[0071]

[0072] in Here are the parameters for the Actor network, and N is the batch size. The gradient of the Critic network with respect to the action. This represents the gradient of the policy function with respect to the parameters. Through the synergistic effect of the above mechanisms, the intelligent decision-maker, while ensuring driving safety, achieves a personalized balance between traffic efficiency and passenger comfort based on driving style preferences, generating optimized lane-changing strategies adapted to complex traffic environments.

[0073] S4. A hybrid fault-tolerant control strategy is adopted to convert decision commands into precise vehicle control quantities. A dynamic switching mechanism based on state estimation confidence is used to ensure the control reliability of the system under different operating conditions.

[0074] The hybrid fault-tolerant control strategy is configured to convert lane-changing commands output by the intelligent decision-maker into precise vehicle control quantities. The strategy integrates a dual-mode architecture of model predictive control (MPC) and PID control, and ensures the control reliability of the system under different operating conditions such as sensor malfunctions and communication interruptions through a dynamic switching mechanism based on state estimation confidence.

[0075] The hybrid control method configures a model predictive controller as the primary control mode, providing accurate trajectory tracking control when the system state estimation confidence level is high; and configures a PID controller as a backup control mode, providing stable and reliable low-level control when the confidence level is below a threshold. Seamless switching between the two control modes is achieved through confidence evaluation based on Kalman filter state estimation.

[0076] The confidence assessment mechanism is based on real-time analysis of the state estimation error covariance matrix to construct a confidence index to quantify the reliability of the current state estimation. When the confidence index is higher than a preset threshold, the model predictive control mode is maintained; when the confidence index is lower than the threshold, it automatically switches to PID control mode to ensure the robustness of the control system.

[0077] In this embodiment, the processing steps of the hybrid fault-tolerant control strategy are as follows:

[0078] Step 1: Model Predictive Controller Design and Objective Function Construction. The model predictive controller solves the optimization problem in the finite time domain in each control cycle. Its objective function is designed as follows:

[0079]

[0080] in, Indicates the length of the prediction time domain. Indicates the length of the control time domain. This indicates the system's prediction step size. The output vector, Indicates the corresponding reference trajectory, The value represents the control increment, and Q and R represent the weight matrices of the output tracking error and the control increment, respectively.

[0081] The constraints for the model predictive controller are set as follows:

[0082]

[0083] in, Indicates the control quantity. , Indicates the upper and lower limits of the control quantity. , This indicates the upper and lower limits of the control increment. , Indicates the upper and lower limits of the output quantity.

[0084] Step 2: PID Controller Configuration and Adaptive Parameter Adjustment. The PID controller is based on feedback control of path tracking error, and its control law is expressed as:

[0085]

[0086] in, Indicates path tracking error. , , These represent the proportional, integral, and differential coefficients, respectively.

[0087] The PID parameters employ an adaptive adjustment mechanism to optimize control parameters in real time based on vehicle status.

[0088]

[0089] in, , , Indicates the reference parameter. Indicates vehicle speed. Indicates the road surface adhesion coefficient. , , This indicates a parameter adjustment function.

[0090] Step 3: Confidence Assessment and Dynamic Switching Mechanism Implementation. The confidence assessment index is calculated based on the state estimation error covariance matrix:

[0091]

[0092] in, This indicates the confidence level index. This represents the state estimation error covariance matrix. Represents the trace of a matrix. This represents a small constant to prevent division by zero.

[0093] The dynamic switching logic is set as follows:

[0094]

[0095] in, This represents the confidence threshold, which is preset according to the system reliability requirements.

[0096] The switching process employs a smooth transition mechanism to avoid abrupt changes in control variables.

[0097]

[0098] in, This represents the transition weight coefficient, which smoothly changes from 1 to 0 or vice versa during the switching process.

[0099] By implementing the above-mentioned hybrid fault-tolerant control strategy, control reliability can be maintained under different operating conditions, and the robustness and safety of the system can be improved while ensuring control accuracy.

[0100] S5. By monitoring vehicle status and environmental changes in real time, a closed-loop decision-making control system is formed through vehicle-road-cloud collaboration. Based on the actual execution effect, the decision-making strategy is optimized online to achieve continuous improvement and adaptive capability of lane-changing decisions.

[0101] After the lane-change control command is issued and executed, multi-source data from vehicle actuators, roadside units, and the cloud control platform are monitored and collected. This data includes the vehicle's actual motion state, real-time interaction with surrounding vehicles, and changes in macroscopic traffic flow parameters. Based on the feedback data, the actual multi-objective reward value obtained within the current decision-making cycle is calculated, thereby generating a temporal difference error. Based on the temporal difference error, online incremental gradient updates are performed on the policy network and value network in the deep deterministic policy gradient algorithm to optimize network parameters and achieve real-time fine-tuning and adaptation of the decision model.

[0102] Simultaneously, based on accumulated historical driving behavior data, the weight coefficients corresponding to traffic efficiency, driving safety, and passenger comfort in the multi-objective reward function are dynamically adjusted to achieve adaptive matching between decision preferences and driving styles. The optimized strategy parameters are collaboratively distributed and shared among connected autonomous vehicles via a cloud control platform, thereby achieving iterative evolution and knowledge fusion of group decision-making experience. Through the above closed-loop process of "decision-execution-evaluation-optimization," this method can achieve continuous performance improvement and enhanced stability and reliability of lane-changing decision strategies in complex dynamic environments.

[0103] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0104] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A CAV lane-changing decision optimization method based on deep reinforcement learning, characterized in that, include: Collect vehicle micro-motion states, lane-level macro-traffic parameters, and global traffic event information; A spatiotemporal attention mechanism is employed to encode and dynamically weight and fuse vehicle micro-motion states and lane-level macro-traffic parameters to obtain a fused environmental state representation. A deep deterministic policy gradient algorithm is used to process this fused environmental state representation, generating preliminary decision commands. A driving style perception mechanism is constructed, and the reward function weights are dynamically adjusted based on this mechanism. The optimized reward function is then used to collaboratively optimize the preliminary decision commands, resulting in optimized decision commands adapted to individual driving styles. A hybrid fault-tolerant strategy is used to convert the optimized decision commands into lateral and longitudinal control variables. Based on these lateral and longitudinal control variables, the vehicle is controlled to change lanes, completing closed-loop decision control under vehicle-road-cloud collaboration. The spatiotemporal attention mechanism is used to encode and dynamically weight and fuse vehicle micro-motion states and lane-level macro-traffic parameters. This includes: at the macro-feature encoding layer, a graph neural network is used to encode lane-level topological connections and macro-traffic flow parameters provided by roadside units and the cloud control platform, generating macro-embedded feature vectors that characterize the road network structure; at the micro-feature encoding layer, a bidirectional long short-term memory network is used to process the historical trajectory sequences of the vehicle and surrounding vehicles provided by the onboard unit, and a temporal attention mechanism is introduced to dynamically focus on historical driving behavior segments that have a key impact on the current decision; at the spatiotemporal attention fusion layer, the macro-feature vector and micro-feature vector are used as inputs, and long-term and short-term temporal dependencies are captured through a multi-head self-attention mechanism in the time dimension, while the interaction strength between vehicles is modeled through a graph attention network in the spatial dimension. The long-term and short-term temporal dependencies and the interaction strength between vehicles are dynamically weighted and fused to obtain an environmental state representation vector. The dynamic adjustment of reward function weights based on driving style perception includes: acquiring users' historical driving data, processing the data using a clustering algorithm to obtain the user's driving style; and dynamically adjusting the weight coefficients of each item in the reward function based on the obtained driving style. The adjustment expression is as follows: ; in, This indicates the baseline weight of the corresponding reward item; This is a style adjustment factor.

2. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, The graph neural network in the macroscopic feature encoding layer is the GraphSAGE model; the expression for processing data by the GraphSAGE model is: ; in, This represents the set of all neighboring lane nodes directly connected to node v in the road graph structure. Then it represents the feature vector obtained after calculating the (k-1)th layer for any neighbor node u. It is a learnable weight matrix specifically defined for the k-th layer of the network. Represents a non-linear activation function. This represents the specific aggregation function used in the k-th layer.

3. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, The temporal attention mechanism introduced in the micro-feature encoding layer, and the formula for data processing by the micro-feature encoding layer are as follows: ; in, Is the decoder in the 1st The hidden state of the step, It is the hidden state of the encoder at time step i. and It is a learnable projection matrix. It is the dimension of the key vector. The scaling factor is used to control the range of the dot product values.

4. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, The spatiotemporal attention fusion layer processes data including: ; in, and These are the feature vectors of the target vehicle i and its neighboring vehicle j, obtained after preliminary feature extraction. is a shared, learnable weight matrix, and 'a' is a learnable attention vector. This indicates a concatenation operation between two feature vectors. LeakyReLU is a linear rectified unit activation function with a small negative slope.

5. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, The method for processing the fused environmental state representation using a deep deterministic policy gradient algorithm includes: constructing an Actor-Critic network architecture, where the Actor network takes the environmental state representation as input and outputs deterministic actions through a deep feedforward neural network containing multiple fully connected layers, including normalized lateral displacement commands and longitudinal acceleration commands; the Critic network takes the concatenated vector of the environmental state representation and actions as input and outputs an estimate of the state-action value function; constructing a multi-objective reward function that integrates traffic efficiency, driving safety, and ride comfort objectives, and achieving multi-objective collaborative optimization through weighted summation; employing a priority experience replay mechanism, assigning sampling priorities to training samples based on temporal difference errors, and introducing Ornstein-Uhlenbeck process noise to enhance exploration efficiency; and co-training the Actor and Critic networks by minimizing temporal difference errors to generate optimized lane-changing commands.

6. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, The expression for the reward function is: ; in, It is an efficiency reward item; It is a security bonus item; It is a comfort-oriented reward item; , , These are the weighting coefficients corresponding to the three reward items mentioned above.

7. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 1, characterized in that, Hybrid fault-tolerant strategies include PID control and model predictive control.

8. The CAV lane-changing decision optimization method based on deep reinforcement learning according to claim 7, characterized in that, The hybrid fault-tolerant strategy for data processing includes: defining a real-time confidence index θ and setting a confidence threshold. ,like In this case, a model predictive controller is used. If the control is not achieved, then control will be switched to the PID controller; the confidence index θ is calculated using the following formula: ; in, It is the error covariance matrix of the state estimation filter output; Representation matrix traces, This represents a small constant to prevent division by zero.

Citation Information

Patent Citations

  • Vehicle lane changing behavior decision-making method based on deep reinforcement learning and system thereof

    CN114074680A

  • Intelligent vehicle lane changing decision-making method and system for LSTM trajectory prediction

    CN117325865A

  • Positioning confidence real-time evaluation method, calculation device and storage medium

    CN118882690A