Automatic driving lane change decision-making method and system based on Q value mask

Abstract

The invention discloses an automatic driving lane change decision-making method and system based on a Q value mask, and belongs to the field of automatic driving and artificial intelligence. According to the method, a hierarchical decision architecture is adopted, a lane change decision is decomposed into two levels of high-level decision learning and low-level control execution, and tight integration is realized through a Q-masking interface. The method comprises the following steps: constructing a hierarchical decision-making architecture; vehicle state representation and a high-level discrete action space are designed, the state comprises normalized scalar information and a time sequence occupation grid, and the action comprises five types including no operation, acceleration, deceleration, left lane change and right lane change; the low-layer module generates a Q value mask according to system constraints, priori knowledge and security detection, and the deep Q network outputs a Q value and then applies the mask to perform action selection; and performing network training by using a double-buffer empirical playback strategy. According to the method, the zero collision rate in the training and testing process is ensured through the Q value mask, the reward function design is simplified, and the training efficiency and safety are improved.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving and artificial intelligence, and particularly relates to an autonomous driving lane change decision method and system based on Q-value mask. Background Technology

[0002] Lane changing is a core capability for autonomous vehicles in multi-lane highway environments, directly impacting driving efficiency and safety. Traditional methods include:

[0003] Rule-based and optimization-based methods, such as model predictive control (MPC) and trajectory planning, are suitable for handling low-level control problems, but it is difficult to formulate complex long-term high-level decision-making strategies.

[0004] End-to-end deep learning methods directly map sensory inputs to control commands, but they suffer from problems such as training difficulties, large data requirements, and difficulty in ensuring security.

[0005] Reinforcement learning methods can learn arbitrary policies and have advantages such as timely error correction, instant learning from environmental feedback, no need for expert models or ground truth, and easy guidance of agent behavior. However, fully end-to-end learning faces challenges such as large network size, unstable training, and difficulty in avoiding collisions.

[0006] The main limitations of existing technologies are: traditional methods struggle to develop long-term high-level decision-making capabilities; end-to-end learning lacks the utilization of prior knowledge, resulting in low training efficiency; and reward function design is complex, requiring a balance of multiple objectives (speed, safety, rule compliance, etc.). Due to the characteristics of Q-learning methods, entropy is required to allow the agent to explore within the agent space, which naturally leads to collisions during exploration, contradicting the requirement of zero accident rate in autonomous driving and making direct deployment to real vehicles difficult.

[0007] Therefore, there is a need for a method that can effectively combine high-level decision learning and low-level control capabilities, and is efficient, safe, and reliable. Summary of the Invention

[0008] This invention discloses a lane change decision-making method and system for autonomous driving based on Q-value masking. The method employs a hierarchical decision architecture, decomposing lane change decisions into two layers: a high-level decision learning layer and a low-level control execution layer, which are tightly integrated through a Q-masking interface.

[0009] An autonomous driving lane change decision-making method based on Q-value masks includes the following steps:

[0010] S1. Construct a hierarchical decision-making architecture, including a high-level decision-making module and a low-level control module, and achieve tight integration through the Q-masking interface;

[0011] Includes the following sub-steps:

[0012] S11. Establish a high-level decision-making module and use a deep Q-network to learn long-term lane change strategies;

[0013] S12. Establish a low-level control module to integrate system constraints, prior knowledge, and collision detection functions;

[0014] S13. The high-level module and the low-level module are tightly integrated through the Q-masking interface. The interface converts the low-level judgment result into a mask vector, which restricts the high-level network to learn and select only from the safe and feasible action subspace.

[0015] S2. Design the vehicle state representation and high-level discrete action space, where the state includes normalized scalar information and temporal occupancy grid;

[0016] Includes the following sub-steps:

[0017] S21. Collect vehicle internal status information, including current speed v, lane l, and distance to target d2g, and normalize the scalar information to map it to the [0,1] interval;

[0018] S22. Construct a binary occupancy grid of the visible area around the vehicle, merge the occupancy grids from multiple historical time steps, and form a temporal occupancy grid representation;

[0019] S23. Define the high-level discrete action space, including five types of actions: no operation (N), acceleration (A), deceleration (D), left lane change (L), and right lane change (R).

[0020] S3. The low-level module generates a Q-value mask based on system constraints, prior knowledge, and security checks. The deep Q-network outputs the Q-value and then applies the mask to select actions.

[0021] Includes the following sub-steps:

[0022] S31. Generate mask components based on lane boundary constraints to shield lane change actions that exceed the road boundary;

[0023] S32. Generate mask components based on speed limit constraints to mask acceleration and deceleration actions that violate speed limits;

[0024] S33. Generate mask components based on collision time-to-time (TTC) detection to shield actions that may cause collision risks;

[0025] S34. Combine the above mask components to generate the final Q-value mask vector;

[0026] S35. The temporal occupancy grid extracts spatial features through convolutional layers, normalized scalar inputs are encoded through fully connected layers, and the fused outputs 5 Q values;

[0027] S36. Apply a Q-value mask to the Q-value output to block Q-values ​​that cannot be acted upon. Perform a maximization operation on the masked Q-values ​​and select the optimal action.

[0028] S4. Use a double-buffered experience replay strategy for network training.

[0029] Includes the following sub-steps:

[0030] S41. Initialize the Good buffer and Bad buffer;

[0031] S42. Use an ε-greedy strategy to explore, selecting only from the action subspace allowed by the mask;

[0032] S43. Execute the complete trajectory until the termination state, and calculate the Q-value target for each time step in reverse from the termination reward;

[0033] S44. Based on whether the trajectory was successful or not, store the transferred sample into the corresponding buffer;

[0034] S45. Perform gradient updates by sampling mini-batch evenly from both buffers.

[0035] This invention provides an autonomous driving lane change decision system based on Q-value masks, comprising:

[0036] Status perception module: Acquires vehicle status and surrounding environment occupancy information;

[0037] High-level decision-making module: Learns long-term lane change strategies based on deep Q-networks;

[0038] Low-level control module: integrates system constraints, prior knowledge, and collision detection to generate Q-value masks;

[0039] Q-masking interface module: applies low-level masks to high-level Q-value output;

[0040] Action execution module: Executes the selected high-level action.

[0041] This invention ensures zero collision rate during training and testing by using Q-value masks, simplifies reward function design, improves training efficiency and security, and has good generalization ability and industrial application prospects. Attached Figure Description

[0042] Figure 1 This is a flowchart of the overall framework for autonomous driving lane change decision-making based on Q-value masks proposed in this invention.

[0043] Figure 2 A flowchart for generating Q-value masks for a mask generator;

[0044] Figure 3 A schematic diagram of the mechanism of the Q-value mask action generator;

[0045] Figure 4 This is a network architecture diagram of a deep Q-network;

[0046] Figure 5 This is a schematic diagram of a double-buffered experience replay training strategy. Detailed Implementation

[0047] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0048] S1. Constructing a hierarchical decision-making architecture

[0049] like Figure 1 As shown, this invention adopts a hierarchical decision architecture, which decomposes the lane change decision problem into two levels: high-level decision learning and low-level control execution. The two levels are tightly integrated through Q-masking technology.

[0050] S11. Establish a high-level decision-making module

[0051] The high-level module employs a Deep Q-Network (DQN) to learn long-term lane change strategies. Its goal is to learn a strategy. This enables vehicles to make optimal lane-changing decisions over a long timeframe, reaching the target lane and location in the shortest possible time.

[0052] The input to the high-level module is the vehicle status. The output is the Q-value estimate for 5 discrete actions at higher levels:

[0053]

[0054] in For network parameters, the action space is... ;

[0055] N (No-op): Maintain current speed and lane;

[0056] A (Accelerate): Accelerate within the current lane. (e.g., 2 m / s²);

[0057] D (Decelerate): Decelerate within the current lane. (e.g., 2 m / s²);

[0058] L (Left): Change lanes to the adjacent lane on the left;

[0059] R (Right): Change to the adjacent lane on the right.

[0060] S12. Establish low-level control modules

[0061] The lower-level module integrates the following three types of information:

[0062] Prior knowledge: Road geometry (number of lanes, lane boundaries);

[0063] System constraints: speed limit ;

[0064] Safety Inspection: Collision Risk Assessment Based on Time-of-Collision (TTC);

[0065] The function of the low-level module is to perform operations at each time step. According to the current state Assess the feasibility and safety of each high-level action.

[0066] S13. Q-masking interface design

[0067] The Q-masking interface converts the judgment results of the lower-level module into a mask vector. ,in:

[0068] Indicates action In state The following is feasible and safe;

[0069] Indicates action It should be blocked;

[0070] The number 5 corresponds to 5 optional actions.

[0071] When selecting actions, only actions allowed by the mask are considered:

[0072]

[0073] This design ensures that the network explores and selects only from feasible and safe action subspaces during training and testing.

[0074] S2. Design vehicle state representation and high-level discrete motion space

[0075] This step defines the input state representation and output action space of the high-level decision-making module.

[0076] S21. Collect and normalize the vehicle's internal status.

[0077] The vehicle's internal state includes three scalar variables:

[0078] : Current vehicle speed (m / s);

[0079] : The lane number at the current moment;

[0080] : The distance (m) from the target location at the current moment;

[0081] To enhance the model's generalization ability, scalar information is normalized:

[0082]

[0083] in Total number of lanes (numbered from 0 to L-1 from right to left):

[0084]

[0085] in The total distance from the starting point to the target. This represents the current distance traveled. The normalized value decreases from 1 (starting point) to 0 (target point).

[0086] The advantages of normalization design are that it makes the model robust to parameters such as the number of lanes, distance, and speed range, and supports zero-sample transfer to different configuration scenarios.

[0087] S22. Construct the occupying grid

[0088] A binary occupancy grid is constructed around the vehicle to represent its surrounding environment. :

[0089] Longitudinal range: front Rice, behind rice;

[0090] Horizontal range: left side One lane, right side One lane;

[0091] Vertical resolution: Meters per grid (e.g., 2.5m);

[0092] Horizontal resolution: one grid per lane;

[0093] Grid size:

[0094]

[0095] Grid definition: This cell contains other vehicles; this cell is empty.

[0096]

[0097] S23. Timing Occupation Raster Construction

[0098] To capture the relative motion information of surrounding vehicles, occupancy grids from multiple historical time steps are fused as input:

[0099]

[0100] in For historical frames, This is the time step interval. For example, This indicates the use of a grid with four consecutive time steps.

[0101] The final state is represented as:

[0102]

[0103] S24. Define the high-level discrete action space

[0104] The high-level decision-making module outputs five types of discrete actions:

[0105] N (No-op): Maintain current speed and lane;

[0106] A (Accelerate): Accelerate within the current lane. (e.g., 2 m / s²);

[0107] D (Decelerate): Decelerate within the current lane. (e.g., 2 m / s²);

[0108] L (Left): Change lanes to the adjacent lane on the left;

[0109] R (Right): Change to the adjacent lane on the right;

[0110] Action space: ;

[0111] Each action corresponds to a high-level decision intention, while the actual execution is completed by a low-level controller. This discrete action space design simplifies the learning problem while retaining sufficient decision-making flexibility.

[0112] S3. The lower-level module generates and applies the Q-value mask.

[0113] This step describes the core mechanism of Q-masking, including the complete process of mask generation, deep Q-network processing, and mask application, such as... Figure 2 As shown. The mechanism of the Q-value mask action generator is as follows: Figure 3 As shown.

[0114] The lower-level module depends on the current state Generate Q-value mask .

[0115] S31. Lane Boundary Constraints

[0116] Left lane boundary detection:

[0117] Already in the leftmost lane:

[0118]

[0119] Right lane boundary detection:

[0120] Already in the rightmost lane

[0121]

[0122] S32. Speed ​​Limit Constraints

[0123] Acceleration constraints:

[0124]

[0125] Deceleration constraints:

[0126]

[0127] S33. Collision Detection Based on TTC

[0128] For lane change actions (L, R), calculate the collision time (TimeToCollision):

[0129] Step 1: Determine the target lane

[0130] If the action is L, the target lane is ;

[0131] If the action is R, the target lane is ;

[0132] Step 2: Identify nearby vehicles in the target lane

[0133] From occupying grid Extract the target lane column and identify the nearest vehicles in front and behind.

[0134] Step 3: Calculate TTC

[0135] For the vehicle ahead:

[0136]

[0137] For vehicles behind:

[0138]

[0139] in The distance is relative. The speed of nearby vehicles.

[0140] Step 4: Collision Risk Assessment

[0141]

[0142] in For safety thresholds (e.g., 10 seconds).

[0143] S34. Masking

[0144] Taking into account the above constraints, the final mask vector is generated:

[0145] No operation is always allowed:

[0146]

[0147] Final mask vector:

[0148]

[0149] This mask ensures that all permitted actions simultaneously meet physical constraints and security requirements.

[0150] S35. Deep Q-Network Architecture and Q-Value Output

[0151] like Figure 4 As shown, the deep Q-network adopts the following architecture:

[0152] Convolutional Feature Extraction: Temporal Occupation Grid Input convolutional layer:

[0153]

[0154] The kernel size is... (like The number of output channels is Flattening operation:

[0155]

[0156] Scalar feature encoding: Normalized scalar input is passed through a fully connected layer:

[0157]

[0158] Output dimension is (e.g., 10).

[0159] Feature fusion: splicing convolutional features and scalar features:

[0160]

[0161] Q-value output layer: Outputs 5 Q values ​​through a fully connected layer:

[0162]

[0163] in These are network parameters.

[0164] S36. Q-masking Application and Action Selection

[0165] Q-masking application: Applying a mask vector to the Q-value output:

[0166]

[0167] In actual implementation, a minimum value (such as -1e9) is used instead. .

[0168] Exercise selection: During the training phase, the following exercises will be used. -greedy strategy:

[0169]

[0170] A greedy strategy is used during the testing phase:

[0171]

[0172] Key advantages:

[0173] Both exploration and selection are limited to the action subspace allowed by the mask;

[0174] There is no need to design complex terms such as collision penalties and speed limit penalties in the reward function;

[0175] The training and testing process guarantees zero collisions;

[0176] Prior knowledge and safety constraints are directly integrated into the learning process.

[0177] S4. Double-buffered experience replay training

[0178] like Figure 5 As shown, this step employs a double-buffer experience replay strategy for network training to improve learning stability and convergence speed.

[0179] S41. Buffer Initialization and Reward Function

[0180] Initialize two experience playback buffers:

[0181] : Transfer samples of successful trajectories;

[0182] : Transfer samples of stored failure trajectories;

[0183] Each sample is a triplet. (Using complete trajectory backtracking) Among them, The target value is Q-target.

[0184] Reward function design: This invention employs a minimalist sparse reward system, providing a reward only in the termination state.

[0185] Distance from target lane to target lane (number of lanes):

[0186]

[0187] The reward for intermediate steps is 0. The key advantage of this design is:

[0188] No collision penalty design is required (Q-masking guarantees zero collisions);

[0189] No speed limit penalties are required (the mask automatically blocks inappropriate actions);

[0190] No lane boundary penalty design is required (masks are handled automatically);

[0191] Only one sparse termination reward is needed, which greatly simplifies the design of the reward function;

[0192] S42. ε-greedy exploration strategy

[0193] Exploration rate from (e.g., 1.0) Linear annealing to (e.g., 0.1):

[0194]

[0195] in This refers to the number of annealing cycles (e.g., 80% of the total training rounds).

[0196] Action selection is performed as per step S36, ensuring that exploration is limited to actions permitted by the mask, thereby fundamentally avoiding collisions and violations during training.

[0197] S43. Complete Trajectory Execution and Q-Value Target Calculation

[0198] From the initial state Start executing the trajectory until the termination condition is met:

[0199] Success: Reached the target location and was in the target lane. and );

[0200] Failure: Reached the target location but was not in the target lane. and );

[0201] Timeout: Exceeded the maximum time step

[0202] Record trajectory and termination status flag.

[0203] Using the Monte Carlo method, the Q-value objective is calculated backward from the termination reward:

[0204]

[0205] in A discount factor (such as 0.99) makes early decisions prioritize reaching the goal quickly, implicitly encouraging high-speed progress.

[0206] S44. Sample Storage

[0207] Store the transfer samples based on the trajectory results:

[0208] if trajectory successful (l_T=0): fortin[0,T]: store (s_t,a_t,y_t) in B_good; else: fortin[0,T]: store (s_t,a_t,y_t) in B_bad

[0209] S45. Equal Sampling and Gradient Update

[0210] Each training iteration:

[0211] Step 1: Equalize sampling mini-batch

[0212]

[0213] in The batch size (e.g., 64).

[0214] Step 2: Calculate the loss function

[0215]

[0216] Step 3: Gradient Descent Update

[0217]

[0218] in The learning rate (e.g., 0.001).

[0219] Double buffer advantage: This design ensures that the network learns from both successes and failures simultaneously, avoiding getting stuck in local optima due to too many failure samples during the exploration phase, thus improving training stability and the quality of the final policy.

[0220] Example

[0221] To verify the effectiveness of the method of the present invention, a comparative experiment was conducted in the SUMO simulation environment. The experiment was carried out using a four-lane highway scenario in the simulation.

[0222] Environment settings:

[0223] Number of lanes: L=4

[0224] Target distance: D=1500m

[0225] Speed ​​limit: 60km / h

[0226] Traffic density: 1000 vehicles / hour, departure probability of each lane: Plane={0.35, 0.30, 0.25, 0.10}

[0227] Time step: 0.1 seconds

[0228] Experimental results:

[0229] Table 1 Comparison of Traffic Efficiency and Collision Rate

[0230] method Average speed (km / h) Collision rate (%) Finite state machine 48.7 0 Model predictive control 49.3 0 DQN 50.1 5.4 PPO 50.7 4.7 DDPG 50.6 3.9 Method of the present invention 52.7 0

[0231] Table 2 Comparison of training efficiency of deep reinforcement learning methods

[0232] method Convergence cycles Convergence time (minutes) DQN 18000 87 PPO 26500 172 DDPG 23000 125 Method of the present invention 8500 26

[0233] Results analysis:

[0234] The comparative experiments in this embodiment fully verify the comprehensive advantages of the method of the present invention. In terms of traffic efficiency, the average speed of this method reaches 52.7 km / h, which is superior to all comparative methods. Simultaneously, it achieves a 0% collision rate throughout the entire training and testing process, while the collision rates of traditional deep reinforcement learning methods (DQN, PPO, DDPG) are all between 3.9% and 5.4%, demonstrating the significant advantage of the Q-masking mechanism in ensuring decision security. Regarding training efficiency, this method converges in only 8500 rounds and 26 minutes, reducing training time by approximately 70% compared to the traditional DQN method (18000 rounds, 87 minutes) and by approximately 85% compared to the PPO method (26500 rounds, 172 minutes). The double-buffered experience replay strategy and Q-masking to limit the exploration space are key to improving sample utilization efficiency. Zero-shot transfer testing further demonstrates that, through the normalized state representation design, the model can adapt to different lane number configurations and different task distances without retraining, maintaining a success rate of 82%-87% and a collision rate of 0%, exhibiting excellent generalization ability and environmental adaptability. The above experimental results fully demonstrate the comprehensive advantages of the method of this invention in terms of traffic efficiency, safety, training efficiency, and generalization ability, providing an efficient, safe, and reliable solution for lane-changing decisions in autonomous driving.

Claims

1. A lane-changing decision-making method for autonomous driving based on Q-value masks, characterized in that, Includes the following steps: S1. Construct a hierarchical decision-making architecture, including a high-level decision-making module and a low-level control module, and achieve tight integration through the Q-masking interface; S2. Design the vehicle state representation and high-level discrete action space, where the state includes normalized scalar information and temporal occupancy grid; S3. The low-level module generates a Q-value mask based on system constraints, prior knowledge, and security checks. The deep Q-network outputs the Q-value and then applies the mask to select actions. S4. Use a double-buffered experience replay strategy for network training.

2. The autonomous driving lane change decision method based on Q-value mask according to claim 1, characterized in that: Step S1 includes the following sub-steps: S11. Establish a high-level decision-making module and use a deep Q-network to learn long-term lane change strategies; S12. Establish a low-level control module to integrate system constraints, prior knowledge, and collision detection functions; S13. The high-level module and the low-level module are tightly integrated through the Q-masking interface. The interface converts the low-level judgment result into a mask vector, which restricts the high-level network to learn and select only from the safe and feasible action subspace.

3. The autonomous driving lane change decision method based on Q-value mask according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21. Collect vehicle internal status information, including current speed v, lane l, and distance to target d2g, and normalize the scalar information to map it to the [0,1] interval; S22. Construct a binary occupancy grid of the visible area around the vehicle, merge the occupancy grids from multiple historical time steps, and form a temporal occupancy grid representation; S23. Define the high-level discrete action space, including five types of actions: no operation (N), acceleration (A), deceleration (D), left lane change (L), and right lane change (R).

4. The autonomous driving lane change decision method based on Q-value mask according to claim 1, characterized in that: Step S3 includes the following sub-steps: S31. Generate mask components based on lane boundary constraints to shield lane change actions that exceed the road boundary; S32. Generate mask components based on speed limit constraints to mask acceleration and deceleration actions that violate speed limits; S33. Generate mask components based on collision time-to-time (TTC) detection to shield actions that may cause collision risks; S34. Combine the above mask components to generate the final Q-value mask vector; S35. The temporal occupancy grid extracts spatial features through convolutional layers, normalized scalar inputs are encoded through fully connected layers, and the fused outputs 5 Q values; S36. Apply a Q-value mask to the Q-value output to block Q-values ​​that cannot be acted upon. Perform a maximization operation on the masked Q-values ​​and select the optimal action.

5. The autonomous driving lane change decision method based on Q-value mask according to claim 1, characterized in that: Step S4 includes the following sub-steps: S41. Initialize the Good buffer and Bad buffer; S42. Use an ε-greedy strategy to explore, selecting only from the action subspace allowed by the mask; S43. Execute the complete trajectory until the termination state, and calculate the Q-value target for each time step in reverse from the termination reward; S44. Based on whether the trajectory was successful or not, store the transferred sample into the corresponding buffer; S45. Perform gradient updates by sampling mini-batch evenly from both buffers.

6. An autonomous driving lane change decision system based on Q-value masking, used to implement the autonomous driving lane change decision method based on Q-value masking as described in any one of claims 1 to 5, characterized in that, include: Status perception module: Acquires vehicle status and surrounding environment occupancy information; High-level decision-making module: Learns long-term lane change strategies based on deep Q-networks; Low-level control module: integrates system constraints, prior knowledge, and collision detection to generate Q-value masks; Q-masking interface module: applies low-level masks to high-level Q-value output; Action execution module: Executes the selected high-level action.

Images