A method and system for automatic driving lane change decision based on Q-value mask

Abstract

The application discloses a lane change decision-making method and system for automatic driving based on Q-value masking, and belongs to the fields of automatic driving and artificial intelligence. The method adopts a hierarchical decision-making architecture, and decomposes lane change decision-making into two levels of high-level decision-making learning and low-level control execution, and realizes close integration through a Q-masking interface. The method comprises the following steps: constructing a hierarchical decision-making architecture; designing vehicle state representation and a high-level discrete action space, wherein the state comprises normalized scalar information and a time-occupancy grid, and the action comprises five types of no operation, acceleration, deceleration, left lane change and right lane change; a low-level module generates a Q-value mask according to system constraints, prior knowledge and safety detection, and applies the mask to action selection after a deep Q network outputs a Q value; and a double-buffer experience replay strategy is used for network training. The application ensures zero collision rate in the training and testing processes through the Q-value mask, simplifies reward function design, and improves training efficiency and safety.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of automatic driving and artificial intelligence, and particularly relates to a lane change decision-making method and system for automatic driving based on Q-value masking. BACKGROUND

[0002] Lane change is a core capability of automatic driving vehicles in a multi-lane highway environment, directly affecting driving efficiency and safety. Traditional methods include:

[0003] Rule-based and optimization-based methods: such as model predictive control (MPC), trajectory planning, etc., are suitable for handling low-level control problems, but it is difficult to develop complex long-term high-level decision-making strategies.

[0004] End-to-end deep learning methods: directly map from perception inputs to control instructions, but have problems such as training difficulty, large data requirements, and difficulty in ensuring safety.

[0005] Reinforcement learning methods: can learn arbitrary strategies, have advantages such as timely error correction, immediate learning from the environment, no need for expert models or true values, and easy-to-guide agent behavior. However, completely end-to-end learning faces challenges such as large network size, unstable training, and difficulty in avoiding collisions.

[0006] The main limitations of existing technologies are that traditional methods are difficult to form long-term high-level decision-making capabilities, end-to-end learning lacks the use of prior knowledge, training efficiency is low, reward function design is complex, and multiple goals (speed, safety, rule compliance, etc.) need to be balanced. Because of the characteristics of Q-learning methods, entropy is allowed for agents to explore within the action space, and collisions will naturally occur during the exploration process, which does not meet the zero-accident rate requirement of automatic driving, making it difficult to be directly deployed to real vehicles.

[0007] Therefore, there is a need for a method that can effectively combine high-level decision-making learning and low-level control capabilities, and is efficient, safe, and reliable. SUMMARY

[0008] The application discloses a lane change decision-making method and system for automatic driving based on Q-value masking. The method uses a hierarchical decision-making architecture to decompose lane change decision-making into two levels: high-level decision-making learning and low-level control execution, and realizes close integration through a Q-masking interface.

[0009] A lane change decision-making method for automatic driving based on Q-value masking, comprising the following steps:

[0010] S1. Construct a hierarchical decision-making architecture, including a high-level decision-making module and a low-level control module, and realize close integration through a Q-masking interface;

[0011] Including the following sub-steps:

[0012] S11. Establish a high-level decision module, using deep Q network to learn long-term lane change strategy;

[0013] S12. Establish a low-level control module, integrating system constraints, prior knowledge and collision detection function;

[0014] S13. Through Q-masking interface to realize the close integration of high-level module and low-level module, the interface will convert the low-level judgment result into a mask vector, limiting the high-level network to learn and select only from the safe and feasible action subspace.

[0015] S2. Design vehicle state representation and high-level discrete action space, state contains normalized scalar information and time-series occupancy grid;

[0016] Including the following sub-steps:

[0017] S21. Collect vehicle internal state information, including current speed v, lane l, distance to target distance d2g, and normalize the scalar information, mapping to [0, 1] interval;

[0018] S22. Construct a binary occupancy grid for the visible area around the vehicle, fuse multiple historical time step occupancy grids to form a time-series occupancy grid representation;

[0019] S23. Define high-level discrete action space, including no operation N, acceleration A, deceleration D, left lane change L, right lane change R five types of action.

[0020] S3. Low-level module generates Q-value mask according to system constraints, prior knowledge and safety detection, deep Q network outputs Q value and applies mask for action selection;

[0021] Including the following sub-steps:

[0022] S31. Generate mask component based on lane boundary constraint, shield lane change action beyond road boundary;

[0023] S32. Generate mask component based on speed limit constraint, shield acceleration and deceleration action that violates speed limit;

[0024] S33. Generate mask component based on time to collision TTC detection, shield action that leads to collision risk;

[0025] S34. Fuse the above mask components to generate the final Q-value mask vector;

[0026] S35. Time-series occupancy grid extracts spatial features through convolution layer, normalized scalar input encodes features through fully connected layer, and outputs 5 Q values after fusion;

[0027] S36. Apply the Q-value mask to the Q-value output, mask the inactionable Q-values, perform a maximization operation on the masked Q-values, and select the optimal action.

[0028] S4. Network training using double-buffered experience replay policy.

[0029] comprising the following sub-steps:

[0030] S41. Initialize the Good buffer and the Bad buffer;

[0031] S42. Use the epsilon-greedy policy for exploration, and select only from the action subspace allowed by the mask;

[0032] S43. Perform a full trajectory until the terminal state, and calculate the Q-value target for each time step from the terminal reward;

[0033] S44. Store the transition samples into the corresponding buffer according to whether the trajectory is successful or not;

[0034] S45. Equally sample a mini-batch from both buffers for gradient update.

[0035] The application provides an automatic driving lane change decision system based on Q-value mask, comprising:

[0036] State perception module: obtain vehicle state and surrounding environment occupancy information;

[0037] High-level decision module: learn long-term lane change strategy based on deep Q network;

[0038] Low-level control module: integrate system constraints, priori knowledge and collision detection to generate Q-value mask;

[0039] Q-masking interface module: apply the low-level mask to the high-level Q-value output;

[0040] Action execution module: execute the selected high-level action.

[0041] The application ensures zero collision rate in the training and testing processes through the Q-value mask, simplifies the reward function design, improves the training efficiency and safety, has good generalization ability and industrial application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The application provides an automatic driving lane change decision system based on Q-value mask, comprising:

[0043] Figure 2 The application provides an automatic driving lane change decision system based on Q-value mask, comprising:

[0044] Figure 3 A schematic diagram of the mechanism of the Q-value mask action generator;

[0045] Figure 4 This is a network architecture diagram of a deep Q-network;

[0046] Figure 5 This is a schematic diagram of a double-buffered experience replay training strategy. Detailed Implementation

[0047] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0048] S1. Constructing a hierarchical decision-making architecture

[0049] like Figure 1 As shown, this invention adopts a hierarchical decision architecture, which decomposes the lane change decision problem into two levels: high-level decision learning and low-level control execution. The two levels are tightly integrated through Q-masking technology.

[0050] S11. Establish a high-level decision-making module

[0051] The high-level module employs a Deep Q-Network (DQN) to learn long-term lane change strategies. Its goal is to learn a strategy. This enables vehicles to make optimal lane-changing decisions over a long timeframe, reaching the target lane and location in the shortest possible time.

[0052] The input to the high-level module is the vehicle status. The output is the Q-value estimate for 5 discrete actions at higher levels:

[0053]

[0054] in For network parameters, the action space is... ;

[0055] N (No-op): Maintain current speed and lane;

[0056] A (Accelerate): Accelerate within the current lane. (e.g., 2 m / s²);

[0057] D (Decelerate): Decelerate within the current lane. (e.g., 2 m / s²);

[0058] L (Left): Change to the adjacent lane on the left;

[0059] R (Right): Change to the adjacent lane on the right.

[0060] S12. Establish low-level control module

[0061] The low-level module integrates the following three types of information:

[0062] Prior knowledge: road geometry (number of lanes, lane boundaries);

[0063] System constraints: speed limit ;

[0064] Safety detection: collision risk assessment based on time to collision (TTC);

[0065] The function of the low-level module is to determine the feasibility and safety of each high-level action at each time step , according to the current state .

[0066] S13. Q-masking interface design

[0067] The Q-masking interface converts the judgment results of the low-level module into a mask vector , where:

[0068] represents the action feasible and safe in state ;

[0069] represents the action should be masked;

[0070] The 5 in corresponds to 5 optional actions.

[0071] When selecting an action, only consider the actions allowed by the mask:

[0072]

[0073] This design ensures that the network explores and selects only from the feasible and safe action subspace during training and testing.

[0074] S2. Design vehicle state representation and high-level discrete action space

[0075] This step defines the input state representation and output action space of the high-level decision module.

[0076] S21. Collect vehicle internal state and normalize

[0077] The vehicle internal state includes three scalar variables:

[0078] : current vehicle speed (m / s);

[0079] : Lane number where current time is located

[0080] : Distance from target position (m) where current time is located

[0081] To enhance the generalization ability of the model, normalize the scalar information:

[0082]

[0083] where L is the total number of lanes (numbered from right to left as 0 to L-1):

[0084]

[0085] where L is the total distance from the starting point to the target, is the current distance traveled. The normalized value decreases from 1 (starting point) to 0 (target point).

[0086] Advantages of normalization design: make the model robust to lane number, distance, speed range, etc. Support zero-shot transfer to different configuration scenarios.

[0087] S22. Build occupancy grid

[0088] Build a binary occupancy grid of the surrounding environment centered on the vehicle :

[0089] Longitudinal range: front meters, rear meters;

[0090] Lateral range: left lanes, right lanes;

[0091] Longitudinal resolution: meters per grid (e.g. 2.5m);

[0092] Lateral resolution: one grid per lane;

[0093] Grid size:

[0094]

[0095] Grid definition: the grid is empty if there is another vehicle in it:

[0096]

[0097] S23. Time series occupancy grid construction

[0098] To capture the relative motion information of surrounding vehicles, multiple history time steps of occupancy grid are fused as input:

[0099]

[0100] where is the history frame number, is the time step interval. For example, denotes the grid using consecutive 4 time steps.

[0101] The final state is represented as:

[0102]

[0103] S24. Define high-level discrete action space

[0104] The high-level decision module outputs 5 types of discrete actions:

[0105] N (No-op): Keep the current speed and lane;

[0106] A (Accelerate): Accelerate in the current lane (e.g. 2 m / s2);

[0107] D (Decelerate): Decelerate in the current lane (e.g. 2 m / s2);

[0108] L (Left): Change to the left adjacent lane;

[0109] R (Right): Change to the right adjacent lane;

[0110] Action space: ;

[0111] Each action corresponds to a high-level decision intention, and the specific execution is completed by the low-level controller. This discrete action space design simplifies the learning problem while retaining sufficient decision flexibility.

[0112] S3. Low-level module generates Q-value mask and applies

[0113] This step describes the core mechanism of Q-masking, including the complete process of mask generation, deep Q network processing, and mask application, as shown in Figure 2 . The mechanism of Q-value mask action generator is shown in Figure 3 .

[0114] The low-level module generates a Q-value mask according to the current state .

[0115] S31. Lane boundary constraint

[0116] Left lane boundary detection:

[0117] Has been in the leftmost lane:

[0118]

[0119] Right lane boundary detection:

[0120] Has been in the rightmost lane

[0121]

[0122] S32. Speed limit constraint

[0123] Acceleration constraint:

[0124]

[0125] Deceleration constraint:

[0126]

[0127] S33. Collision detection based on TTC

[0128] For lane-changing action (L, R), calculate Time To Collision (TTC):

[0129] Step 1: Determine target lane

[0130] If action is L, target lane is ;

[0131] If action is R, target lane is ;

[0132] Step 2: Identify neighboring vehicles in target lane

[0133] Extract target lane column from occupancy grid , identify nearest vehicles in front and back.

[0134] Step 3: Calculate TTC

[0135] For front vehicle:

[0136]

[0137] For back vehicle:

[0138]

[0139] where is relative distance, is neighboring vehicle speed.

[0140] Step 4: Collision risk judgment

[0141]

[0142] where is a safety threshold (e.g., 10 seconds).

[0143] S34. Mask fusion

[0144] With the above constraints, the final mask vector is generated:

[0145] No operation is always allowed:

[0146]

[0147] Final mask vector:

[0148]

[0149] This mask ensures that all allowed actions simultaneously satisfy physical constraints and safety requirements.

[0150] S35. Deep Q network architecture and Q value output

[0151] As shown in Figure 4 , the deep Q network adopts the following architecture:

[0152] Convolutional feature extraction: time-occupancy grid Input convolutional layer:

[0153]

[0154] where the convolution kernel size is (e.g., 3x3x3),and the number of output channels is . Flatten operation:

[0155] Scalar feature encoding: normalized scalar input passes through a fully connected layer:

[0156]

[0157] The output dimension is

[0158] (e.g., 10).

[0159] Feature fusion: concatenate convolutional features and scalar features:

[0160]

[0161] Q value output layer: output 5 Q values through a fully connected layer:

[0162]

[0163] where is the network parameter.

[0164] S36. Q-masking application with action selection

[0165] Q-masking application: apply the mask vector to the Q-value output:

[0166]

[0167] In practical implementation, replace the minimum value (such as -1e9) with .

[0168] Action selection: use -greedy policy in training stage:

[0169]

[0170] Use greedy policy in test stage:

[0171]

[0172] Key advantages:

[0173] Both exploration and selection are limited within the action subspace allowed by the mask;

[0174] No need to design complex items such as collision penalty, speed limit penalty, etc. in the reward function;

[0175] The training and test processes guarantee zero collision;

[0176] Prior knowledge and safety constraints are directly integrated into the learning process.

[0177] S4. Double-buffer experience replay training

[0178] As shown in Figure 5 , this step uses a double-buffer experience replay strategy for network training to improve learning stability and convergence speed.

[0179] S41. Buffer initialization and reward function

[0180] Initialize two experience replay buffers:

[0181] : store transition samples of successful trajectories;

[0182] : store transition samples of failed trajectories;

[0183] Each sample is a triple (Using full trajectory backtracking) where, Q-target is the Q value target.

[0184] Reward function design: The present invention adopts a minimalist sparse reward, only giving reward at the termination state:

[0185] Distance to target lane is the distance to the target lane:

[0186]

[0187] Intermediate step reward is 0. The key advantage of this design is that:

[0188] No need to design collision penalty (Q-masking guarantees zero collision);

[0189] No need to design speed limit penalty (masking automatically shields illegal actions);

[0190] No need to design lane boundary penalty (masking automatically handles);

[0191] Only one sparse termination reward, greatly simplifying the reward function design;

[0192] S42. ε-greedy exploration strategy

[0193] Exploration rate Linearly anneal from (eg. 1.0) to (eg. 0.1):

[0194]

[0195] where is the number of annealing cycles (eg. 80% of total training rounds).

[0196] Action selection is performed according to step S36, ensuring that only actions allowed by the mask are explored, fundamentally avoiding collisions and illegal actions during training.

[0197] S43. Full trajectory execution and Q value target calculation

[0198] Start executing the trajectory from the initial state until the termination condition:

[0199] Success: Reach the target location and be in the target lane ( and );

[0200] Failure: Reach the target location but not in the target lane ( and );

[0201] Time-out: exceed maximum time step

[0202] Recorded trajectory and termination state flag.

[0203] Backward calculate Q value target from termination reward using Monte Carlo method:

[0204]

[0205] where is a discount factor (e.g. 0.99) that makes early decisions more important for fast reaching the goal, implicitly encouraging high-speed driving.

[0206] S44. Sample storage

[0207] Store transition samples according to trajectory results:

[0208] if trajectory succeeds (l_T=0): for t in [0, T]: store (s_t, a_t, y_t) in B_good else: for t in [0, T]: store (s_t, a_t, y_t) in B_bad

[0209] S45. Balanced sampling and gradient update

[0210] Each training iteration:

[0211] Step 1: Balanced sampling mini-batch

[0212]

[0213] where is the batch size (e.g. 64).

[0214] Step 2: Calculate loss function

[0215]

[0216] Step 3: Gradient descent update

[0217]

[0218] where is the learning rate (e.g. 0.001).

[0219] Double-buffer advantage: this design ensures that the network learns both successful and failed experiences at the same time, avoiding falling into local optimum due to too many failed samples in the exploration stage, improving training stability and final policy quality.

[0220] Embodiment

[0221] To verify the effectiveness of the method, comparative experiments were conducted in the SUMO simulation environment. The experiment was completed using a four-lane highway scene in the simulation.

[0222] Environment settings:

[0223] Number of lanes: L = 4

[0224] Target distance: D = 1500m

[0225] Speed limit: 60km / h road speed limit

[0226] Traffic density: 1000 vehicles / hour, each lane Plane = {0.35, 0.30, 0.25, 0.10}

[0227] Time step: 0.1 seconds

[0228] Experimental results:

[0229] Table 1 Comparison of traffic efficiency and collision rate

[0230] Method Average speed (km / h) Collision rate (%) Finite state machine 48.7 0 Model predictive control 49.3 0 DQN 50.1 5.4 PPO 50.7 4.7 DDPG 50.6 3.9 The method of the invention 52.7 0

[0231] Table 2 Comparison of training efficiency of deep reinforcement learning method

[0232] Method Convergence rounds Convergence time (minutes) DQN 18000 87 PPO 26500 172 DDPG 23000 125 The method of the invention 8500 26

[0233] Result analysis:

[0234] The comparative experiments of the embodiment fully verify the comprehensive advantages of the method. In terms of traffic efficiency, the average speed of the method reaches 52.7 km / h, which is better than all the comparative methods, and at the same time, the collision rate of the whole process of training and testing is 0%, while the collision rates of the traditional deep reinforcement learning methods (DQN, PPO and DDPG) are all between 3.9% and 5.4%, which proves that the Q-masking mechanism has a significant advantage in ensuring the safety of decision-making. In terms of training efficiency, the method only needs 8500 rounds and 26 minutes to converge, which is about 70% less than the training time of the traditional DQN method (18000 rounds and 87 minutes) and about 85% less than the training time of the PPO method (26500 rounds and 172 minutes), and the double-buffer experience replay strategy and the Q-masking limited exploration space are the key to improving the sample utilization efficiency. The zero-sample migration test further shows that through the design of the normalized state representation, the model can adapt to different lane number configurations and different task distances without retraining, the success rate remains at 82%-87%, and the collision rate is always 0%, which shows excellent generalization ability and environmental adaptability. The above experimental results fully prove the comprehensive advantages of the method in terms of traffic efficiency, safety, training efficiency and generalization ability, and provide an efficient, safe and reliable solution for automatic driving lane changing decision-making.

Claims

1. A Q-value mask based autonomous lane change decision method, characterized in that, Comprising the following steps: S1. Construct a hierarchical decision architecture, including a high-level decision module and a low-level control module, and realize close integration through a Q-masking interface; S2. Design a vehicle state representation and a high-level discrete action space, the state containing normalized scalar information and a time-series occupancy grid; Step S2 comprises the following sub-steps: S21. Collect vehicle internal state information, including current speed v, lane l, and distance to goal d2g, and normalize the scalar information to the [0, 1] interval; S22. Construct a binary occupancy grid for the visible area around the vehicle, fuse multiple historical time step occupancy grids to form a time-series occupancy grid representation; S23. Define a high-level discrete action space, including five types of actions: no operation N, acceleration A, deceleration D, left lane change L, and right lane change R; S3. The low-level module generates a Q-value mask according to system constraints, prior knowledge, and safety detection, and the deep Q network outputs Q values, which are then applied to the mask for action selection; Step S3 comprises the following sub-steps: S31. Generate a mask component based on lane boundary constraints to mask lane change actions that exceed road boundaries; S32. Generate a mask component based on speed limit constraints to mask acceleration and deceleration actions that violate speed limits; S33. Generate a mask component based on time-to-collision TTC detection to mask actions that pose collision risks; S34. Fuse the above mask components to generate a final Q-value mask vector; S35. The time-series occupancy grid extracts spatial features through a convolutional layer, and the normalized scalar input is encoded through a fully connected layer. The fused output is five Q values; S36. Apply the Q-value mask to the Q-value output to mask the Q-values of infeasible actions. Perform a maximization operation on the masked Q-values to select the optimal action; S4. Use a double-buffered experience replay strategy for network training. 2.The Q-value mask based automatic lane change decision method of claim 1, wherein: Step S1 comprises the following sub-steps: S11. Establish a high-level decision module to learn long-term lane change strategies using a deep Q network; S12. Establish a low-level control module to integrate system constraints, prior knowledge, and collision detection functions; S13. Realize close integration of the high-level module and the low-level module through a Q-masking interface. The interface converts low-level judgment results into a mask vector, limiting the high-level network to learn and select from only a safe and feasible action subspace. 3.The Q-value mask based automatic lane change decision method of claim 1, wherein: Step S4 comprises the following sub-steps: S41. Initialize the Good buffer and the Bad buffer; S42. Use an ε-greedy strategy for exploration, selecting only from the action subspace allowed by the mask; S43. Execute the complete trajectory until the termination state, and calculate the Q-value target for each time step from the termination reward; S44. Store the transition samples in the corresponding buffer according to whether the trajectory was successful or not; S45. Equally sample mini-batches from both buffers for gradient updates.

4. A Q-value mask based autonomous lane change decision system for implementing the Q-value mask based autonomous lane change decision method of any one of claims 1 to 3, characterized in that, Comprise: State perception module: acquire vehicle state and surrounding environment occupancy information; High-level decision module: learn long-term lane change strategies based on a deep Q network; Low-level control module: integrate system constraints, prior knowledge, and collision detection to generate a Q-value mask; Q-masking interface module: applies low-level masks to high-level Q-value outputs; Action execution module: executes selected high-level actions.

Images