Intelligent vehicle autonomous lane-changing decision system and method based on proximal policy optimization

Abstract

The application discloses an intelligent vehicle autonomous lane-changing decision system and method based on proximal policy optimization, belongs to the technical field of intelligent traffic and automatic driving, and solves technical problems of deficiencies of the prior art in state space design, safety constraint modeling and training strategy optimization. The system comprises a traffic simulation environment module, a state observation and feature construction module, a proximal policy optimization lane-changing decision module, a reward calculation and safety supervision module, a training control and data recording module and a strategy derivation and application module. A training environment is constructed based on a micro-traffic simulation platform, different traffic densities and speed distributions are configured, the training scene is closer to real road traffic, and the strategy has better adaptability under different working conditions.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and autonomous driving technology, specifically relating to an intelligent vehicle autonomous lane-changing decision-making system and method based on near-end strategy optimization. Background Technology

[0002] With the continuous growth of motor vehicle ownership, traffic congestion and traffic accidents have become increasingly prominent problems. Research and statistics show that driver factors play a dominant role in the causes of traffic accidents, with cognitive errors, decision-making errors, and improper operation accounting for a very high proportion of accidents. This has driven research into autonomous driving decision-making technologies focused on safety and efficiency.

[0003] Existing lane-changing decision-making methods for autonomous driving can be mainly divided into two categories: one is rule-based or model-based decision-making methods, such as finite state machines, behavior trees, and model predictive control (MPC); the other is decision-making methods based on deep reinforcement learning.

[0004] Rule-based or model-based methods trigger behaviors such as lane changing, deceleration, or lane keeping through manually set thresholds and logic. They are clear in structure and easy to implement, but in complex traffic scenarios, the rules are difficult to exhaust, have limited adaptability to environmental changes, and are difficult to handle emergencies.

[0005] Deep reinforcement learning-based decision-making methods utilize deep neural networks to extract features from high-dimensional environmental states. Through repeated trial and error in a simulation environment, they learn driving strategies for different scenarios. Existing research has employed algorithms such as DQN, DDPG, and TD3 to implement vehicle behavior or lane-changing decisions, achieving some success in simulation environments. Other works have combined traffic simulation platforms like SUMO with the Gym environment, using the PPO algorithm to train lane-changing strategies, demonstrating the feasibility of deep reinforcement learning in lane-changing tasks.

[0006] However, existing technical solutions still have shortcomings in state space design, safety constraint modeling, and training strategy optimization. For example, they can only be verified in simple scenarios, lack dynamic safety distance and collision prediction mechanisms, and have slow training convergence speed.

[0007] Existing autonomous driving lane-changing decision-making methods generally suffer from problems in engineering applications, such as limited training stability and sample efficiency, insufficient dynamic safety constraints and exploration safety, weak coupling with traffic flow micro-simulation, and a lack of curriculum-based progressive design in the training process. On the one hand, existing solutions based on algorithms such as Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Double-Delay Deep Deterministic Policy Gradient (TD3) are prone to policy update oscillations and slow convergence in high-dimensional states and complex traffic scenarios, requiring a large number of simulation samples and long training time to obtain usable policies, resulting in high training costs and low efficiency. On the other hand, many methods simply add collision penalties to the reward function. The lack of clear modeling for "dynamic safe distance" and "short-term collision risk" leads to frequent unsafe behaviors such as rear-end collisions and sudden braking during the exploration phase. The proportion of effective samples is low, and the strategy is prone to getting trapped in local optima. At the same time, some studies use self-built simplified simulation environments or only utilize a small amount of surrounding vehicle information, which does not fully integrate the urban traffic simulation software SUMO with the microscopic characteristics of traffic flow. It fails to fully consider factors such as traffic density and speed distribution, resulting in insufficient adaptability and generalization ability of lane-changing strategies under different working conditions. In addition, existing methods are mostly trained directly in fixed high-difficulty scenarios. In the early stage of training, the agent frequently collides or fails the task, effective reward signals are scarce, the training cycle is long, and the cost of adjusting participation and verification is high. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides an intelligent vehicle autonomous lane-changing decision-making system and method based on near-end strategy optimization.

[0009] Firstly, the system includes:

[0010] The system includes a traffic simulation environment module, a state observation and feature construction module, a near-end strategy optimization lane-changing decision module, a reward calculation and safety supervision module, a training control and data recording module, and a strategy export and application module.

[0011] Traffic simulation environment module: Based on the urban traffic simulation software SUMO, the initial simulation environment and traffic flow parameters are constructed to provide controllable overtaking conditions;

[0012] State observation and feature construction module: At each simulation time step, the module obtains the driving information of the vehicle and the vehicle in front from the traffic simulation environment module through the traffic control interface, and combines and normalizes it to form a state vector;

[0013] The near-end strategy optimization lane-changing decision module includes a strategy network submodule and a value network submodule. After receiving the state vector, the strategy network submodule outputs the probabilities of two discrete actions: "keeping the lane" and "changing lanes". The value network submodule estimates the long-term reward of the current state.

[0014] Reward Calculation and Safety Supervision Module: Evaluates the simulation results at each step, calculates the composite reward and constructs a dynamic safety distance that varies with speed, determines whether there is a risk of collision in the future, and applies penalties in advance or terminates the current round when there is a risk of collision.

[0015] Training control and data recording module: Organizes the entire reinforcement learning training process to form a complete closed-loop training process;

[0016] Policy Export and Application Module: After training convergence, the policy network parameters are exported as model files and integrated into the simulation platform or hardware-in-the-loop system of the actual application.

[0017] Furthermore, the traffic simulation environment module initializes the simulation environment and constructs traffic flow parameters by: constructing a two-lane straight road scenario, setting scenario parameters, including road length, number of lanes, lane width, and lane speed limit, and setting the arrival rate, vehicle type, and speed range of the background traffic flow through a configuration file.

[0018] Furthermore, when the state observation and feature construction module constructs the state vector, it forms a six-dimensional state vector. This six-dimensional state vector is then normalized to form the state vectors. The six-dimensional state vectors are as follows:

[0019] The longitudinal position of the vehicle is used to represent the vehicle's progress on the road;

[0020] The longitudinal speed of the vehicle is used to represent its current speed.

[0021] The longitudinal relative distance between the vehicle in front and the vehicle in front in the current lane is used to measure the following distance;

[0022] The longitudinal relative speed between the vehicle in front and the vehicle in front in the current lane is used to determine the trend of changing following distance;

[0023] The longitudinal relative distance between the vehicle in front in the target lane and the vehicle itself is used to assess the safety clearance in the target lane;

[0024] The longitudinal relative speed between the vehicle in front and the vehicle in the target lane is used to determine the change in the safety clearance of the target lane over time.

[0025] Furthermore, in the policy network submodule, the state vector is input into the policy network, and the policy network outputs the probability distribution of the two actions of "keeping the lane" and "changing lanes". The policy network adopts a two- or three-layer fully connected neural network structure, and the hidden layer uses a non-linear activation function.

[0026] When training the policy network, a near-end policy optimization algorithm is used, which limits the magnitude of each update by pruning the ratio of the output probabilities of the new and old policies.

[0027] Furthermore, the compound reward is represented as:

[0028] Compound rewards are represented as follows:

[0029] ;

[0030] in, For the first Total reward for each time step; Rewards for space efficiency; Penalty for speed deviation; Penalty for lateral trajectory deviation; For collision and safety penalties; These are the weighting coefficients for each part.

[0031] Furthermore, the training control and data recording module employs a four-stage progressive training strategy, specifically:

[0032] Phase 1: Simple scenario with low traffic flow. Only a small number of low-speed vehicles are placed in front of the vehicle, and there is little background traffic. The main goal is to teach the intelligent agent to perform basic following and safe lane changing while maintaining a safe distance.

[0033] Phase Two: Fixed Overtaking Task Scenario. While maintaining low traffic density, a significantly slow-moving vehicle in front and a relatively empty target lane are set up to guide the intelligent agent to learn to actively initiate lane changes and complete overtaking, thereby improving task completion efficiency.

[0034] Phase 3: Multi-condition randomized scenarios. The number of background vehicles is increased, and the initial positions and speeds of the vehicle and the vehicle in front are randomized, so that the agent experiences a variety of overtaking distances, speed differences, and queue-jumping situations, thereby enhancing the strategy's adaptability to changes in operating conditions.

[0035] Phase 4: High-density complex scenarios. Increase traffic density and add multiple interfering vehicles to further train the agent in a complex interactive environment, enabling it to maintain high safety and success rate under high load and strong interference conditions.

[0036] Furthermore, the training control and data recording module organizes the entire reinforcement learning training process, including: environment initialization, simulation time step progression, state acquisition, action execution, reward recording, advantage function calculation, and triggering the update of the proximal policy optimization algorithm for each round; and records the parameters of each round of training process.

[0037] Secondly, the methods include:

[0038] S1. Initialize the simulation environment and traffic flow parameters, set road geometry information, number of lanes, lane speed limit parameters, and arrival rate and speed distribution of background traffic flow, and start SUMO simulation;

[0039] S2. Set the initial position and speed of the vehicle and the vehicle in front, start a simulation round, and clear the data record of the previous round.

[0040] S3. At each simulation time step, the state observation and feature construction module collects the state of the vehicle and the vehicle in front from the simulation environment and generates a state vector of fixed dimension.

[0041] S4. Input the current state vector into the near-end policy optimization lane-changing decision module, and the policy network outputs the action decision of "keep lane" or "change lane".

[0042] S5. Perform this action in the traffic simulation environment module, and SUMO and its built-in driver model will automatically update the vehicle position and speed.

[0043] S6. The reward calculation and safety supervision module reads the updated vehicle status, calculates the total reward for the current time step, and determines whether the dynamic safety distance has been violated or a collision has occurred. If the dynamic safety distance has been violated or a collision has occurred, the current round is terminated immediately.

[0044] S7, the training control and data recording module records the current step's state acquisition, action execution, reward record and value estimation data. When the number of experience samples reaches a preset threshold, it calls the proximal policy optimization algorithm to update the policy network and value network in multiple rounds.

[0045] S8. Repeat steps S2 to S7 until the number of training rounds or performance indicators reach the preset requirements. Finally, the strategy export and application module exports the final strategy model for subsequent deployment and application.

[0046] The beneficial effects of the system described in this invention are as follows:

[0047] 1. High training stability and good convergence efficiency

[0048] By introducing the Proximal Policy Optimization (PPO) algorithm and employing methods such as... Figure 2 The four-stage progressive training strategy shown (in order: simple low-traffic scenario, fixed overtaking task scenario, multi-condition random scenario, and high-density complex scenario) significantly improves the problems of training instability and slow convergence, and can obtain high-quality lane-changing strategies within a limited training time.

[0049] 2. Safer exploration process

[0050] This invention introduces dynamic safe distance modeling based on an Intelligent Driver Model (IDM) in the reward calculation and safety supervision module, combined with a short-term trajectory prediction method, to identify and constrain potential collision risks in advance. Specifically, the system first calculates the dynamic safe distance under the current conditions based on parameters such as the vehicle's current speed and the relative speed between the vehicle and the vehicle in front. Then, it predicts the longitudinal relative distance between the vehicle and the vehicle in front within several future simulation steps. Once it detects that the current distance or any predicted distance is about to be less than the dynamic safe distance, it provides a large negative reward for the corresponding action during training and can terminate the round early. This filters out a large number of high-risk behaviors during the exploration phase, significantly reduces the proportion of unsafe trajectories in the empirical data, and improves the robustness and safety of the finally learned lane-changing strategy. The training environment is built based on a microscopic traffic simulation platform. By configuring different traffic flow densities and speed distributions, the training scenario is made closer to real road traffic, and the strategy has better adaptability under different conditions.

[0051] 3. Achieve a comprehensive improvement in lane-changing efficiency and safety.

[0052] The simulation results show that the method of the present invention can shorten the overtaking time while ensuring zero collisions throughout the process, achieving a good balance between safety and traffic efficiency.

[0053] 4. The system has a clear structure and is easy to implement in engineering.

[0054] This invention clearly divides the functional modules, which can be easily integrated and deployed on vehicle-road cooperative testing platforms or hardware-in-the-loop platforms, and has good prospects for engineering applications. Attached Figure Description

[0055] Figure 1 This is a structural diagram of the intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the four-stage progressive training strategy in an embodiment of the present invention;

[0057] Figure 3 This is a flowchart of the intelligent vehicle autonomous lane-changing decision-making method based on near-end strategy optimization in an embodiment of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0059] Example 1

[0060] To address the problems existing in the prior art, this embodiment aims to propose an intelligent vehicle autonomous lane-changing decision-making system based on Proximal Policy Optimization (PPO). By constructing a system architecture consisting of a traffic simulation environment module, a state observation and feature construction module, a lane-changing decision-making module based on PPO, a reward calculation and safety supervision module, and a training control and policy derivation module, a PPO-based PPO algorithm with a pruning mechanism is employed, combined with compact six-dimensional state modeling and a multi-layer feedforward neural network, to improve the stability of policy updates and sample utilization, shorten training time, and reduce computational overhead. Furthermore, by designing a composite reward function integrating efficiency rewards, speed deviation penalties, trajectory deviation penalties, and collision safety penalties, and applying it in SUMO... In a simulation environment, a dynamic safe distance model and a short-term predicted path algorithm are constructed by combining an Intelligent Driver Model (IDM) and the Traffic Control Interface (TraCI). Potentially hazardous situations are constrained in advance during training and decision-making, significantly improving the safety of exploration and decision-making while ensuring lane-changing efficiency. Furthermore, a four-stage progressive training strategy is introduced based on the simulation environment, gradually transitioning from low-density, simple operating conditions to high-density, multi-interference scenarios. This allows the agent to steadily accumulate effective experience at each stage, improving convergence quality and the strategy's adaptability to traffic flow changes. Thus, this comprehensively improves existing autonomous driving lane-changing decision-making technology in terms of safety, efficiency, and training stability.

[0061] like Figure 1 As shown, the intelligent vehicle autonomous lane-changing decision-making system in this embodiment mainly consists of the following modules: traffic simulation environment module; state observation and feature construction module; near-end strategy optimization lane-changing decision-making module; reward calculation and safety supervision module; training control and data recording module; and strategy export and application module.

[0062] The traffic simulation environment module is based on the urban traffic simulation software SUMO (Simulation of UrbanMobility). It constructs a two-lane straight road scenario, sets parameters such as road length, number of lanes, lane width, and lane speed limit, and sets the arrival rate, vehicle type, and speed range of background traffic flow through configuration files to provide controllable overtaking conditions.

[0063] The State Observation and Feature Construction Module obtains information such as the speed, position, and lane of the vehicle and the vehicle in front from the traffic simulation environment module at each simulation time step through the Traffic Control Interface (TRACI). It then combines and normalizes this information according to pre-set rules to form a fixed-dimensional state vector, which serves as the input feature of the reinforcement learning agent.

[0064] In a typical two-lane overtaking task, this embodiment compresses the vehicle motion information closely related to lane-changing decisions into a six-dimensional state vector, which is constructed by the state observation and feature construction module in the following manner:

[0065] The longitudinal position of the vehicle is used to represent the vehicle's progress on the road;

[0066] The longitudinal speed of the vehicle is used to represent its current speed.

[0067] The longitudinal relative distance between the vehicle in front and the vehicle in front in the current lane is used to measure the following distance;

[0068] The longitudinal relative speed between the vehicle in front and the vehicle in front in the current lane is used to determine the trend of changing following distance;

[0069] The longitudinal relative distance between the vehicle in front in the target lane and the vehicle itself is used to assess the safety clearance in the target lane;

[0070] The longitudinal relative speed between the vehicle in front and the vehicle in the target lane is used to determine the change in the safety clearance of the target lane over time.

[0071] After appropriate normalization, the above six quantities form the state vector input policy network and value network, which reduces computational complexity while retaining the safety distance and speed relationship information that is crucial for lane-changing decisions.

[0072] In this embodiment, the action space is designed as two discrete actions:

[0073] Action "0": Maintain current lane driving and do not initiate lane change request;

[0074] Action "1": Initiate a lane change request to the target lane. The traffic simulation environment module and the underlying driver model will execute a lane change operation under the premise of satisfying road and dynamic constraints.

[0075] The specific longitudinal acceleration, deceleration, and steering trajectories are automatically calculated by the SUMO internal control model (including the intelligent driver model, etc.). The proximal strategy optimization lane-changing decision module only outputs the instruction on whether to change lanes at each decision time step, transforming the complex continuous control problem into a low-dimensional decision problem, which is convenient for training and engineering implementation.

[0076] The near-end strategy optimization lane-changing decision module internally comprises a policy network submodule and a value network submodule. The policy network submodule receives the state vector and outputs the probability of either "maintain current lane" or "execute lane change". The value network submodule outputs the probability of the state vector at the t-th simulation time step. For input, state vector The value network consists of six state variables (vehicle longitudinal position, vehicle longitudinal velocity, relative distance and velocity to the vehicle in the current lane, and relative distance and velocity to the vehicle in the target lane) described in the "State Space Design" section above, combined in a fixed order. The value network maps this state vector into a scalar using several layers of fully connected neural networks and nonlinear activation functions. This is used to approximate the state under the current policy. The cumulative discounted reward that can be obtained in each future step can be expressed mathematically as:

[0077] ;

[0078] in, For the first The instantaneous reward obtained at the simulation time step (corresponding to the previously defined first simulation time step) (Step-by-step compound reward or its normalized form); This is a discount factor used to gradually reduce the weight of more distant future rewards in the current value during summation; Let be the mathematical expectation operator, representing the averaging operation on all possible state-action trajectories under given policy and environmental randomness; that is, averaging the discounted rewards obtained from a large number of simulation experiments. Therefore, the state vector of the previous time step... Obtained through the value network , can represent the expected value of all future reward discounts when starting from this state and continuing according to the current strategy.

[0079] During the training phase, this embodiment first assigns a reward sequence to each time step based on the reward sequence collected in the simulation environment. Calculate target return Discounted returns are offered.

[0080] ;in, This is the time step number at the end of this round (counting from 0, the total number of steps in this round is...). ), As a discount factor, Indicates the first The immediate reward obtained at each time step. Then, the variance loss function. Gradient descent is used to update the value network parameters, gradually bringing the value network output closer to the true cumulative return; simultaneously, based on the advantage function... Constructing advantage estimation in proximal policy optimization algorithms, combined with pruning objective functions The policy network parameters are updated iteratively in multiple rounds.

[0081] This embodiment employs a proximal policy optimization algorithm for training the policy network. This algorithm improves training stability and sample utilization efficiency by pruning the ratio of the output probabilities of the new and old policies, thus limiting the magnitude of each update.

[0082] During the application phase, the near-end strategy optimization lane-changing decision module directly outputs lane-changing decisions through the policy network based on the current state vector, sending the "keep lane" or "execute lane change" instructions to the traffic simulation environment module or the actual vehicle control system.

[0083] Let the policy network parameters be... The old strategy parameters are , No. The state of the step is The action is The advantage function is The probability ratio of the new and old strategies under this state action is defined as:

[0084] ;

[0085] in, This indicates that the policy network parameters are... At that time, the policy is in the state Select action The conditional probability; This represents the conditional probability of choosing this action under the same state as the old policy. The ratio of the two is used to measure the difference between the policy before and after the update.

[0086] The magnitude of change in this state-action pair.

[0087] The shearing objective function of the near-end policy optimization algorithm can be expressed as:

[0088]

[0089] in, This is a hyperparameter for the cropping range. The function will ratio of probability Limited to Within the interval, This represents the time steps of a batch of trajectories sampled using the current strategy. The empirical average calculation.

[0090] In this embodiment, to balance learning efficiency and driving safety, the pruning mechanism in the near-end policy optimization algorithm was improved. First, based on each time step... Corresponding collision and safety penalties The training samples are divided into normal samples and high-risk samples; a basic pruning factor is used for normal samples. For high-risk samples, a more stringent pruning factor is used. And satisfy Thus, the above probability is higher than The cropping range is no longer fixed. Instead, it is based on the safety level of the sample. or Adaptive switching between them, denoted as .

[0091] When the probability ratio If the value falls outside this interval, it is considered that the policy update is too large, and the corresponding term in the objective function will automatically use the pruned portion. To replace the original item The strategy is only allowed to change slowly near the interval boundaries. For samples judged to be high-risk, because... Smaller values ​​restrict the update magnitude of the strategy around these states, preventing an overly aggressive lane-changing behavior caused by a single gradient update. For ordinary samples, a relatively large pruning interval is used, allowing the strategy to still have enough room for improvement, thereby improving the overall sample utilization efficiency while ensuring training stability and safety.

[0092] Advantage function It is obtained from the difference between cumulative return and the value function, for example: ;

[0093] in, From time The initial discount accumulates into a reward. For value network to state Value estimation. By maximizing the above-mentioned shearing objective function, sample utilization efficiency can be improved while ensuring training stability.

[0094] In this implementation, both the policy network and the value network take the aforementioned six-dimensional state vector as input and employ a two- or three-layer fully connected neural network structure, with common non-linear activation functions used in the hidden layers. The policy network outputs the probability distributions of the two actions, "keeping the lane" and "changing lanes," while the value network outputs a value estimate of the current state.

[0095] The training control and data recording module collects multiple simulation trajectories according to the above method and calculates the cumulative reward at each time step. and dominance function Then call the near-end strategy optimization algorithm, based on The policy network parameters are iteratively updated multiple times, while the value network is updated in the form of mean squared error, so that its fit to the real return gradually converges.

[0096] The reward calculation and safety monitoring module evaluates the simulation results at each step. This module calculates a composite reward based on information such as the vehicle's speed, position, lane changes, and whether a collision occurred. For safety monitoring, this module uses the longitudinal relative distance between the vehicle and the vehicle in front. Relative velocity and the vehicle's current speed By combining an intelligent driver model (IDM), a dynamic safe distance is constructed that varies with speed and relative speed. The short-time trajectory prediction method is used to calculate the workshop distance sequence for several simulation steps within a preset prediction time window. When the current distance or any predicted distance Less than the corresponding dynamic safety distance If a potential collision risk is identified, a large negative reward is applied to the action at that time step in the reward function, and the current simulation round may be terminated early if necessary. The specific calculation formula and judgment conditions for this short-time trajectory prediction method are detailed later in the section "Collision and Safety Penalties". Further details are provided in the subsection.

[0097] Compound rewards can be represented as:

[0098] ;

[0099] in, For the first Total reward for each time step; Rewards for space efficiency; Penalty for speed deviation; Penalty for lateral trajectory deviation; For collision and safety penalties; These are the weighting coefficients for each part.

[0100] (1) Efficiency Rewards For non-terminating time steps, the efficiency reward is zero, meaning that when the current time step... If it is not the end time step of this round, then

[0101] Let the maximum number of moves allowed in this round be... (Corresponding to the reasonable maximum overtaking time). When this round is at the time step Successfully overtook the vehicle in front and resumed steady driving, and the time step Simultaneously, this is the end time step for this round and satisfies... At this final step, a one-time positive reward is given, defined as follows:

[0102]

[0103] in, The basic reward constant for a successful overtaking maneuver. Termination time step. The smaller the value (i.e., the shorter the overtaking time), the better. The larger the value, the more it encourages the agent to complete the overtaking task as quickly as possible while ensuring safety.

[0104] (2) Speed ​​deviation penalty This embodiment will use the vehicle speed. With expected speed The deviation between them is penalized using a quadratic form, for example:

[0105] ;

[0106] in This is the speed deviation penalty coefficient. The larger the speed deviation, the more negative this term becomes, thereby suppressing prolonged low-speed or significant speeding behavior.

[0107] (3) Trajectory deviation penalty To constrain lateral smoothness during lane changes, this embodiment uses the lateral offset between the vehicle and the centerline of the target lane. Set penalties, for example:

[0108] ;

[0109] in, This is the trajectory deviation penalty coefficient. The greater the lateral offset, the stronger this penalty, leading to a smoother lateral trajectory.

[0110] (4) Collisions and safety penalties This embodiment combines an Intelligent Driver Model (IDM) to define a longitudinal dynamic safety distance that varies with speed and relative speed. For example, it can be calculated based on parameters such as minimum static distance, expected headway, vehicle speed, and relative speed. Based on this, in order to predict potential rear-end collisions in advance, this embodiment uses a short-time trajectory prediction method to estimate the inter-vehicle distance for several simulation steps. Assuming that the relative speed is approximately constant within a short time window, then the... The shop distance for each prediction step can be expressed as: ;

[0111] in, This represents the current longitudinal distance between the vehicle and the vehicle in front. Given the current relative speed, For the simulation time step, This is the index for the prediction steps. The upper limit for the number of prediction steps is denoted as . Generally, 5 to 10 steps can be selected based on the simulation time step and vehicle speed, with the corresponding prediction time window typically not exceeding 2 seconds. If a certain... ( ) makes or the current actual distance It is already smaller than If a potential or actual collision risk is identified, a significant negative reward is applied through collision and safety penalties in the reward function. The current round may be terminated if necessary. For example:

[0112] like Or a collision occurs, ;otherwise .

[0113] in, Here, represents the penalty coefficient for collisions and serious unsafe behaviors. To ensure that the collision penalty is significantly greater than other reward items, this embodiment first calculates the upper bound of the absolute values ​​of the efficiency reward, velocity deviation penalty, and trajectory deviation penalty in a single-step simulation, denoted as [missing information]. , and And define the single-step baseline reward magnitude:

[0114] ;

[0115] Based on this, a safety weight coefficient is introduced. The collision penalty coefficient is determined by the following formula: .

[0116] In this embodiment, the upper bound of the absolute value of each sub-reward in a single step does not exceed 1, and the weight... Take 1.0, therefore To ensure that the penalty for a single collision is significantly greater than the positive reward gained from several steps of normal driving, this embodiment selects... ,get Thus, a collision incurs a negative penalty equivalent to approximately 15 time steps of the maximum positive reward, causing the reinforcement learning agent to significantly favor collision avoidance during training. In other road scenarios, those skilled in the art can also adjust the safety weight coefficients... The value of is used to quantitatively adjust the collision penalty intensity in order to achieve a reasonable trade-off between safety and traffic efficiency.

[0117] The training control and data recording module is responsible for organizing the entire reinforcement learning training process, including environment initialization, simulation time step progression, state acquisition, action execution, reward recording, advantage function calculation, and triggering updates of the proximal policy optimization algorithm for each round, forming a complete closed-loop training process.

[0118] This embodiment designs a four-stage progressive training strategy based on the above training framework:

[0119] like Figure 2 As shown, the four-stage progressive training strategy is as follows:

[0120] Phase 1: Simple Low-Traffic Scenario. Only one slow-moving vehicle is positioned in front of the agent, with minimal background traffic. No more than four vehicles exist in each lane simultaneously, corresponding to a traffic flow of approximately 200 vehicles per hour. The primary objective is for the agent to perform basic following and safe lane changing while maintaining a safe distance. To reduce scenario randomness, the agent's initial speed is set to 18 m / s, the speed of the vehicle in front is set to 14 m / s, and the initial distance between the two vehicles is uniformly randomized within a range of 30–50 meters.

[0121] Phase Two: Fixed Overtaking Task Scenario. While maintaining low traffic density, the number of vehicles in each lane simultaneously should not exceed 6, and the total traffic flow should not exceed 300 vehicles / hour. A significantly slower vehicle is positioned ahead, traveling at 12-14 meters per second. Simultaneously, the initial longitudinal distance between the nearest vehicle in the target lane and the agent should not be less than 60 meters. Through these constraints, the agent is guided to learn to proactively initiate lane changes and overtake when the speed of the vehicle ahead is significantly lower than the desired speed, thereby improving the task completion rate.

[0122] Phase 3: Multi-condition Random Scenarios. The number of background vehicles is increased, with 6-10 vehicles simultaneously in each lane, and a total traffic flow of approximately 400-600 vehicles / hour. The initial positions and speeds of the driver and the preceding vehicle are randomized: the driver's initial speed is uniformly randomized within the range of 16-22 m / s, the preceding vehicle's speed is uniformly randomized within the range of 10-18 m / s, and the initial inter-vehicle distance is randomly selected within the range of 20-80 meters. By introducing different overtaking distances, speed differences, and queue-jumping scenarios, the agent experiences various different conditions, enhancing the strategy's adaptability to changing conditions.

[0123] Phase Four: High-Density Complex Scenarios. Traffic density is further increased, with no fewer than 10 vehicles simultaneously in each lane and a total traffic flow of no less than 700 vehicles per hour. Multiple interfering vehicles are introduced, comprising approximately 30% or more of the background vehicles. These interfering vehicles can randomly decelerate, accelerate, or cut in front of the trainer during training, creating a complex interactive environment. In this phase, the agent needs to complete lane-changing and overtaking tasks under high traffic and strong interference conditions, training it to maintain high safety and success rates even under extreme conditions. The training control and data recording module automatically switches from lower to higher phases based on preset phase switching conditions (e.g., successful overtaking rate, collision rate, and average overtaking time reaching given thresholds), and records statistical indicators such as collision rate, success rate, and overtaking time for each phase for subsequent parameter tuning and performance evaluation.

[0124] In one specific embodiment, the road is a two-lane straight highway approximately one kilometer long, with the same speed limit in both lanes. The accelerator vehicle initially positions itself in the right lane near the starting point, with an initial speed slightly higher than the vehicle in front. The vehicle in front travels at a lower, fixed speed. Several background vehicles are randomly generated in the left lane, with their initial spacing and speed randomly selected within a given interval. The simulation time step is a fixed value, and the decision cycle is consistent with the time step.

[0125] After training for tens of thousands of steps according to the above four-stage training strategy, the best-performing strategy model in the validation scenario was selected from multiple candidate models for testing. In a typical overtaking test, the vehicle first follows the vehicle in front while maintaining a dynamic safe distance. When it detects that the speed of the vehicle in front in the current lane is significantly lower than its expected speed and that the target lane has sufficient safety clearance, the strategy network outputs a "lane change" action, and the vehicle smoothly changes lanes to the left lane to complete the overtaking maneuver. After overtaking, the vehicle continues to travel stably at a higher speed, and no collision occurs throughout the process. In multiple simulation tests, the strategy model successfully completed the overtaking task within a limited time, verifying the effectiveness of the technical solution of this invention in terms of safety and efficiency.

[0126] After training convergence, the policy export and application module exports the parameters of the high-performance policy network as a model file and provides an interface for integration into other simulation platforms or hardware-in-the-loop systems. In practical applications, only the environment perception and state construction components need to be deployed, and the real-time state vectors are input into the model to obtain online lane-changing decision results.

[0127] Example 2

[0128] This embodiment is a further explanation of Embodiment 1, providing alternative solutions.

[0129] Algorithm alternatives: While keeping the overall system structure and training process unchanged, the PPO algorithm can be replaced with other policy gradient-based algorithms, such as A3C, SAC, TRPO, etc. As long as stable updates are still achieved through policy pruning or trust domain constraints, they can be regarded as equivalent implementations of this embodiment.

[0130] State and action space expansion scheme: Dimensions such as vehicle acceleration, road curvature, and oncoming lane vehicle information can be added to the state vector, or the action space can be expanded to include various discrete actions such as deceleration, acceleration, and avoidance, or even continuous control quantities can be used as outputs. As long as the overall idea is still based on lane-changing decisions based on reinforcement learning, it is within the protection scope of this embodiment.

[0131] Reward function weights and form variations: Depending on different road scenarios (such as highways, urban expressways, etc.), the weights of each sub-item of the reward function can be adjusted, or a non-linear penalty form can be introduced to impose stronger constraints on serious safety risks.

[0132] Alternative simulation platforms: The micro-traffic simulation platform can be replaced with other simulation platforms such as CARLA and VISSIM. As long as it can provide vehicle motion status and support interaction with the reinforcement learning environment, it is still an equivalent solution in this embodiment.

[0133] Deployment and application scenario expansion: The trained lane-changing decision model can be deployed to the vehicle-road cooperative system and combined with the V2X communication module to achieve cooperative lane changing; it can also be migrated to scenarios such as intelligent truck platooning and ramp lane merging, and can be quickly adapted by adjusting the state and reward definitions.

[0134] Example 3

[0135] This embodiment provides an intelligent vehicle autonomous lane-changing decision-making method based on near-end strategy optimization. The flowchart of the method is as follows: Figure 3 As shown. The method is based on the system in Examples 1 and 2.

[0136] S1. Initialize the simulation environment and traffic flow parameters, set road geometry information, number of lanes, lane speed limit parameters, and arrival rate and speed distribution of background traffic flow, and start SUMO simulation;

[0137] S2. Set the initial position and speed of the vehicle and the vehicle in front, start a simulation round, and clear the data record of the previous round.

[0138] S3. At each simulation time step, the state observation and feature construction module collects the state of the vehicle and the vehicle in front from the simulation environment and generates a state vector of fixed dimension.

[0139] S4. Input the current state vector into the near-end policy optimization lane-changing decision module, and the policy network outputs the action decision of "keep lane" or "change lane".

[0140] S5. Perform this action in the traffic simulation environment module, and SUMO and its built-in driver model will automatically update the vehicle position and speed.

[0141] S6. The reward calculation and safety supervision module reads the updated vehicle status, calculates the total reward for the current time step, and determines whether the dynamic safety distance has been violated or a collision has occurred. If there is a serious unsafe situation, the current round can be terminated immediately.

[0142] S7, the training control and data recording module records the current step's state acquisition, action execution, reward record and value estimation data. When the number of experience samples reaches a preset threshold, it calls the proximal policy optimization algorithm to update the policy network and value network in multiple rounds.

[0143] S8. Repeat steps S2 to S7 until the number of training rounds or performance indicators reach the preset requirements. Finally, the strategy export and application module exports the final strategy model for subsequent deployment and application.

Claims

1. An intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization, characterized in that, The system includes: The system includes a traffic simulation environment module, a state observation and feature construction module, a near-end strategy optimization lane-changing decision module, a reward calculation and safety supervision module, a training control and data recording module, and a strategy export and application module. Traffic simulation environment module: Based on the urban traffic simulation software SUMO, the initial simulation environment and traffic flow parameters are constructed to provide controllable overtaking conditions; State observation and feature construction module: At each simulation time step, the module obtains the driving information of the vehicle and the vehicle in front from the traffic simulation environment module through the traffic control interface, and combines and normalizes it to form a state vector; The near-end strategy optimization lane-changing decision module includes a strategy network submodule and a value network submodule. After receiving the state vector, the strategy network submodule outputs the probabilities of two discrete actions: "keeping the lane" and "changing lanes". The value network submodule estimates the long-term reward of the current state. When training the policy network, the near-end policy optimization algorithm is used, and the pruning mechanism in the near-end policy optimization algorithm is improved, specifically as follows: First, based on each time step Corresponding collision and safety penalties The training samples are divided into normal samples and high-risk samples; a basic pruning factor is used for normal samples. For high-risk samples, a more stringent pruning factor is used. And satisfy Therefore, the probability of the shearing objective function of the near-end policy optimization algorithm is higher than that of the near-end policy optimization algorithm. The cropping range is determined based on the sample's security level. or Adaptive switching between them The definition process is as follows: Definition of the first Shop distance per prediction step: ; in, This represents the current longitudinal distance between the vehicle and the vehicle in front. Given the current relative speed, For the simulation time step, The prediction step index is denoted as ; the upper limit of the prediction steps is denoted as . If there exists a certain ( ) makes or the current actual distance It is already smaller than If so, it is determined that the vehicle has a potential or actual risk of collision. ;otherwise ; Indicates the longitudinal dynamic safety distance; Penalty coefficients for collisions and serious unsafe acts. , , For safety weighting coefficients, , , and These are the upper bounds of the absolute values ​​of the space efficiency reward, velocity deviation penalty, and lateral trajectory deviation penalty in single-step simulation, respectively. Reward Calculation and Safety Supervision Module: Evaluates the simulation results at each step, calculates the composite reward and constructs a dynamic safety distance that varies with speed, determines whether there is a risk of collision in the future, and applies penalties in advance or terminates the current round when there is a risk of collision. Compound rewards are represented as follows: ； in, For the first Total reward for each time step; Rewards for space efficiency; Penalty for speed deviation; Penalty for lateral trajectory deviation; For collision and safety penalties; These are the weighting coefficients for each part; Training control and data recording module: Organizes the entire reinforcement learning training process to form a complete closed-loop training process; Policy Export and Application Module: After training convergence, the policy network parameters are exported as model files and integrated into the simulation platform or hardware-in-the-loop system of the actual application.

2. The intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization according to claim 1, characterized in that, The traffic simulation environment module initializes the simulation environment and constructs traffic flow parameters by: building a two-lane straight road scenario, setting scenario parameters, including road length, number of lanes, lane width and lane speed limit, and setting the arrival rate, vehicle type and speed range of background traffic flow through configuration files.

3. The intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization according to claim 2, characterized in that, When the state observation and feature construction module constructs the state vector, it generates a six-dimensional state vector. This six-dimensional state vector is then normalized to form the state vectors. The six-dimensional state vectors are as follows: The longitudinal position of the vehicle is used to represent the vehicle's progress on the road; The longitudinal speed of the vehicle is used to represent its current speed. The longitudinal relative distance between the vehicle in front and the vehicle in front in the current lane is used to measure the following distance; The longitudinal relative speed between the vehicle in front and the vehicle in front in the current lane is used to determine the trend of changing following distance; The longitudinal relative distance between the vehicle in front in the target lane and the vehicle itself is used to assess the safety clearance in the target lane; The longitudinal relative speed between the vehicle in front and the vehicle in the target lane is used to determine the change in the safety clearance of the target lane over time.

4. The intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization according to claim 3, characterized in that, In the policy network submodule, the state vector is input into the policy network, and the policy network outputs the probability distribution of the two actions "keeping the lane" and "changing lanes". The policy network adopts a two- or three-layer fully connected neural network structure, and the hidden layers use non-linear activation functions.

5. The intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization according to claim 4, characterized in that, The training control and data recording module employs a four-stage progressive training strategy, specifically: Phase 1: Simple scenario with low traffic flow. Only a small number of low-speed vehicles are placed in front of the vehicle, and there is little background traffic. The main goal is to teach the intelligent agent to perform basic following and safe lane changing while maintaining a safe distance. Phase Two: Fixed Overtaking Task Scenario. While maintaining low traffic density, a significantly slow-moving vehicle in front and a relatively empty target lane are set up to guide the intelligent agent to learn to actively initiate lane changes and complete overtaking, thereby improving task completion efficiency. Phase 3: Multi-condition randomized scenarios, increasing the number of background vehicles and randomizing the initial positions and speeds of the vehicle and the vehicle in front, so that the agent experiences various overtaking distances, speed differences and queue-jumping situations, enhancing the strategy's adaptability to changes in operating conditions; Phase 4: High-density complex scenarios. Increase traffic density and add multiple interfering vehicles to further train the agent in a complex interactive environment, enabling it to maintain high safety and success rate under high load and strong interference conditions.

6. The intelligent vehicle autonomous lane-changing decision-making system based on near-end strategy optimization according to claim 5, characterized in that, The training control and data recording module organizes the entire reinforcement learning training process, including: environment initialization, simulation time step progression, state acquisition, action execution, reward recording, advantage function calculation, and triggering the update of the proximal policy optimization algorithm for each round; and records the parameters of each round of training process.

7. An intelligent vehicle autonomous lane-changing decision-making method based on near-end strategy optimization, characterized in that, The method is performed using the system as described in any one of claims 1-6, and specifically comprises: S1. Initialize the simulation environment and traffic flow parameters, set road geometry information, number of lanes, lane speed limit parameters, and arrival rate and speed distribution of background traffic flow, and start SUMO simulation; S2. Set the initial position and speed of the vehicle and the vehicle in front, start a simulation round, and clear the data record of the previous round. S3. At each simulation time step, the state observation and feature construction module collects the state of the vehicle and the vehicle in front from the simulation environment and generates a state vector of fixed dimension. S4. Input the current state vector into the near-end policy optimization lane-changing decision module, and the policy network outputs the action decision of "keep lane" or "change lane"; S5. Perform this action in the traffic simulation environment module, and SUMO and its built-in driver model will automatically update the vehicle position and speed. S6. The reward calculation and safety supervision module reads the updated vehicle status, calculates the total reward for the current time step, and determines whether the dynamic safety distance has been violated or a collision has occurred. If the dynamic safety distance has been violated or a collision has occurred, the current round is terminated immediately. S7, the training control and data recording module records the current step's state acquisition, action execution, reward record and value estimation data. When the number of experience samples reaches a preset threshold, it calls the proximal policy optimization algorithm to update the policy network and value network in multiple rounds. S8. Repeat steps S2 to S7 until the number of training rounds or performance indicators reach the preset requirements. Finally, the strategy export and application module exports the final strategy model for subsequent deployment and application.

Images