Intelligent highway lane changing method for autonomous vehicle based on reinforcement learning

Abstract

The invention discloses an automatic driving vehicle highway intelligent lane changing method based on reinforcement learning, which optimizes a highway automatic driving decision strategy by designing an improved Huber regularization reward threshold adaptive double depth Q network (HRA-DDQN) framework. The method comprises the following steps: firstly, constructing a state space and an action space to describe dynamic behaviors of a vehicle, and introducing a comprehensive reward function based on safety, efficiency and lane changing requirements; secondly, through a dynamic target network updating mechanism triggered by reward differences, the adaptability of the model to traffic environment changes is improved; and finally, introducing a hybrid loss function combining Huber loss and L2 regularization to enhance the robustness and generalization ability of the model. Simulation results show that the driving efficiency and safety of the automatic driving vehicle in the expressway scene are remarkably improved, the collision rate is reduced, the stability of lane changing decision making is optimized, and the method is suitable for the complex and changeable expressway traffic environment.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for intelligent lane changing on a highway for an autonomous driving vehicle based on reinforcement learning. Background Art

[0002] With the rapid development of autonomous driving technology, it has shown great potential in improving traffic safety and optimizing road efficiency. However, in complex and dynamic highway scenarios, especially when it comes to vehicle lane change decisions, there are still many challenges. Lane change operations require accurate environmental perception, prediction of surrounding vehicle behavior, and smooth action execution. In a complex and unpredictable environment, autonomous vehicles must be able to understand the goals of surrounding vehicles and make coordinated driving decisions based on them. The proficiency of these decision-making capabilities has a profound impact on the overall performance and safety of autonomous vehicles. Current autonomous driving technology is still lacking in the face of complex and changeable highway traffic environments. The driving efficiency is not high, and there are certain safety issues. The collision rate is high and it cannot meet the stability requirements of lane change decisions. Summary of the invention

[0003] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide an intelligent lane changing method for autonomous driving vehicles on highways based on reinforcement learning, so as to ensure that the autonomous driving vehicles can travel efficiently and safely in a specific highway environment.

[0004] Technical solution: A method for intelligent lane changing on a highway for an autonomous driving vehicle based on reinforcement learning, comprising the following steps:

[0005] Step 1: Set the prerequisites of road environment and vehicle settings, and construct the state space and action space of the autonomous driving vehicle in the highway scenario. The state space includes the vehicle's lateral and longitudinal position, speed and surrounding environment information; the action space includes acceleration, deceleration, changing lanes to the left, changing lanes to the right and staying in the current lane;

[0006] Step 2: Design reward functions, including speed reward function, collision reward function, lane keeping reward function, lane change reward function and action stability reward function;

[0007] Step 3: The Huber regularized reward threshold adaptive dual deep Q network model, namely HRA-DDQN model, is used to train and optimize the lane change decision strategy. HRA-DDQN includes a target network and an online network. The dynamic reward difference threshold is used to trigger the update of the target network parameters to optimize the model's adaptability to dynamic traffic environments.

[0008] Step 4: Verify the optimized HRA-DDQN model in the set simulation environment and evaluate the model's security, efficiency, and stability indicators.

[0009] Furthermore, in step one, the road environment selects a road scene as a one-way multi-lane highway, and all vehicles on the highway are intelligent connected autonomous driving vehicles. The dynamic control of all vehicles is subject to the constraints of the vehicle dynamics model, and information can be exchanged between vehicles, including real-time speed, position and collision information; the autonomous driving vehicle has a perception and information processing system, so that the host vehicle can grasp the surrounding environment information in real time, including the speed and position information of obstacle vehicles in other lanes, and all obstacle vehicles use an intelligent driving model.

[0010] Furthermore, in step 1, the state space is expressed as:

[0011]

[0012] Where x represents the lateral position of the vehicle; y represents the longitudinal position of the vehicle; v x Indicates the lateral velocity of the vehicle; v y Indicates the longitudinal speed of the vehicle; s 0 Indicates the status of the host vehicle; i (i∈{1,2,...,N}) represents the state of surrounding vehicles; N represents the number of surrounding vehicles;

[0013] The action space is represented as:

[0014] A={"accelerate","decelerate","lane change right","lane change left","lane keep"}.

[0015] Furthermore, in step 2, the reward function R is the speed reward function R v , collision reward function R c , Lane keeping reward function R l , Lane change reward function R d and the action stability reward function R e The weighted sum of the five reward functions is:

[0016] R=c v R v +c c R c +c l R l +c d R d +c e R e ;

[0017] Among them, c v 、c c、c l 、c d and c e They are speed reward functions R v , collision reward function R c , Lane keeping reward function R l , Lane change reward function R d and the action stability reward function R e The weight of .

[0018] The optimal,speed indicator reward function is:

[0019]

[0020] Where V represents the current speed of the host vehicle, v min Indicates the minimum speed, v max Indicates the maximum speed, v target represents the target speed, v tolerance =max{v target -v min ,v max -v target} is the speed tolerance;

[0021] The collision reward function is:

[0022]

[0023] The lane keeping reward function is:

[0024]

[0025] Among them, c i represents the center position of the lane, kd is a predefined constant;

[0026] The lane change reward function is:

[0027]

[0028] Among them, a represents the current action, RC represents reasonable lane change, UC represents unreasonable lane change, and NC represents no lane change;

[0029] The action stability reward function is:

[0030]

[0031] Where λ is the decay coefficient, t represents the time step, and a t Indicates the current action, a t-1 Indicates the previous action.

[0032] Furthermore, in step three, the HRA-DDQN model is divided into two networks, namely the online Q network and the target Q network, through action selection and Q value estimation; the task of the online Q network is to select the best action by selecting the action with the highest predicted Q value; the target Q network estimates the Q value of the selected action.

[0033] The optimal, online Q network is set to: Q(s t ,a t ;θ), the target Q network is set to: Q(s t+1 ,a t+1 θ - );

[0034] The experience replay memory stores the state transition tuple as: (s t ,a t ,r t ,s t+1 ), used to update the network; action a max The online Q network is based on the next state s t+1 The action selection that produces the maximum Q value is expressed as:

[0035] a max = argmax a Q(s t+1 ,a;θ);

[0036] The target Q network is then used to estimate the Q value of the selected action, expressed as:

[0037] y t =r t +γQ′(s t+1 ,a max θ - );

[0038] Among them, r t represents the immediate reward, γ is the discount factor, Q′(s t+1 ,a max θ - ) is the next state s using the target Q network t+1 The estimated Q value of the selected action; the target value y t It is used to update the online Q network by minimizing the loss function.

[0039] The optimal loss function includes a combination of the Huber loss function and the L2 regularization term, expressed as:

[0040]

[0041] Where θ represents the parameters of the online Q network, N is the batch size of the experience replay buffer, Huber(x) represents the Huber loss function, and the parameter λ is the regularization coefficient used to adjust the weight of the regularization term.

[0042] The optimal, target Q network parameter update condition is determined based on the difference between the previous reward and the current reward, expressed as:

[0043] if|R t -R t-1 |>K, then updateθ - to θ;

[0044] Among them, R t is the current reward value at time step t, R t-1 is the reward value at time step t-1, θ and θ - denote the parameters of the target network and the online network respectively, and K is the predefined reward difference threshold.

[0045] Beneficial effects: Compared with the prior art, the advantages of the present invention are: introducing a comprehensive reward function based on safety, efficiency and lane changing requirements, optimizing the highway automatic driving decision-making strategy, improving the adaptability of the model to changes in the traffic environment through a dynamic target network update mechanism triggered by reward differences, introducing a hybrid loss function combining Huber loss and L2 regularization to enhance the robustness and generalization ability of the model, and the established HRA-DDQN model performs well in all key indicators, especially in safety, achieving target speed and optimizing lane changing strategies. It is very effective in complex traffic environments, significantly improving the driving efficiency and safety of autonomous driving vehicles in highway scenarios, reducing collision rates, and improving the stability of lane changing decisions. It is suitable for complex and changeable highway traffic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a framework diagram of the HRA-DDQN behavior decision model of the present invention;

[0047] Figure 2 A schematic diagram of a four-lane highway environment for training and testing the present invention;

[0048] Figure 3 This is a comparison chart of the experimental results of the HRA-DDQN behavior decision model and other models' behavior decision. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0050] A reinforcement learning-based intelligent lane-changing method for autonomous vehicles on highways. Figures 1 to 3 As shown, the following steps are included:

[0051] Step 1: Assume the road environment and vehicle settings, including:

[0052] (1) The road scene is a one-way multi-lane highway (excluding ramp entrances and ramp exits).

[0053] (2) Autonomous driving vehicles already have a complete perception and information processing system, and the host vehicle can grasp the surrounding environment information in real time, including the speed, position and other information of vehicles with obstacles in other lanes.

[0054] (3) All vehicles on the highway are intelligent connected autonomous vehicles, and vehicles can exchange information with each other, including real-time speed, location and collision information.

[0055] (4) All obstacle vehicles use the Intelligent Driver Model (IDM);

[0056] (5) The dynamic control of all vehicles is subject to the constraints of the vehicle dynamics model.

[0057] Step 2: Design the state space, action space, and reward function of the host vehicle;

[0058] Step 1: Determine the state space;

[0059] In the highway driving environment, the present invention selects the most representative speed and position of the vehicle as state variables. The state space can be expressed as follows:

[0060] x: represents the lateral position of the vehicle. y: represents the longitudinal position of the vehicle. x : Indicates the lateral velocity of the vehicle. v y : Indicates the longitudinal speed of the vehicle. s 0 Represents the state of the host vehicle, s i (i∈{1,2,...,N}) represents the state of surrounding vehicles, and N represents the number of surrounding vehicles.

[0061] Step 2: Determine the action space;

[0062] In the present invention, the action space of the host vehicle is defined as a set of five discrete actions: acceleration, deceleration, lane change to the left, lane change to the right, and lane keeping. Therefore, the action space can be expressed as A = {"accelerate",

[0063] "decelerate", "lane change right", "lane change left", "lane keep"}.

[0064] Step 3: Design reward function;

[0065] The reward function is a core component of the reinforcement learning algorithm, which directly determines the direction of model learning and the final decision quality. In the autonomous driving scenario, the reward function determines how the vehicle balances speed, safety, and efficiency. This invention combines the five aspects of speed, collision, lane keeping, lane changing, and action stability to design the reward function:

[0066] (1) Speed ​​Reward:

[0067] A speed reward is given based on the current speed of the host vehicle. The specific speed reward function is as follows:

[0068]

[0069] Where V represents the current speed of the host vehicle, and v min Minimum speed, v max Indicates the maximum speed, v target represents the target speed, v tolerance =max{v target -v min ,v max -v target} is the speed tolerance. In the present invention, the minimum speed, maximum speed and target speed are set to 20m / s, 40m / s and 30m / s respectively. The current speed V exceeds the maximum speed v max or below the minimum speed v min , a negative reward of -2 will be given to penalize the deviation from the target speed range. Otherwise, the reward function will be based on V and v target The reward is calculated based on the deviation of , thus motivating the vehicle to stay within the specified speed range. This design meets the goal of stable driving. In addition, by introducing the speed tolerance range v tolerance ,The speed reward function is not only based on the current speed but also considers the allowed speed variation range.

[0070] (2) Collision rewards;

[0071] To ensure driving safety, the host vehicle must avoid collisions. Therefore, the collision penalty is designed to be significantly higher than other reward components. The specific collision reward function is as follows:

[0072]

[0073] (3) Lane keeping reward;

[0074] To achieve lane keeping, the ego vehicle needs to drive near the center of the lane. This strategy encourages the vehicle to stay within a safe range, minimize excessive deviation from the lane, and reduce the risk of accidents. The specific lane keeping reward function is as follows:

[0075]

[0076] Among them, c i Indicates the center position of the lane. When the vehicle position x is equal to the lane center c i When the distance is less than or equal to kd (where k and d are predefined constants), the reward value is 1, indicating that the vehicle is within the acceptable deviation range. On the contrary, if the deviation exceeds the threshold kd, the reward value is 0, indicating that the vehicle has deviated from the center of the lane.

[0077] (4) Lane change rewards;

[0078] In order to effectively guide vehicles to choose the optimal lane in a complex traffic environment, thereby improving driving efficiency and safety, we encourage reasonable lane changes and punish unnecessary lane changes. The specific lane change reward function is as follows:

[0079]

[0080] RC stands for reasonable lane change, UC stands for unreasonable lane change, and NC stands for no lane change. To make a lane change reasonable, the following criteria must be met:

[0081] 1. Speed ​​difference: The current vehicle speed V needs to be lower than the target vehicle speed v target .

[0082] 2. Adjacent lane speed: The average speed of vehicles in the adjacent lane should be at least 10 units higher than the current speed (v lane >v current +10).

[0083] When the above two conditions are met, executing a lane change (to the left or right) is considered reasonable and receives a reward of 1. Conversely, if a lane change is executed without meeting these conditions, it is considered an unreasonable lane change (UC) with a penalty of -0.5. If no lane change occurs (NC), the reward is 0.

[0084] (5) Action stability reward;

[0085] In order to achieve efficient lane change decisions for self-driving cars on highways, this paper designs an action stability reward function. This reward function calculates rewards based on the changes in actions and encourages the ego vehicle to remain stable. The specific action stability reward function is as follows:

[0086]

[0087] Where λ is the attenuation coefficient and t is the time step. t ≠a t-1 ), a negative reward is imposed that decays exponentially over time. This suggests that over time, the system becomes more stable and the cost associated with lane changes decreases. t =a t-1 ), the reward value is zero. This design is intended to encourage the vehicle to maintain stability, resulting in more efficient and stable driving on highways.

[0088] In summary, the total reward is designed as the weighted sum of speed reward, collision reward, lane keeping reward, lane change reward, and action stability reward, as shown below:

[0089] R=c v R v +c c R c +c l R l +c d R d +c e R e ;

[0090] Among them, c v 、c c 、c l 、c d , and c e It is a positive weight coefficient and needs to be adjusted according to different driving conditions.

[0091] Step 3: Construct HRA-DDQN behavior decision model;

[0092] In this invention, DDQN is introduced. DDQN improves the traditional DQN by separating action selection and Q-value estimation into two different networks: online Q network and target Q network. Figure 1 The input and output of the HRA-DDQN behavior decision model framework are given. The task of the online Q network is to select the best action by choosing the action with the highest predicted Q value, while the target Q network estimates the Q value of the selected action. This distinction helps to reduce the overestimation bias, thereby achieving more stable learning.

[0093] First, the online Q network Q(s t ,a t ;θ) and the target Q network Q(s t+1 ,a t+1 θ -) is created, and the experience replay (Replay) stores the transformation tuple (s t ,a t ,r t ,s t+1 ), these tuples are used to update the network. Action a max Selected by the online Q network, based on the next state s t+1 The action with the maximum Q value, the specific formula is as follows:

[0094] a max = argmax a Q(s t+1 ,a;θ);

[0095] The target Q network is then used to evaluate the Q value of the selected operation, as shown below:

[0096] y t =r t +γQ′(s t+1 ,a max θ - );

[0097] Among them, r t represents the immediate reward, γ is the discount factor, Q′(s t+1 ,a max θ - ) is the next state s calculated using the target Q network t+1 The estimated Q value of t It is then used to update the online Q-network by minimizing the loss function.

[0098] In the invention, the loss function is designed as a combination of the Huber loss function and the L2 regularization term. Compared with the MSE loss, the Huber loss is less sensitive to outliers and helps stabilize the training process. The enhanced loss function is as follows:

[0099]

[0100] Where θ represents the parameters of the online Q network, N is the batch size of the experience replay buffer, Huber(x) represents the Huber loss function, and the parameter λ is the regularization coefficient, which is used to adjust the weight of the regularization term.

[0101] The Huber loss function combines the advantages of mean square error (MSE) and mean absolute error (MAE). MSE is used for smaller errors and MAE is used for larger errors, thus showing higher robustness to outliers. The Huber loss function is shown below:

[0102]

[0103] The parameter δ is a threshold that determines the point at which the loss function transitions from the quadratic form of MSE to the linear form of MAE. When the absolute error |x| is less than δ, the Huber loss behaves similarly to the MSE, being more sensitive to small errors and providing a smooth gradient, thereby facilitating small adjustments in the optimization process. In contrast, when |x| exceeds the threshold δ, the Huber loss transitions to a linear form, similar to the MAE. This linear behavior mitigates the impact of large errors and enhances robustness. By properly adjusting the threshold parameter δ, a balance can be achieved between sensitivity to small errors and robustness to outliers.

[0104] To reduce model overfitting, the L2 regularization term is added to the loss function of HRA-DQN. This term penalizes the sum of squares of model parameters and encourages parameters to take smaller values, thereby improving the generalization ability of the model. The specific definition of L2 regularization is as follows:

[0105]

[0106] where λ is the regularization coefficient, θ represents the parameters of the online Q network, and ||θ j || 2 Denotes each individual parameter θ j The square of .

[0107] The online Q network parameters θ are updated using the gradient descent algorithm, which is as follows:

[0108]

[0109] Where α is the learning rate, Represents the gradient of the loss function with respect to the network parameters. By updating these parameters, the network performance can be optimized.

[0110] In traditional DQN methods, the target network parameters are updated at fixed time intervals. However, in the present invention, the target network parameters are updated according to specific conditions related to the reward difference. When the reward changes significantly, the method can update the parameters of the target network, thereby enhancing the adaptability of the model and improving the training effect.

[0111] The update condition of the target network parameters is based on the difference between the current reward and the previous reward. The specific update conditions are as follows:

[0112] if|R t -R t-1 |>K, then updateθ - to θ;

[0113] Where R t is the current reward at time step t, R t-1is the previous reward at time step t-1, θ - and θ denote the parameters of the target network and the online network respectively, and K is a predefined reward difference threshold.

[0114] When the current reward R t and the previous reward R t-1 When the difference between them exceeds a predefined threshold K, the parameters of the target network are updated to match those of the online network. This update mechanism ensures that the target network can quickly adapt to significant reward changes.

[0115] Step 4: Verify the optimized HRA-DDQN model in a simulation environment. The simulation scenario includes a four-lane highway and evaluates the safety, efficiency, and stability indicators of the model.

[0116] In order to further verify the performance of the HRA-DDQN behavior decision model established by the present invention, the present invention selects the highway-env simulator based on OpenAI Gym to evaluate the performance of HRA-DDQN. highway-env provides six different driving scenarios: highway, intersection, roundabout, merge, racetrack, and parking. Figure 2 As shown in the figure, a four-lane highway environment for training and testing is constructed, including the following steps:

[0117] Step 1: Model parameter setting;

[0118] In order to comprehensively evaluate the performance of the HRA-DDQN behavioral decision model, we loaded and tested the model and compared it with the basic models Dueling DQN and ETDQN. During the training process, we trained the HRA-DDQN, ETDQN, and Dueling DDQN models under the same settings for a total of 10 5 The neural network architecture of the HRA-DDQN, ETDQN, and DuelingDDQN models is a fully connected deep neural network (DNN) with three hidden layers and 1024 units in each layer.

[0119] At each time step t, the interaction data between the agent and the environment is stored in an experience replay buffer with a capacity of N. The network samples training data from the buffer with a batch size of B. When calculating the Q value, a discount factor γ (usually between 0 and 1) is applied to consider potential future rewards. The corresponding model parameter settings are listed in Table 1.

[0120]

[0121] Table 1

[0122] Step 2: Evaluate the metrics;

[0123] In order to evaluate and compare the performance of the models, we selected a set of metrics. The performance metrics used for evaluation include: 1) average reward, 2) average number of steps, 3) average speed, and 4) average action change frequency.

[0124] The average reward is defined as the sum of the cumulative rewards in each round divided by the number of rounds. This metric evaluates the overall performance of the algorithm in the complete task, with higher values ​​indicating better results.

[0125] Average speed refers to the average speed of the vehicle during the entire round, obtained by calculating the average speed of each round. The target speed is set to 30m / s, and values ​​close to this target reflect efficient driving behavior.

[0126] The average number of steps refers to the total number of steps taken by the vehicle to complete an event, that is, the average of all events. It is specifically expressed as follows:

[0127]

[0128] Where N k Represents the number of driving steps in round K. Fewer steps indicate a more effective driving strategy.

[0129] The average action change frequency refers to the average frequency at which the control algorithm initiates an action. It is calculated by dividing the total number of actions by the total number of rounds, as follows:

[0130]

[0131] Where T k Indicates the frequency of action changes in round K. Lower values ​​indicate more stable and cautious decisions.

[0132] By comparing these metrics, this allows us to comprehensively evaluate the strengths and weaknesses of various algorithms (i.e., HRA-DDQN, ETDQN, and Dueling DQN) in the context of the autonomous driving decision-making problem.

[0133] Step 3: Analysis of experimental results;

[0134] 1) Average Reward

[0135] like Figure 3(a) shows the trend of the average reward during training. We can see that HRA-DDQN outperforms the other two methods in terms of learning speed and training stability. HRA-DDQN shows rapid improvement after about 10,000 training steps and consistently performs well, especially after 20,000 time steps, when the average reward stabilizes above 15. In contrast, the average rewards of ETDQN and Dueling DQN are relatively low and fluctuate greatly within the same training range. Notably, the average reward of Dueling DQN is often below zero.

[0136] 2) Average number of steps

[0137] Figure 3 (b) illustrates the variation of the average number of steps throughout the training process. HRA-DDQN shows a significant advantage, with the average number of steps being stable between 20 and 40. In contrast, ETDQN and Dueling DQN show large fluctuations, especially ETDQN, which often reaches 100. The average number of steps of HRA-DDQN remains most stable between 20,000 and 80,000 training steps, reflecting high efficiency and stability.

[0138] 3) Average speed

[0139] Figure 3 (c) shows the average speed variation of the three methods. Significant fluctuations are observed in the initial stages of training. However, HRA-DDQN consistently outperforms the others throughout the training process. After approximately 10,000 time steps, the average speed of HRA-DDQN converges to the target speed of 30m / s, remaining relatively stable compared to the other methods. On the other hand, both ETDQN and Dueling DQN show greater variability, with ETDQN frequently falling below 25m / s. HRA-DDQN highlights its advantage in speed control, improving rapidly and stabilizing after 10,000 training steps.

[0140] 4) Average action change frequency

[0141] Figure 3 (d) depicts the change in average action change frequency during training. As training progresses, the action change frequency tends to stabilize. However, HRA-DDQN maintains a lower and more stable action change frequency, while both ETDQN and DuelingDQN show greater fluctuations. HRA-DDQN quickly reduces its action change frequency in the early stages and maintains stability, showing significant advantages in reducing unnecessary action changes and improving efficiency.

[0142] In summary, HRA-DDQN demonstrates superior performance in all key metrics, especially in terms of safety, reaching target speed, and optimizing lane-changing strategies, making it very effective in complex traffic environments.

Claims

1. A method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning, characterized in that The following steps are involved: Step 1: Set the prerequisites of road environment and vehicle settings, and construct the state space and action space of the autonomous driving vehicle in the highway scenario. The state space includes the vehicle's lateral and longitudinal position, speed and surrounding environment information; the action space includes acceleration, deceleration, changing lanes to the left, changing lanes to the right and staying in the current lane; Step 2: Design reward functions, including speed reward function, collision reward function, lane keeping reward function, lane change reward function and action stability reward function; Step 3: The Huber regularized reward threshold adaptive dual deep Q network model, namely HRA-DDQN model, is used to train and optimize the lane change decision strategy. HRA-DDQN includes a target network and an online network. The dynamic reward difference threshold is used to trigger the update of the target network parameters to optimize the model's adaptability to dynamic traffic environments. Step 4: Verify the optimized HRA-DDQN model in the set simulation environment and evaluate the model's security, efficiency, and stability indicators.

2. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 1, characterized in that: In step one, the road environment selects a one-way multi-lane highway as the road scene. All vehicles on the highway are intelligent connected autonomous driving vehicles. The dynamic control of all vehicles is subject to the constraints of the vehicle dynamics model. Vehicles can exchange information, including real-time speed, position and collision information. Autonomous driving vehicles have perception and information processing systems, so that the host vehicle can grasp the surrounding environment information in real time, including the speed and position information of obstacle vehicles in other lanes. All obstacle vehicles use intelligent driving models.

3. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 1, characterized in that: In step 1, the state space is represented as: Where x represents the lateral position of the vehicle; y represents the longitudinal position of the vehicle; v x Indicates the lateral velocity of the vehicle; v y represents the longitudinal speed of the vehicle; s0 represents the state of the host vehicle; s i (i∈{1,2,...,N}) represents the state of surrounding vehicles; N represents the number of surrounding vehicles; The action space is represented as: A={"accelerate","decelerate","lane change right","lane change left","lane keep"}.

4. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 1, characterized in that: In step 2, the reward function R is the speed reward function R v , collision reward function R c , Lane keeping reward function R l , Lane change reward function R d and the action stability reward function R e The weighted sum of the five reward functions is: R=c v R v +c c R c +c l R l +c d R d +c e R e ; Among them, c v 、c c 、c l 、c d and c e They are speed reward functions R v , collision reward function R c , Lane keeping reward function R l , Lane change reward function R d and the action stability reward function R e The weight of .

5. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 4, characterized in that: The speed indicator reward function is: Where V represents the current speed of the host vehicle, v min Indicates the minimum speed, v max Indicates the maximum speed, v target represents the target speed, v tolerance =max{v target -v min ,v max -v target } is the speed tolerance; The collision reward function is: The lane keeping reward function is: Among them, c i represents the center position of the lane, kd is a predefined constant; The lane change reward function is: Among them, a represents the current action, RC represents reasonable lane change, UC represents unreasonable lane change, and NC represents no lane change; The action stability reward function is: Where λ is the decay coefficient, t represents the time step, and a t Indicates the current action, a t-1 Indicates the previous action.

6. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 1, characterized in that: In step three, the HRA-DDQN model is divided into two networks, namely the online Q network and the target Q network, through action selection and Q value estimation; the task of the online Q network is to select the best action by selecting the action with the highest predicted Q value; the target Q network estimates the Q value of the selected action.

7. The method for intelligent lane changing on highways for an autonomous driving vehicle based on reinforcement learning according to claim 6, characterized in that: The online Q network is set as: Q(s t ,a t ;θ), the target Q network is set to: Q(s t+1 ,a t+1 θ - ); The experience replay memory stores the state transition tuple as: (s t ,a t ,r t ,s t+1 ), used to update the network; action a max The online Q network is based on the next state s t+1 The action selection that produces the maximum Q value is expressed as: a max =argmax a Q(s t+1 ,a;i); The target Q network is then used to estimate the Q value of the selected action, expressed as: y t =r t +γQ′(s t+1 ,amax;θ - ); Among them, r t represents the immediate reward, γ is the discount factor, Q′(s t+1 ,a max θ - ) is the next state s using the target Q network t+1 The estimated Q value of the selected action; the target value y t It is used to update the online Q network by minimizing the loss function.

8. The method for intelligent lane changing on highways for an autonomous driving vehicle based on reinforcement learning according to claim 7, characterized in that: The loss function includes a combination of the Huber loss function and the L2 regularization term, expressed as: Where θ represents the parameters of the online Q network, N is the batch size of the experience replay buffer, Huber(x) represents the Huber loss function, and the parameter λ is the regularization coefficient used to adjust the weight of the regularization term.

9. The method for intelligent lane changing on highways for autonomous driving vehicles based on reinforcement learning according to claim 7 or 8, characterized in that: The update condition of the target Q network parameters is determined by the difference between the previous reward and the current reward, expressed as: if|R t -R t-1 |>K,then updateθ - toθ; Among them, R t is the current reward value at time step t, R t-1 is the reward value at time step t-1, θ and θ - denote the parameters of the target network and the online network respectively, and K is the predefined reward difference threshold.

Images