A vehicle intelligent driving method based on reinforcement learning

Abstract

The application discloses a kind of vehicle intelligent driving method based on reinforcement learning, first establish state space and action space, construct initial scene and interactive training model, then establish double-delay deep deterministic policy gradient algorithm and corresponding hyperparameter;Based on intelligent driver model, double-delay deep deterministic policy gradient algorithm is improved;Finally, the initial state information obtained is input into the neural network of improved algorithm for deep reinforcement learning training, and the optimal vehicle driving strategy is determined.The application strengthens double-delay deep deterministic policy gradient algorithm by expert experience model;Make reinforcement learning algorithm can be quickly trained to a better basic strategy, avoid the useless random exploration of traditional reinforcement learning in early stage.Then again on this basis, network learning is updated again, improve vehicle driving strategy quality, and the generalization ability and response ability of the model are improved by reinforcement learning.

Description

Technical Field

[0001] This invention belongs to the field of vehicle intelligent driving decision-making technology, and in particular relates to a vehicle intelligent driving method based on reinforcement learning. Background Technology

[0002] In recent years, autonomous driving technology has developed rapidly, with various algorithms emerging and being applied in an endless stream, accelerating the realization of autonomous driving. Among these technologies, decision-making technology, as the brain of an intelligent vehicle, needs to issue correct behavioral planning instructions to the vehicle when facing various dangerous scenarios, playing a crucial role in the entire autonomous driving system. Following other vehicles, sudden stops by vehicles in front, and vehicles cutting in from the left or right are common scenarios in road driving. How the driver can improve driving efficiency at the optimal speed when facing these scenarios, and how to make reasonable decisions when faced with sudden stops or cut-ins by vehicles in front, are all crucial research topics.

[0003] Existing decision-making algorithms can be broadly categorized into two types: finite state machines based on expert rules and learning-based algorithms. Expert rule-based algorithms typically establish a rule base using existing expert experience and then select an appropriate strategy based on the vehicle's current state. However, expert rule-based decisions are complex and lack human-like qualities and comfort. For example, rule-based algorithms often employ emergency braking when encountering obstacles, reducing passenger comfort. Furthermore, if the vehicle faces unforeseen scenarios, effective expert rule-based state machine solutions will fail. Therefore, using a limited set of expert rules to handle numerous complex scenarios is clearly unreasonable, resulting in many long-tail problems that are difficult to solve. Learning-based algorithms mainly include imitation learning and reinforcement learning. Imitation learning fits an appropriate strategy by mimicking high-quality expert datasets. However, high-quality expert datasets are difficult to obtain, and imitation learning has poor generalization ability; the quality of the obtained strategy will not exceed that of the original expert dataset, exhibiting limitations similar to rule-based algorithms. Reinforcement learning, through its interaction with the environment and exploratory trial-and-error self-learning principles, possesses the theoretical ability to achieve perfect vehicle driving based on established expert rules. This has led to widespread attention on reinforcement learning in recent years, as it effectively enhances the decision-making capabilities of intelligent agents in complex environments, holding significant research value in the field of autonomous driving decision-making.

[0004] Traditional reinforcement learning algorithms, such as Q-learning, often use Q-tables for value function updates and iterations. However, this approach is prone to the curse of dimensionality when dealing with large-dimensional state spaces. Therefore, deep neural networks are introduced into Q-learning algorithms to form the DQN algorithm, thus addressing the curse of dimensionality. However, the DQN algorithm is only applicable to discrete action spaces. A limited number of discrete actions makes it difficult to formulate a truly optimal policy in complex scenarios. The selection of discrete, finite, and fixed throttle, brake, and steering values ​​reduces vehicle safety, traffic efficiency, and comfort. The DDPG algorithm, with its actor-commentator structure, addresses this problem, but as an earlier algorithm, it suffers from significant Q-value overestimation and policy instability. These problems were ultimately solved in the TD3 algorithm. However, all the reinforcement learning algorithms mentioned above share a significant issue: the lack of expert-like guidance leads to a large amount of aimless exploration in the early stages of interaction with complex environments. Overexploration significantly increases training convergence time and makes it more difficult to converge to the optimal policy. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a vehicle intelligent driving method based on reinforcement learning, so as to improve the comfort, traffic efficiency and safety during the autonomous driving process of vehicles, and further enhance the convergence speed and the ability to obtain the optimal strategy of the intelligent driving algorithm when dealing with complex environments.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a vehicle intelligent driving method based on reinforcement learning, comprising the following steps: S1. Build a vehicle driving simulation scenario; including establishing state space vectors and action space vectors, constructing the initial scenario, designing the reward function, and building an interactive training model. S2. Establish the dual-delay deep deterministic policy gradient algorithm and its corresponding hyperparameters; S3. Improve the gradient algorithm for dual-delay deep deterministic policies using intelligent driver models; S4. Obtain the initial state of the vehicle and use the improved algorithm for training to determine the optimal driving strategy.

[0007] Furthermore, S1 specifically includes the following steps: S101. Establish the state space and action space: The state space vector is State=[ego_v x ego_v y ego_x, ego_y, other_v x other_v y[other_x, other_y]; where ego_v x and ego_v y `ego_x` represents the velocity of the main vehicle in the x and y directions; `ego_x` and `ego_y` represent the position coordinates of the main vehicle in the x and y directions; `other_v` represents the velocity of the main vehicle in the x and y directions. x and other_v y Other_x and other_y are the velocities of the environmental vehicle in the x and y directions, respectively; other_x and other_y are the position coordinates of the environmental vehicle in the x and y directions, respectively. The action space vector is Action=[a], where a is the acceleration of the main vehicle; S102. Establish the initial scene: This includes acquiring the vehicle and collision sensors, and generating the corresponding initial positions; S103, Design the reward function; S104. Building an environmental interaction training model: Using acceleration as the primary vehicle control variable, the velocity and position of the primary vehicle are calculated based on a kinematic model:

[0008]

[0009] Where, the time difference from time t to time t+1 is Δt = 0.05s, ego_v t Corresponding to the speed of the main vehicle at time t, Corresponding to the speed of the main vehicle at time t+1, ego_x t Corresponding to the main vehicle's coordinate position at time t, The acceleration of the main vehicle corresponding to time t.

[0010] Furthermore, the reward function Specifically:

[0011] in, The collision detection identification variable is identified by the collision sensor. If a collision occurs, the value is 1, otherwise the value is 0. The corresponding coefficient k is the collision reward coefficient. b is the speed bonus coefficient. The speed bonus items are as follows:

[0012] In the formula The initial speed of the main vehicle; Main vehicle speed:

[0013] c is the expected distance reward coefficient. The expected distance bonus represents the absolute value of the difference between the target distance and the actual distance between the two vehicles; specifically:

[0014] Where rel_dis is the relative distance between the two vehicles, specifically:

[0015] In the formula, ego_x and ego_y are the x and y coordinates of the main vehicle, other_x and other_y are the x and y coordinates of the environmental vehicle; tar_dis is the expected travel distance, specifically:

[0016] in Let ego_v be the initial distance between the environmental vehicle and the host vehicle, T be the speed of the host vehicle, T be the reaction time of the host vehicle driver, and rel_v be the speed difference between the two vehicles. The maximum acceleration of the main vehicle. This is to improve the comfort and acceleration of the main vehicle.

[0017] Furthermore, the specific steps of S2 include: S201, Designing the neural network structure for actors and critics; S202, Design Hyperparameters; Furthermore, S201 specifically includes: The actor-critic neural network structure comprises two parts: an actor network and a critic network. The actor network (Actor) takes the state space vector as input via its forward propagation function. This vector then undergoes a two-layer linear transformation and ReLU activation, finally outputting the vehicle's acceleration using the tanh activation function. The critic network (Critic) takes the state space vector and action space vector concatenated by the cat function as input via its forward propagation function, where the action space vector represents the vehicle's acceleration. This vector then undergoes a two-layer linear transformation and ReLU activation, outputting the critic network's output. Both the actor and critic networks include an action network and a target network. The dual-delay deep deterministic policy gradient algorithm has one actor and two critics, totaling six layers. The action network is updated using its corresponding target network, while the target network itself is updated using a soft update method.

[0018] Furthermore, the hyperparameters of S202 include: the number of neurons in the intermediate layers of the actor-critic network (Linear1 and Linear2), the learning rate of the actor network (Lr_A), the learning rate of the critic network (Lr_C), the soft update rate (Tau), the learning rate (γ), the policy noise (Policy_Noise), the delayed update frequency (Policy_Freq), the action noise (Action_Noise), the experience replay pool capacity (Memory_Cap), and the number of samples (Batch_Size).

[0019] Furthermore, the Action_Noise uses OU noise, specifically:

[0020] Where, ou_noise is the current noise state, state_init is the state when the OU noise is initialized, θ is the recovery rate of the OU process, mu is the mean of the OU process, and ou_noise last The previous noise state is represented by δ, which indicates the standard deviation of the OU process. The OU process refers to noise continuously reverting to the mean, described by a stochastic differential equation. The specific standard deviation δ is:

[0021] Where δ max With δ min These represent the maximum and minimum standard deviations, respectively; episode is the number of training epochs in progress, and decay-period is the noise decay period.

[0022] Furthermore, S3 specifically refers to: S301. Establish an intelligent driver model, with the model's output being acceleration:

[0023] in, The acceleration output by the intelligent driver model. `ego_v` represents the maximum acceleration of the main vehicle, `ego_v0` represents the current speed of the main vehicle, and `ego_v0` represents the expected speed of the main vehicle. For acceleration index, The speed difference between the main vehicle and the environmental vehicle, The desired distance is s, which is the distance between the main vehicle and the environmental vehicle. S302. Modify the contents stored in the experience replay pool: Store the action_idm output by the intelligent driver model into the experience replay pool replay_buffer of the dual-delay deep deterministic policy gradient algorithm; S303. Modify the content of experience playback pool data collection: Perform data and format preprocessing on action_idm together and include it in the sampling; S304. Modify the update rules of the dual-delay deep deterministic policy gradient algorithm network: Use the actions of the intelligent driver model extracted from the experience replay pool to assist the update of the actor action network.

[0024] Furthermore, S4 specifically includes the following steps: S401, Obtain the initial master vehicle , The state-space vector of the direction:

[0025] S402. Input the acquired state space vector into the improved dual-delay deep deterministic policy gradient algorithm to obtain the output acceleration action. S403, Introduce an OU noise attenuation mechanism to the output main vehicle acceleration a:

[0026] S404. The action with added noise is passed to the interaction function of the environment, causing the agent to start interacting with the environment, and the function returns the next state (next_state), the reward value obtained in this interaction, and the end flag parameter (done), specifically:

[0027] S405. Add the acceleration output by the intelligent driver model to the experience replay pool for subsequent network update training, specifically:

[0028] Where state is the current state of the agent, action is the acceleration action output by the actor action network of the dual-delay deep deterministic policy gradient algorithm, action_idm is the acceleration action output by the intelligent driver model, next_state is the state of the agent at the next moment, reward is the reward value obtained in this interaction, and done is a flag indicating whether this interaction has ended. S406. Extract Batch_Size pieces of experience pool data from the experience replay pool and update the commentator action network. S407. Calculate the action value (action) in the current state using the actor's motion network. new Specifically:

[0029] In the formula, Actor represents the actor network, which takes the current state as input and calculates the returned action after forward propagation, θ. A Parameters representing the actor network; S408. Use this newly calculated action value. new The input is passed to either of the two critic action networks, and Q is calculated using the first critic action network. new Specifically:

[0030] In the formula, Critic1() represents the action network that takes the first critic as input, the current state and the new action value as input, and returns the corresponding estimated value Q after forward propagation. new θ C1 These are the parameters of the commentator's action network; S409. Calculate the gradient of the sampling policy and use gradient ascent to update the actor motion network, specifically:

[0031] In the formula ▽ θ J is the policy gradient. For gradient calculation, θ A These are the network parameters corresponding to the actor's motion network. For actor motion network, state i Let 'i' be the i-th state value, and 'a' represent the output value of the actor's motion network. S410. The target network for actors and critics is updated using a soft update method, specifically as follows:

[0032]

[0033] Where Tau is the soft update parameter, specifically 0.005. , This is an update method used by both critics' target networks.

[0034] Furthermore, the specific steps of S406 are as follows: A. The actor's target network outputs the action_target, which is then subjected to policy noise and smoothing to obtain the action_next_state, which is input to the critic's target network. Specifically:

[0035] B. Pass the next_action calculated by the actor's target network to the two critics' target networks for output, obtaining the target Q-values. Subsequently, the two critics' target networks will output two target Q-values, Q1 and Q2. At this point, the minimum value between the two Q-values ​​will be taken as the target Q-value used for temporal difference updates. Specifically:

[0036] C. Use the target Q value to calculate the target value y during time-series difference updates. i Specifically:

[0037] D. Use the gradient descent algorithm to minimize the evaluation target loss function Loss to update the action network for both commentators, specifically:

[0038] Where Q(state) i action i ) represents the predicted Q-value of the commentator's action network output, and N represents the number of commentators.

[0039] The beneficial effects of this invention are This invention introduces an IDM model into the TD3 reinforcement learning algorithm. The IDM model establishes a benchmark for action exploration in the early stages of the TD3 algorithm, significantly improving the quality of action exploration in the early stages. Furthermore, because the IDM model guides the exploration of better action strategies in the early stages, the fusion algorithm converges faster than the original TD3 algorithm and can converge to a better strategy in the later stages. Compared to the IDM model, the algorithm of this invention has stronger generalization ability. After training, the agent quickly reaches the level of the IDM model, and then, with the help of reinforcement learning, the agent can achieve decision-making and generalization abilities that surpass the IDM model. Moreover, the fusion algorithm, through training, allows the agent to reach the optimal driving speed, reducing the difficulty of selecting the desired speed parameter in the IDM model, while also possessing stronger scene adaptability and generalization ability. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1This is a diagram of the overall macroscopic network architecture of the algorithm of this invention; Figure 2 This is a diagram illustrating the entire interactive training process between the algorithm and the environment of this invention. Figure 3 It is a reward curve. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 This invention provides a vehicle intelligent driving method based on reinforcement learning. Because the IDM model requires setting the desired speed of the lead vehicle appropriately when facing different speeds of vehicles ahead, this process is cumbersome and difficult to fine-tune. If the desired speed is too high, safety will suffer; if it is too low, driving efficiency will be low. Furthermore, the lead vehicle's generalization ability is poor when using only the IDM model, making it difficult to cope with complex changes in the scenario. For example, when following a vehicle, the lead vehicle may suddenly stop or cut in, or a third vehicle may cut in from the side. In such dangerous scenarios, the IDM model often struggles to make appropriate actions. While the TD3 reinforcement learning algorithm obtains an agent model capable of making a better strategy through trial-and-error interaction and has some generalization ability, its slow convergence speed leads to high training costs, and the quality of the final strategy depends heavily on the quality of the initial strategy exploration. However, TD3 reinforcement learning's action strategies are relatively random during the initial exploration, making it difficult to guarantee exploration quality. Therefore, this invention proposes a novel decision-making and planning method, which includes the following steps: Step 1: Building the simulation scenario: The building process includes establishing the state space and action space, constructing the initial scenario and designing the reward function, and building the interactive training model; (1) Establish state space and action space Establish dimensions as The state space; specifically, the state space is set as the velocities ego_v of the main vehicle in the x and y directions. x and ego_v y The position coordinates of the main vehicle in the x and y directions are ego_x and ego_y; the velocity of the environmental vehicle in the x and y directions is other_v. x and other_v y The position coordinates of the vehicle in the x and y directions are other_x and other_y, respectively, which represent the state space State = [ego_v]. x ego_vy ego_x, ego_y, other_v x other_v y [other_x, other_y]. Design a continuous action space with dimension 1. The specific continuous action space is selected as the main vehicle acceleration, i.e., the action space Action=[a].

[0044] (2) Establish the initial scene Acquire vehicle and collision sensor data, and generate corresponding initial positions.

[0045] (3) Design reward function reward function Specifically:

[0046] in, The collision detection identification variable is identified by the collision sensor. If a collision occurs, the value is 1; otherwise, the value is 0. The corresponding coefficient k is the collision reward coefficient, with a value of -200.

[0047] The speed bonus items are as follows:

[0048] In the formula The speed of the main vehicle is specifically derived by adding the square roots of the squares of its speeds in the x and y directions. The initial speed of the main vehicle is given by b, which is the speed reward coefficient with a value of 1. The coefficient multiplied in front of the speed reward function depends on the actual interaction between the agent and the environment. Too large or too small coefficients can easily lead to vehicle collisions and excessively slow vehicle speeds.

[0049] The expected distance bonus represents the absolute value of the difference between the target distance and the actual distance between the two vehicles. c is the expected distance bonus coefficient, with a value of -0.5. Specifically:

[0050] In the formula, rel_dis represents the relative distance between the two vehicles, specifically:

[0051] In the formula, ego_x and ego_y are the x and y coordinates of the main vehicle, and other_x and other_y are the x and y coordinates of the environmental vehicle.

[0052] In the formula, tar_dis represents the desired travel distance, specifically:

[0053] In the formula, s0 is the initial distance between the environmental vehicle and the host vehicle, ego_v is the speed of the host vehicle, T is the reaction time of the host vehicle driver, and rel_v is the speed difference between the two vehicles. These are the maximum acceleration of the main vehicle and the comfortable acceleration of the main vehicle, respectively, where ego_v specifically refers to:

[0054] (4) Build an environment interaction training model: Using acceleration as the primary vehicle control variable, the velocity and position of the primary vehicle are calculated based on a kinematic model:

[0055]

[0056] Where Δt is 0.05s, ego_v t Corresponding to the speed of the main vehicle at time t, ego_x t Corresponding to the main vehicle's coordinate position at time t, The acceleration of the main vehicle corresponding to time t.

[0057] Step 2: Design the dual-delay deep deterministic policy gradient algorithm and its corresponding hyperparameters; (1) Designing the neural network structure of actors-critics The forward propagation function of the Actor network takes a 1×8 dimensional state space as input, i.e., the velocity and position coordinates (ego_v) of the main vehicle in the x and y directions. x ego_v y and ego_x, ego_y; the velocity and position coordinates of the vehicle in the x and y directions, other_v x other_v y The input to the other_x and other_y layers is a linear transformation and ReLU activation function applied to two intermediate layers. Finally, the output layer uses the tanh activation function as the output value, specifically the vehicle's acceleration. The Critic network's forward propagation function takes a 1×8 dimensional state space vector and a 1-dimensional action space vector concatenated by the cat function. The action space vector is the vehicle's acceleration. This vector is then processed through two intermediate layers of linear transformation and ReLU activation function, and the output layer outputs the critic's Q-value. The actor and critic networks each have an action network and a target network, respectively. The dual-delay deep deterministic policy gradient algorithm has one actor and two critics, resulting in a total of six network layers. The action network is updated using its corresponding target network, while the target network itself is updated using a soft update method.

[0058] (2) Hyperparameter design The hyperparameters should include at least the following: the number of neurons in the intermediate layers of the actor / critic network (Linear1 and Linear2), the learning rate of the actor network (Lr_A), the learning rate of the critic network (Lr_C), the soft update rate (Tau), the learning rate (γ), the policy noise (Policy_Noise), the delayed update frequency (Policy_Freq), the action noise (Action_Noise), the experience replay pool capacity (Memory_Cap), and the number of samples (Batch_Size). Specifically, the action noise parameter uses OU (Ornstein-Uhlenbeck Noise), which is a dynamically changing parameter that decreases as the training epochs increase. This improves the agent's exploratory nature in the early stages and reduces the interference of noise on convergence actions in the later stages. Specifically:

[0059] Where, ou_noise is the current noise state, state_init is the state when the OU noise is initialized, θ is the recovery rate of the OU process, mu is the mean of the OU process, and ou_noise last Let δ represent the previous noise state, and δ be the standard deviation of the OU process. The OU process mainly refers to noise continuously reverting to the mean, often described using stochastic differential equations. The specific value of the standard deviation δ is:

[0060] In the formula δ max With δ min These correspond to the maximum and minimum values ​​of the standard deviation, respectively. "episode" refers to the number of rounds in the current training session, and "decay-period" refers to the noise decay period.

[0061] Step 3: Improve the dual-delay deep deterministic policy gradient algorithm based on the intelligent driver model; obtain the IDM-TD3 algorithm; (1) Establish an intelligent driver model, with the model's output being acceleration:

[0062] in, The acceleration output by the intelligent driver model. `ego_v` represents the maximum acceleration of the main vehicle, `ego_v0` represents the current speed of the main vehicle, and `ego_v0` represents the expected speed of the main vehicle. For acceleration index, The speed difference between the main vehicle and the environmental vehicle, The desired distance is s, where s is the distance between the main vehicle and the environmental vehicle.

[0063] (2) Modify the contents stored in the experience replay pool: Store the action_idm output by the IDM algorithm into the experience replay pool replay_buffer of the double-delay deep deterministic policy gradient algorithm; (3) Modify the content of the experience playback pool data collection: Perform data and format preprocessing on action_idm together and participate in the sampling.

[0064] (4) Modify the update rules of the dual-delay deep deterministic policy gradient algorithm network: use the IDM algorithm extracted from the experience replay pool to update the action aid actor action network.

[0065] Step 4: Based on the acquired initial state information, the neural network of the improved algorithm is fed into the deep reinforcement learning training to determine the optimal policy; the TD3 algorithm is improved by fusing the intelligent driver model, and on this basis, a deep reinforcement learning model in the continuous action space is trained based on the dual-delay deep deterministic policy gradient algorithm to solve for the optimal policy.

[0066] (1) Obtain the initial master vehicle , The 1×8 dimensional state space vector in the direction is as follows:

[0067] (2) Pass the obtained state space vector to the forward propagation function of the improved algorithm's actor motion network to obtain the output motion values. That is, the acceleration of the main vehicle.

[0068] (3) An OU noise attenuation mechanism is introduced into the output vehicle acceleration a to improve the agent's exploration performance. At the same time, the attenuation mechanism can reduce the interference of later noise on the convergence strategy. Specifically:

[0069] (4) such as Figure 2 The noisy action is passed to the interaction function of the environment, causing the agent to begin interacting with the environment and returning the next state (next_state), the reward value obtained in this interaction, and the end flag parameter (done).

[0070] (5) The acceleration output by the IDM algorithm is added to the experience replay pool for subsequent network update training, specifically:

[0071] In the formula, state is the current state of the agent, action is the acceleration action output by the actor action network of the dual-delay deep deterministic policy gradient algorithm, action_idm is the acceleration action output by the intelligent driver model, next_state is the state of the agent at the next moment, reward is the reward value obtained in this interaction, and done is a flag indicating whether this interaction has ended.

[0072] (6) Extract Batch_Size pieces of experience pool data from the experience replay pool and update the commentator action network. The main steps are as follows: A. The actor's target network outputs the action_target, which is then subjected to policy noise and smoothing to obtain the action_next_state, which is input to the critic's target network. Specifically:

[0073] B. Pass the next_action calculated by the actor's target network to the two critics' target networks for output, obtaining the target Q-values. Subsequently, the two critics' target networks will output two target Q-values, Q1 and Q2. At this point, the minimum value between the two Q-values ​​will be taken as the target Q-value used for temporal difference updates. Specifically:

[0074] C. Use the target Q value to calculate the target value y during time-series difference updates. i Specifically:

[0075] D. Use the gradient descent algorithm to minimize the evaluation target loss function Loss to update the action network for both commentators, specifically:

[0076] In the formula, Q(state) i action i ) represents the predicted Q-value of the commentator's action network output, and N represents the number of commentators, specifically 2.

[0077] (7) Calculate the action value in the current state using the actor's motion network. new Specifically:

[0078] In the formula, Actor represents the actor's motion network, which takes the current state as input, calculates the returned action after forward propagation, and θ A This represents the parameters of the actor's motion network.

[0079] (8) Calculate this newly obtained action value. new The input is passed to either of the two critic action networks, and Q is calculated using the first critic action network. new Specifically:

[0080] In the formula, Critic1 represents the action network of the first critic, which takes the current state and new action value as input, and returns the corresponding estimated value Q after forward propagation. new θ C1 These are the parameters of the commentator's action network.

[0081] (9) Calculate the sampling policy gradient and use the gradient ascent method to update the actor motion network, specifically:

[0082] In the formula ▽ θ J is the policy gradient. For gradient calculation, θ A These are the network parameters corresponding to the actor's motion network. For actor motion network, state i Let be the i-th state value, and 'a' represent the output value of the actor's motion network.

[0083] (10) The target network of actors and critics is updated using a soft update method, specifically:

[0084]

[0085] In the formula, Tau is the soft update parameter, with a specific value of 0.005. , This is an update method used by both critics' target networks.

[0086] The reward curve of this invention is as follows: Figure 3 As shown, through multiple rounds of repeated interactive training from steps 1 to 4, the algorithm will gradually converge, and the decision-making level of the agent will continuously improve. In the real environment, this translates to the lead vehicle maintaining a safe following distance with high driving efficiency when the preceding vehicle is traveling, or being able to promptly decelerate or brake when the preceding vehicle suddenly stops or cuts into the lane from the side, thus enabling it to complete the driving task effectively. Training results show that the TD3 algorithm incorporating IDM achieves significant improvements in convergence speed, policy stability, and reward acquisition capability.

[0087] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A vehicle intelligent driving method based on reinforcement learning, characterized in that, Includes the following steps: S1. Build a vehicle driving simulation scenario; This includes establishing state space vectors and action space vectors, constructing an initial scene, designing a reward function, and building an interactive training model. S2. Establish the dual-delay deep deterministic policy gradient algorithm and its corresponding hyperparameters; S3. Improve the dual-delay deep deterministic policy gradient algorithm using an intelligent driver model; specifically: S301. Establish an intelligent driver model, with the model's output being acceleration: ； in, The acceleration output by the intelligent driver model. `ego_v` represents the maximum acceleration of the main vehicle, `ego_v0` represents the current speed of the main vehicle, and `ego_v0` represents the expected speed of the main vehicle. For acceleration index, The speed difference between the main vehicle and the environmental vehicle, The desired distance is s, which is the distance between the main vehicle and the environmental vehicle. S302. Modify the contents stored in the experience replay pool: Store the action_idm output by the intelligent driver model into the experience replay pool replay_buffer of the dual-delay deep deterministic policy gradient algorithm; S303. Modify the content of experience playback pool data collection: Perform data and format preprocessing on action_idm together and include it in the sampling; S304. Modify the network update rules of the dual-delay deep deterministic policy gradient algorithm: use the actions of the intelligent driver model extracted from the experience replay pool to assist the update of the actor action network. S4. Obtain the initial state of the vehicle and train it using the improved algorithm to determine the optimal driving strategy; specifically: S401, Obtain the initial master vehicle , The state-space vector of the direction: ； S402. Input the acquired state space vector into the improved dual-delay deep deterministic policy gradient algorithm to obtain the output acceleration action. S403, Introduce an OU noise attenuation mechanism to the output main vehicle acceleration a: ； S404. The action with added noise is passed to the interaction function of the environment, causing the agent to start interacting with the environment, and the function returns the next state (next_state), the reward value obtained in this interaction, and the end flag parameter (done), specifically: ； S405. Add the acceleration output by the intelligent driver model to the experience replay pool for subsequent network update training, specifically: ； Where state is the current state of the agent, action is the acceleration action output by the actor action network of the dual-delay deep deterministic policy gradient algorithm, action_idm is the acceleration action output by the intelligent driver model, next_state is the state of the agent at the next moment, reward is the reward value obtained in this interaction, and done is a flag indicating whether this interaction has ended. S406. Extract Batch_Size pieces of experience pool data from the experience replay pool and update the commentator action network. S407. Calculate the action value (action) in the current state using the actor's motion network. new Specifically: ； In the formula, Actor represents the actor network, which takes the current state as input, calculates the return action after forward propagation, and θ A Parameters representing the actor network; S408. Use this newly calculated action value. new The input is passed to either of the two critic action networks, and Q is calculated using the first critic action network. new Specifically: ； In the formula, Critic1() represents the action network that takes the first critic as input, the current state and the new action value as input, and returns the corresponding estimated value Q after forward propagation. new θ C1 These are the parameters of the commentator's action network; S409. Calculate the gradient of the sampling policy and use gradient ascent to update the actor motion network, specifically: ； In the formula ▽ θ J is the policy gradient. For gradient calculation, θ A These are the network parameters corresponding to the actor's motion network. For actor motion network, state i Let 'i' be the i-th state value, and 'a' represent the output value of the actor's motion network. S410. The target networks for actors and critics are updated using a soft update method, specifically: ； ； Where Tau is the soft update parameter, specifically 0.

005. , This is an update method used by both critics' target networks.

2. The intelligent vehicle driving method based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes the following steps: S101. Establish the state space and action space: The state space vector is State=[ego_v x ego_v y ego_x, ego_y, other_v x other_v y [other_x, other_y]; where ego_v x and ego_v y `ego_x` represents the velocity of the main vehicle in the x and y directions; `ego_x` and `ego_y` represent the position coordinates of the main vehicle in the x and y directions; `other_v` represents the velocity of the main vehicle in the x and y directions. x and other_v y Other_x and other_y are the velocities of the environmental vehicle in the x and y directions, respectively; other_x and other_y are the position coordinates of the environmental vehicle in the x and y directions, respectively. The action space vector is Action=[a], where a is the acceleration of the main vehicle; S102. Establish the initial scene: This includes acquiring the vehicle and collision sensors, and generating the corresponding initial positions; S103, Design the reward function; S104. Building an environmental interaction training model: Using acceleration as the primary vehicle control variable, the velocity and position of the primary vehicle are calculated based on a kinematic model: ； ； Where, the time difference from time t to time t+1 is Δt = 0.05s, ego_v t Corresponding to the speed of the main vehicle at time t, Corresponding to the speed of the main vehicle at time t+1, ego_x t Corresponding to the main vehicle's coordinate position at time t, The acceleration of the main vehicle corresponding to time t.

3. The intelligent vehicle driving method based on reinforcement learning according to claim 2, characterized in that, reward function Specifically: ； in, The collision detection identification variable is identified by the collision sensor. If a collision occurs, the value is 1, otherwise the value is 0. The corresponding coefficient k is the collision reward coefficient. b is the speed bonus coefficient. The speed bonus items are as follows: ； In the formula The initial speed of the main vehicle; Main vehicle speed: ； c is the expected distance reward coefficient. The expected distance bonus represents the absolute value of the difference between the target distance and the actual distance between the two vehicles; specifically: ； Where rel_dis is the relative distance between the two vehicles, specifically: ； In the formula, ego_x and ego_y are the x and y coordinates of the main vehicle, other_x and other_y are the x and y coordinates of the environmental vehicle; tar_dis is the expected travel distance, specifically: ； in Let ego_v be the initial distance between the environmental vehicle and the host vehicle, T be the speed of the host vehicle, T be the reaction time of the host vehicle driver, and rel_v be the speed difference between the two vehicles. The maximum acceleration of the main vehicle. This is to improve the comfort and acceleration of the main vehicle.

4. The intelligent vehicle driving method based on reinforcement learning according to claim 1, characterized in that, The specific steps in S2 include: S201, Designing the neural network structure for actors and critics; S202, Design hyperparameters.

5. The intelligent vehicle driving method based on reinforcement learning according to claim 4, characterized in that, Specifically, S201 is as follows: The actor-critic neural network structure comprises two parts: an actor network and a critic network. The actor network (Actor) takes the state space vector as input via its forward propagation function. This vector then undergoes a two-layer linear transformation and ReLU activation, finally outputting the vehicle's acceleration using the tanh activation function. The critic network (Critic) takes the state space vector and action space vector concatenated by the cat function as input via its forward propagation function, where the action space vector represents the vehicle's acceleration. This vector then undergoes a two-layer linear transformation and ReLU activation, outputting the critic network's output. Both the actor and critic networks include an action network and a target network. The dual-delay deep deterministic policy gradient algorithm has one actor and two critics, totaling six layers. The action network is updated using its corresponding target network, while the target network itself is updated using a soft update method.

6. The intelligent vehicle driving method based on reinforcement learning according to claim 4, characterized in that, The S202 hyperparameters include: the number of neurons in the middle layers of the actor / critic network (Linear1 and Linear2), the learning rate of the actor network (Lr_A), the learning rate of the critic network (Lr_C), the soft update rate (Tau), the learning rate (γ), the policy noise (Policy_Noise), the delayed update frequency (Policy_Freq), the action noise (Action_Noise), the experience replay pool capacity (Memory_Cap), and the number of samples (Batch_Size).

7. The intelligent vehicle driving method based on reinforcement learning according to claim 6, characterized in that, Action noise uses OU noise, specifically: ； Where, ou_noise is the current noise state, state_init is the state when the OU noise is initialized, θ is the recovery rate of the OU process, mu is the mean of the OU process, and ou_noise last The previous noise state is represented by δ, which indicates the standard deviation of the OU process. The OU process refers to noise continuously reverting to the mean, described by a stochastic differential equation. The specific standard deviation δ is: ； Where δ max With δ min These represent the maximum and minimum standard deviations, respectively; episode is the number of training epochs in progress, and decay-period is the noise decay period.

8. The intelligent vehicle driving method based on reinforcement learning according to claim 1, characterized in that, The specific steps of S406 are as follows: A. The actor's target network outputs the action_target, which is then subjected to policy noise and smoothing to obtain the action_next_state, which is input to the critic's target network. Specifically: ； B. Pass the next_action calculated by the actor's target network to the two critics' target networks for output, obtaining the target Q-values. Subsequently, the two critics' target networks will output two target Q-values, Q1 and Q2. At this point, the minimum value between the two Q-values ​​will be taken as the target Q-value used for temporal difference updates. Specifically: ； C. Use the target Q value to calculate the target value y during time-series difference updates. i Specifically: ； Where γ is the learning rate; D. Use the gradient descent algorithm to minimize the evaluation target loss function Loss to update the action network for both commentators, specifically: ； Where Q(state) i action i ) represents the predicted Q-value of the commentator's action network output, and N represents the number of commentators.

Images