Vehicle following behavior decision-making method based on improved DDPG algorithm

Through the improvement of dual experience playback pool and reward function, the traditional DDPG algorithm is solved, and the problem of instability in training and insufficient high-dimensional state processing capabilities in vehicle follow-up behavior decisions is achieved, and efficient and high-precision vehicle follow-up behavior decisions are improved. The decision-making capabilities of the autonomous driving system.

CN118928397APending Publication Date: 2024-11-12NINGBO UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411096657.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional DDPG algorithms are susceptible to noise and randomness in vehicle follow-up behavior decisions, and are unstable in training, making them difficult to efficiently handle high-dimensional state spaces, resulting in low decision efficiency and accuracy.

Method used

Dual experience playback pool and delay interval random sampling are used, and the reward function with safety, efficiency and comfort considerations are combined to improve decision efficiency and accuracy by improving the DDPG algorithm.

Benefits of technology

It realizes efficient and high-precision vehicle follow-up behavior decision-making, and improves the decision-making capabilities of the autonomous driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118928397A_ABST
    Figure CN118928397A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicle behavior decision making, in particular to a vehicle following behavior decision making method based on an improved DDPG algorithm, which comprises the following steps: screening out vehicle following event data meeting conditions according to an original natural data set; the method comprises the following steps of: screening out vehicle following events, selecting proper driving behavior data characteristics in a data set, establishing a vehicle following model based on an improved DDPG algorithm, setting a state space, setting an action space and a reward function R, comprehensively considering three factors, and ending a training process when training and strategy convergence or a preset training round number is reached; according to the method, storage and reuse of historical empirical data are facilitated through dual empirical playback pools, two independent playback pools are introduced, delayed interval random sampling is carried out to carry out data acquisition, decision making efficiency is planned to be improved, a reward function comprehensively considers three aspects, namely safety, efficiency and comfort, so that an intelligent agent can consider comprehensive performance of a current time step, and decision making efficiency is improved. Therefore, the high-precision vehicle following behavior decision is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle behavior decision-making, and in particular to a vehicle following behavior decision-making method based on an improved DDPG algorithm. Background Art

[0002] With the rapid development of vehicle networking, vehicle-road collaboration, artificial intelligence and other disciplines and technologies, today's vehicle driving is gradually developing towards intelligence, efficiency and safety. As a major branch of intelligent driving, following a car has become the most common driving behavior in daily driving. The main goal of following a car is for the driver to adjust his own speed to the same speed as the slower car and keep the distance between the cars within the expected value.

[0003] In recent years, deep reinforcement learning (DRL) has shown great potential in the field of autonomous driving. For example, the autonomous driving method and system based on deep reinforcement learning disclosed in the invention patent with publication number CN112904864B and the autonomous driving vehicle control method integrating physical information and deep reinforcement learning disclosed in the invention patent with publication number CN116052412B can both realize autonomous driving of vehicles.

[0004] As an advanced deep reinforcement learning algorithm, the Deep Deterministic Policy Gradient (DDPG) algorithm combines the advantages of deep learning and reinforcement learning, and can efficiently learn and make decisions in continuous action space. Therefore, the application research of DDPG algorithm in autonomous driving has gradually increased. In practical applications, the vehicle following behavior decision method based on the traditional DDPG algorithm is susceptible to noise and random factors during the training process, resulting in instability of the training process and easy to fall into the local optimal solution, which will make the vehicle unable to make the best decision in a complex traffic environment and affect the decision accuracy. On the other hand, the processing capacity in the high-dimensional state space is insufficient. The vehicle following behavior involves a large amount of environmental perception information and decision variables. The traditional DDPG algorithm is difficult to effectively process and learn this information in a short time, resulting in a decrease in decision delay. Therefore, the decision efficiency and accuracy of the existing vehicle following behavior decision method based on the traditional DDPG algorithm are not high, and there is a large room for improvement. Therefore, a vehicle following behavior decision method based on the improved DDPG algorithm with high decision efficiency and accuracy is urgently needed to improve the above problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a vehicle following behavior decision-making method based on the improved DDPG algorithm, which helps store and reuse historical experience data through a dual experience replay pool, introduces two independent replay pools and performs delayed interval random sampling to acquire data to improve the efficiency of decision-making. The reward function comprehensively considers three aspects: safety, efficiency and comfort, so that the intelligent body can take into account the comprehensive performance of the current time step, thereby achieving a vehicle following behavior decision with higher precision.

[0006] A vehicle following behavior decision method based on an improved DDPG algorithm of the present invention comprises the following steps:

[0007] S1. Based on the original natural data set, the data of car-following events that meet the conditions are screened out. The screened car-following events consider the following of the front car by default, and do not directly consider the following car. This is because when drivers are driving on the highway, they usually think that the driving behavior of the vehicle behind them is reasonable, that is, the following car will not rear-end the front car by default. Finally, 70% of the screened car-following events are used as the training data set, and 30% as the test data set;

[0008] S2. Select appropriate driving behavior data features from the data set. Specifically, the selected feature data include FV_Vel, FV_Acc, Space_Headwauy, V_Rel, LV_Vel and LV_Acc, THW and TTCi, which represent the speed and acceleration of the following vehicle, the relative distance between the following vehicle and the leading vehicle, the relative speed, and the speed and acceleration of the leading vehicle, respectively.

[0009] S3. Clarify the correlation between the driving behavior characteristic parameters in the data set, and perform Pearson correlation analysis on the data features selected from the data set;

[0010] S4. Establish a vehicle following model based on the improved DDPG algorithm and set up a reinforcement learning environment for the vehicle following scenario, including the state space, action space and reward function.

[0011] S5. Set the state space and select the longitudinal vehicle velocity v FV , relative speed and the position on the longitudinal trajectory The DDPG agent is able to control the vehicle's trajectory by controlling the speed of the following vehicle. The acceleration value of the vehicle in the process of following the vehicle is obtained by using the DDPG algorithm. The update state is calculated by the classical kinematic model, which is a physical model used to describe the motion of the vehicle. In the process of longitudinal motion of the vehicle, the classical kinematic model is used to calculate the changes of the vehicle's speed, relative speed and position over time.

[0012] S6. Setting the action space: The output of the action space is the acceleration value of the vehicle, and then the future speed, relative speed and relative distance are iteratively generated according to the motion point equation;

[0013] S7, the reward function R comprehensively considers three factors, which are used to evaluate the behavior of the vehicle and provide feedback based on these behaviors, so as to promote the model to develop in the desired direction. The main factors considered are safety, efficiency and comfort;

[0014] S8. Initialize the participant network parameters θ μ and the critic network parameters θ Q , initialize the participant target network θ μ′ and the critic target network θ Q′ , initialize the experience pool D1 and experience pool D2, where the participant network is responsible for iteratively updating the network parameters and is responsible for updating the network parameters according to the current state S t Select the current action a t , which is used to interact with the environment, and the target actor network is responsible for replaying the next state S sampled in the pool based on the experience t+1 ) Select the best next action a t+1 , the critic network is responsible for iteratively updating the value network parameters and calculating the current action value q(s t ,a t ), the critic target network is responsible for calculating the target action value q(s t+1 ,a t+1 )part;

[0015] S9, start training, observe the current state S t , select action a according to the policy network t , perform action a t Get Rewards t And observe the new state S t+1 ;

[0016] S10, according to the feedback of the reward value, {s t ,a t ,s t+1 ,r tThe four-tuple experience data of} is divided into positive samples and negative samples, and stored in two experience replay pools respectively. The positive samples are samples that complete the car-following task and samples that neither complete the car-following task nor collide with each other. The negative samples are samples that collide with each other.

[0017] S11, sampling from the experience pool, first sampling from the positive sample experience pool D1, and then randomly sampling from the D1 experience pool and the D2 experience pool in proportion after a fixed delay, and then updating the network

[0018] S12. Use the critic target network to calculate the Q value of the target:

[0019] y t′ =r t′ +γQ(s t′ ,a t′ |θ Q )

[0020] Among them, y t′ represents the target Q value, which is used to update the critic network, r t′ At time step t i The immediate reward obtained, γ represents the discount factor, and γ is between 0 and 1, indicating the importance of future rewards, Q(s t′ ,a t′ |θ Q ) represents the critic execution network with parameters θ Q Based on the state t′ and action a t′ , the calculated Q value;

[0021] S13. Update the parameters θ of the critic network by minimizing the loss function Q , the loss function L takes the form of mean square error, which is the sum of the squares of the differences between the actual Q value and the target Q value, and then takes the average:

[0022]

[0023] Among them, L is the value of the loss function; N is the batch size, that is, the number of samples used in each parameter update, y i represents the target Q value of the i-th sample, which is obtained by step S12. i -a i θ Q ) represents the state s of the Q network for the i-th sample i and action a i The predicted Q value of

[0024] S14. Update the parameters θ of the participant network using policy gradient μ ;

[0025] S15. Update the parameters of the critic target network based on the updated parameters of the critic network, and update the parameters of the participant target network based on the updated parameters of the participant network; update in a soft update manner:

[0026]

[0027] Among them, ← represents assignment, v is a preset hyperparameter, a small value between 0 and 1, which is used to control the speed of updating the target network parameters, and θ μ is the participant network parameter, θ μ′ is the participant target network parameter, θ Q is the critic network parameter, θ Q′ is the critic target network parameter;

[0028] S16: When the strategy converges or reaches the preset number of training rounds, terminate the training process.

[0029] Preferably, the rules for filtering out vehicle following events include:

[0030] S1.1. To facilitate the analysis of vehicle-following events, the vehicle type in the data set is selected as a sedan, excluding motorcycles and large vehicles;

[0031] S1.2. The vehicle number identified by the radar remains unchanged, indicating that the front and rear vehicles in the car-following event are a unique group of vehicles;

[0032] S1.3. All following vehicles are in the same lane;

[0033] S1.4. The duration of following the vehicle must be >15s and continuous to ensure that there is enough data for analysis during the following time.

[0034] Preferably, the THW and TTCi are expressed as follows:

[0035] THW(t)=d FV,LV (t) / v FV (t)

[0036] TTCi(t)=Δv rel (t) / d FV,LV (t)

[0037] THW is the time from the front center of vehicle FV to the front center of the preceding vehicle LV, d FV,LV (t) The distance between the following vehicle FV and the leading vehicle LV at time t, v FV (t) represents the speed of the following vehicle at time t;

[0038] TTC refers to the time required for two vehicles to collide during the following process, also known as the minimum collision time, which ensures the minimum distance between vehicles under dangerous braking. When the relative speed of the vehicles is 0, TTC will have no solution, so the deformation of TTC is TTCi, that is, the inverse of TTC, Δv rel (t) represents the relative speed between the following vehicle and the preceding vehicle at time t, d FV,LV (t) represents the distance between the following vehicle and the preceding vehicle at time t.

[0039] Preferably, the Pearson correlation analysis calculation formula is:

[0040]

[0041] Among them, ρ(x,y) is the Pearson correlation coefficient, which indicates the linear correlation between variables x and y, cov(x,y) indicates the covariance between variables x and y, σ(x) is the standard deviation of variable x, σ(y) is the standard deviation of variable y, E[(x-μ x )(y-μ y )] is the covariance calculation formula of variables x and y, E represents the expected value, μ x is the mean of variable x, μ y is the mean of the variable y, where the correlation coefficient ρ(x,y) ranges from [-1, 1]. Positive and negative values ​​indicate positive and negative correlations between the two variables, respectively. When the absolute value of the correlation coefficient between the two variables is greater than 0.8, the two variables are extremely correlated. When the absolute value of the correlation coefficient is less than 0.4, the correlation between the two variables is weak.

[0042] Preferably, the reward function R is calculated as follows:

[0043] R total =w1R1+w2R2+w3R3+w4R4+w5R5

[0044] Among them, R1 is represented as the reward function of safety, which is measured by the inverse of the minimum collision time TTCi to ensure that the vehicle is traveling within a specific threshold. The safety reward function considers the minimum collision time of the vehicle and defines the reward value in the form of a logarithmic function to encourage the vehicle to maintain a safe distance during the following process to avoid collision risks. Its design ensures that when the vehicle is close to or below the safe time distance threshold, the agent will be punished, and when it is outside the safe distance, the reward value is zero, which maintains the smoothness and rationality of the reward function;

[0045] The safety reward function is:

[0046]

[0047] TTCi represents the inverse value of the current minimum collision time. * represents the safety threshold, that is, under this threshold, the current vehicle distance is considered safe. If the current headway is less than this threshold, it is considered that there is a collision risk. α represents the weight parameter;

[0048] R2 is represented as an efficiency reward function, which is measured by the headway time width (THW) in the car-following scenario. The efficiency reward function considers the headway time width (THW) and combines it with kernel density estimation to enable the agent to maintain an efficient and safe following distance during the car-following process. The function gives a higher reward when the headway time width is small, and gradually reduces the reward value when the headway time width is too large, ensuring that the agent can make the best decision in various situations.

[0049] The efficiency reward function is:

[0050]

[0051] THW(t) represents the current headway, that is, the time interval between the current vehicle and the preceding vehicle, Threshold * is the threshold. When the headway is greater than this threshold, the efficiency reward will be penalized. β is an adjustment parameter used to control the speed at which the reward value decays. f KDE is the THW distribution function value calculated based on kernel density estimation, K is the kernel function used for kernel density estimation, h is the bandwidth parameter used to control the smoothness of kernel density estimation, x is the current THW value, and x i are independently distributed sampling points, n is the amount of data;

[0052] R3 is expressed as a comfort reward function, which considers the jerk of the vehicle and combines the reference value J value , encourages the agent to maintain a stable driving behavior during the vehicle following process, thereby improving the comfort of passengers. The function gives negative rewards when the jerk is large, which drives the agent to optimize its control strategy to reduce acceleration changes, thereby improving driving comfort;

[0053] The comfort function is:

[0054]

[0055] Jerk represents the vehicle's jerk, that is, the rate of change of acceleration, defined as J value Expressed as a baseline value, it is defined as the sum of the squares of the acceleration changes over a period of time, a max Indicates the maximum acceleration of the vehicle within a period of time ΔT, a min Indicates the minimum acceleration of the vehicle within a period of time ΔT.

[0056] Preferably, R4 and R5 are used for collision and early stop scenarios in vehicle following behavior decision making, and define penalty values ​​when collision and early stop occur respectively;

[0057] R4 encourages the agent to avoid collisions during vehicle following by penalizing the relative speed difference when a collision occurs. Since collision is an undesirable event, the value of this function is usually negative. Through the logarithmic function, the penalty value can be smoothly increased to make it logarithmically related to the relative speed difference.

[0058] R5 is to encourage the agent to avoid early stops during the following process by penalizing the relative distance difference when stopping early. This function is usually used to encourage vehicles to keep a distance when following. In order to prevent vehicles from stopping early frequently, its value is usually negative to avoid unreasonable early stops.

[0059] The formula is:

[0060]

[0061] δ and η represent weight parameters, which are used to adjust the size of the reward value. Indicates the relative speed between the front and rear vehicles. iscollision is a Boolean variable that is true when a collision occurs. Indicates the relative distance between the front and rear vehicles. isstop is a Boolean variable, which is true when the vehicle stops early.

[0062] Preferably, in S14, the policy gradient is used to update the parameter θ of the participant network μ :

[0063]

[0064] in, Represents the gradient of the strategy, which represents the objective function J that we want to maximize with respect to the strategy parameter θ μ The gradient of the policy network parameter θ can be optimized by maximizing the expected reward J μ , It means averaging N samples in a batch, Q(s,aθ Q ) represents the critic network used to estimate the future cumulative reward obtained by performing action a in state s, with parameter θ Q , μ(s; θ μ ) indicates that the participant network outputs action a in a given state s, and the parameter is θ μ ,s i is the state of the i-th sample, μ(s i ) indicates that the policy network is in state s i The output action is: a=μ(s i ) represents the Q function Q(s,aθ Q ) The gradient of action a, evaluated at state s i Use the strategy μ to calculate the action μ(s i ), represents the policy function μ(s; θ μ ) for the policy parameter θ μ The gradient of , evaluated at state s i The action μ(s) output by the next policy network i ) for the parameter θ μ gradient.

[0065] Preferably, the participant network and the critic network are both composed of an input layer, a hidden layer and an output layer, and the hidden layers of the participant network and the critic network are 30 neurons each, and the input layer of the participant network has three inputs, namely, the longitudinal vehicle speed v FV , relative speed Δ vt and the relative position on the longitudinal trajectory Δ dt , the output layer is the acceleration a value, which will also serve as the input of the critic network, so the critic network has 4 inputs and the output layer is the Q value.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a vehicle following behavior decision method based on the improved DDPG algorithm with high decision-making efficiency and accuracy. The dual experience replay pool can help store and reuse historical experience data. The introduction of two independent replay pools and delayed interval random sampling for data acquisition is intended to improve the efficiency of decision-making. The reward function comprehensively considers three aspects: safety, efficiency and comfort, which enables the intelligent body to take into account the comprehensive performance of the current time step, thereby achieving a higher-precision vehicle following behavior decision, which provides a new research direction and technical support for the development of future autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of a vehicle following behavior decision method based on an improved DDPG algorithm of the present invention;

[0068] Figure 2 2. It is a schematic diagram of the vehicle following behavior trajectory structure of the present invention;

[0069] Figure 3 It is a schematic diagram of the convolutional neural network structure of the present invention. DETAILED DESCRIPTION

[0070] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0071] Example

[0072] like Figures 1 to 3 As shown, a vehicle following behavior decision method based on an improved DDPG algorithm includes the following steps:

[0073] S1. Based on the original natural data set, the data of car-following events that meet the conditions are screened out. The screened car-following events consider the following of the front car by default, and do not directly consider the following car. This is because when drivers are driving on the highway, they usually think that the driving behavior of the vehicle behind them is reasonable, that is, the following car will not rear-end the front car by default. Finally, 70% of the screened car-following events are used as the training data set, and 30% as the test data set;

[0074] S2. Select appropriate driving behavior data features from the data set. Specifically, the selected feature data include FV_Vel, FV_Acc, Space_Headwauy, V_Rel, LV_Vel and LV_Acc, THW and TTCi, which represent the speed and acceleration of the following vehicle, the relative distance between the following vehicle and the leading vehicle, the relative speed, and the speed and acceleration of the leading vehicle, respectively.

[0075] S3. Clarify the correlation between the driving behavior characteristic parameters in the data set, and perform Pearson correlation analysis on the data features selected from the data set;

[0076] S4. Establish a vehicle following model based on the improved DDPG algorithm and set up a reinforcement learning environment for the vehicle following scenario, including the state space, action space and reward function.

[0077] S5. Set the state space and select the longitudinal vehicle velocity v FV , relative speed and the position on the longitudinal trajectory The DDPG agent can control the vehicle's trajectory by controlling the speed of the following vehicle. The acceleration value of the vehicle in the process of following the vehicle is obtained by using the DDPG algorithm. The state update state is calculated by the classical kinematic model. In the process of longitudinal motion of the vehicle, it is used to calculate the changes of the vehicle's speed, relative speed and position over time. The vehicle following behavior trajectory is shown in the figure below. Figure 2 As shown;

[0078] v FV (t+1)=v FV (t)+a FV (t)Δ(T)

[0079] Δ v (t+1)=v LV (t+1)-v FV (t+1)

[0080]

[0081] Among them, Δ d represents the distance between vehicles, Δ(T) represents the sampling time, Δ v represents the relative speed, a FV represents the acceleration of the following vehicle, v FV and v LV Respectively represent the current speed of the rear vehicle and the front vehicle;

[0082] S6. Setting the action space: The output of the action space is the acceleration value of the vehicle, and then the future speed, relative speed and relative distance are iteratively generated according to the motion point equation;

[0083] S7, the reward function R comprehensively considers three factors, which are used to evaluate the behavior of the vehicle and provide feedback based on these behaviors, so as to promote the model to develop in the desired direction. The main factors considered are safety, efficiency and comfort;

[0084] S8. Initialize the participant network parameters θ μ and the critic network parameters θ Q , initialize the participant target network θ μ′ and the critic target network θ Q′ , initialize the experience pool D1 and experience pool D2, where the participant network is responsible for iteratively updating the network parameters and is responsible for updating the network parameters according to the current state S t Select the current action a t , which is used to interact with the environment, and the target actor network is responsible for replaying the next state S sampled in the pool based on the experience t+1 ) Select the best next action a t+1 , the critic network is responsible for iteratively updating the value network parameters and calculating the current action value q(s t ,a t ), the critic target network is responsible for calculating the target action value q(s t+1 ,a t+1 )part;

[0085] S9, start training, observe the current state S t , select action a according to the policy network t, perform action a t Get Rewards t And observe the new state S t+1 ;

[0086] S10, according to the feedback of the reward value, {s t ,a t ,s t+1 ,r t The four-tuple experience data of} is divided into positive samples and negative samples, and stored in two experience replay pools respectively. The positive samples are samples that complete the car-following task and samples that neither complete the car-following task nor collide with each other. The negative samples are samples that collide with each other.

[0087] S11, sampling from the experience pool, first sampling from the positive sample experience pool D1, and then randomly sampling from the D1 experience pool and the D2 experience pool in proportion after a fixed delay, and then updating the network

[0088] S12. Use the critic target network to calculate the Q value of the target:

[0089] y t′ =r t′ +γQ(s t′ ,a t′ |θ Q )

[0090] Among them, y t′ represents the target Q value, which is used to update the critic network, r t′ At time step t i The immediate reward obtained, γ represents the discount factor, and γ is between 0 and 1, indicating the importance of future rewards, Q(s t′ ,a t′ |θ Q ) represents the critic execution network with parameters θ Q Based on the state t′ and action a t′ , the calculated Q value;

[0091] S13. Update the parameters θ of the critic network by minimizing the loss function Q , the loss function L takes the form of mean square error, which is the sum of the squares of the differences between the actual Q value and the target Q value, and then takes the average:

[0092]

[0093] Among them, L is the value of the loss function; N is the batch size, that is, the number of samples used in each parameter update, y i represents the target Q value of the i-th sample, which is obtained by step S12. i -ai θ Q ) represents the state s of the Q network for the i-th sample i and action a i The predicted Q value of

[0094] S14. Update the parameters θ of the participant network using policy gradient μ ;

[0095] S15. Update the parameters of the critic target network based on the updated parameters of the critic network, and update the parameters of the participant target network based on the updated parameters of the participant network; update in a soft update manner:

[0096]

[0097] Among them, ← represents the assignment, τ is a preset hyperparameter, a small value between 0 and 1, which is used to control the speed of updating the target network parameters, and θ μ is the participant network parameter, θ μ′ is the participant target network parameter, θ Q is the critic network parameter, θ Q′ is the critic target network parameter;

[0098] S16: When the strategy converges or reaches the preset number of training rounds, terminate the training process.

[0099] The rules for filtering out vehicle-following events include:

[0100] S1.1. To facilitate the analysis of vehicle-following events, the vehicle type in the data set is selected as a sedan, excluding motorcycles and large vehicles;

[0101] S1.2. The vehicle number identified by the radar remains unchanged, indicating that the front and rear vehicles in the car-following event are a unique group of vehicles;

[0102] S1.3. All following vehicles are in the same lane;

[0103] S1.4. The duration of following the vehicle must be >15s and continuous to ensure that there is enough data for analysis during the following time.

[0104] The THW and TTCi are expressed as follows:

[0105] THW(t)=d FV,LV (t) / v FV (t)

[0106] TTCi(t)=Δv rel (t) / d FV,LV (t)

[0107] THW is the time from the front center of vehicle FV to the front center of the preceding vehicle LV, d FV,LV (t) The distance between the following vehicle FV and the leading vehicle LV at time t, v FV (t) represents the speed of the following vehicle at time t;

[0108] TTC refers to the time required for two vehicles to collide during the following process, also known as the minimum collision time, which ensures the minimum distance between vehicles under dangerous braking. When the relative speed of the vehicles is 0, TTC will have no solution, so the deformation of TTC is TTCi, that is, the inverse of TTC, Δv rel (t) represents the relative speed between the following vehicle and the preceding vehicle at time t, d FV,LV (t) represents the distance between the following vehicle and the preceding vehicle at time t.

[0109] The Pearson correlation analysis calculation formula:

[0110]

[0111] Among them, ρ(x,y) is the Pearson correlation coefficient, which indicates the linear correlation between variables x and y, cov(x,y) indicates the covariance between variables x and y, σ(x) is the standard deviation of variable x, σ(y) is the standard deviation of variable y, E[(x-μ x )(y-μ y )] is the covariance calculation formula of variables x and y, E represents the expected value, μ x is the mean of variable x, μ y is the mean of the variable y, where the correlation coefficient ρ(x,y) ranges from [-1, 1]. Positive and negative values ​​indicate positive and negative correlations between the two variables, respectively. When the absolute value of the correlation coefficient between the two variables is greater than 0.8, the two variables are extremely correlated. When the absolute value of the correlation coefficient is less than 0.4, the correlation between the two variables is weak.

[0112] The reward function R is calculated as follows:

[0113] R total =w1R1+w2R2+w3R3+w4R4+w5R5

[0114] Among them, R1 is represented as the reward function of safety, which is measured by the inverse of the minimum collision time TTCi to ensure that the vehicle is traveling within a specific threshold. The safety reward function considers the minimum collision time of the vehicle and defines the reward value in the form of a logarithmic function to encourage the vehicle to maintain a safe distance during the following process to avoid collision risks. Its design ensures that when the vehicle is close to or below the safe time distance threshold, the agent will be punished, and when it is outside the safe distance, the reward value is zero, which maintains the smoothness and rationality of the reward function;

[0115] The safety reward function is:

[0116]

[0117] TTCi represents the inverse value of the current minimum collision time. * represents the safety threshold, that is, under this threshold, the current vehicle distance is considered safe. If the current headway is less than this threshold, it is considered that there is a collision risk. α represents the weight parameter;

[0118] R2 is represented as an efficiency reward function, which is measured by the headway time width (THW) in the car-following scenario. The efficiency reward function considers the headway time width (THW) and combines it with kernel density estimation to enable the agent to maintain an efficient and safe following distance during the car-following process. The function gives a higher reward when the headway time width is small, and gradually reduces the reward value when the headway time width is too large, ensuring that the agent can make the best decision in various situations.

[0119] The efficiency reward function is:

[0120]

[0121] THW(t) represents the current headway, that is, the time interval between the current vehicle and the preceding vehicle, Threshold * is the threshold. When the headway is greater than this threshold, the efficiency reward will be penalized. β is an adjustment parameter used to control the speed at which the reward value decays. f KDE is the THW distribution function value calculated based on kernel density estimation, K is the kernel function used for kernel density estimation, h is the bandwidth parameter used to control the smoothness of kernel density estimation, x is the current THW value, and x i are independently distributed sampling points, n is the amount of data;

[0122] R3 is expressed as a comfort reward function, which considers the jerk of the vehicle and combines the reference value J value, encourages the agent to maintain a stable driving behavior during the vehicle following process, thereby improving the comfort of passengers. The function gives negative rewards when the jerk is large, which drives the agent to optimize its control strategy to reduce acceleration changes, thereby improving driving comfort;

[0123] The comfort function is:

[0124]

[0125] Jerk represents the vehicle's jerk, that is, the rate of change of acceleration, defined as J value Expressed as a baseline value, it is defined as the sum of the squares of the acceleration changes over a period of time, a max Indicates the maximum acceleration of the vehicle within a period of time ΔT, a min Indicates the minimum acceleration of the vehicle within a period of time ΔT.

[0126] R4 and R5 are used for collision and early stop scenarios in vehicle following behavior decision making, and define penalty values ​​when collision and early stop occur respectively;

[0127] R4 encourages the agent to avoid collisions during vehicle following by penalizing the relative speed difference when a collision occurs. Since collision is an undesirable event, the value of this function is usually negative (i.e., penalty). Through the logarithmic function, the penalty value can be smoothly increased so that it is logarithmically related to the relative speed difference.

[0128] R5 is to encourage the agent to avoid early stops during the following process by penalizing the relative distance difference when stopping early. This function is usually used to encourage vehicles to keep a distance when following. In order to prevent vehicles from stopping early frequently, its value is usually negative (i.e., penalty) to avoid unreasonable early stops.

[0129] The formula is:

[0130]

[0131] δ and η represent weight parameters, which are used to adjust the size of the reward value. Indicates the relative speed between the front and rear vehicles. iscollision is a Boolean variable that is true when a collision occurs. Indicates the relative distance between the front and rear vehicles. isstop is a Boolean variable, which is true when the vehicle stops early.

[0132] In S14, the policy gradient is used to update the parameters θ of the participant network μ :

[0133]

[0134] in, Represents the gradient of the strategy, which represents the objective function J that we want to maximize with respect to the strategy parameter θ μ The gradient of the policy network parameter θ can be optimized by maximizing the expected reward J μ , It means averaging N samples in a batch, Q(s,aθ Q ) represents the critic network used to estimate the future cumulative reward obtained by performing action a in state s, with parameter θ Q , μ(s; θ μ ) indicates that the participant network outputs action a in a given state s, and the parameter is θ μ ,s i is the state of the i-th sample, μ(s i ) indicates that the policy network is in state s i The output action is: a=μ(s i ) represents the Q function Q(s,aθ Q ) The gradient of action a, evaluated at state s i Use the strategy μ to calculate the action μ(s i ), represents the policy function μ(s; θ μ ) for the policy parameter θ μ The gradient of , evaluated at state s i The action μ(s) output by the next policy network i ) for the parameter θ μ gradient.

[0135] The participant network and the critic network are both composed of an input layer, a hidden layer and an output layer, and the hidden layers of the participant network and the critic network are 30 neurons each. The input layer of the participant network has three inputs, namely, the longitudinal vehicle speed v FV , relative speed Δ vt and the relative position on the longitudinal trajectory Δ dt , the output layer is the acceleration a value, which will also serve as the input of the critic network, so the critic network has 4 inputs and the output layer is the Q value.

[0136] The vehicle following behavior decision method based on the improved DDPG algorithm of the present invention has common mechanical installation, connection or setting methods, which can be implemented as long as they can achieve their beneficial effects. Technical personnel in the industry only need to install and operate it according to the accompanying instruction manual, without the need for technical personnel in this field to make creative labor.

[0137] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A vehicle following behavior decision method based on improved DDPG algorithm, It is characterized in that The following steps are involved: S1. Based on the original natural data set, the data of car-following events that meet the conditions are screened out. The screened car-following events consider the following of the front car by default, and do not directly consider the following car. This is because when drivers are driving on the highway, they usually think that the driving behavior of the vehicle behind them is reasonable, that is, the following car will not rear-end the front car by default. Finally, 70% of the screened car-following events are used as the training data set, and 30% as the test data set; S2. Select appropriate driving behavior data features from the data set. Specifically, the selected feature data include FV_Vel, FV_Acc, Space_Headwauy, V_Rel, LV_Vel and LV_Acc, THW and TTCi, which represent the speed and acceleration of the following vehicle, the relative distance between the following vehicle and the leading vehicle, the relative speed, and the speed and acceleration of the leading vehicle, respectively. S3. Clarify the correlation between the driving behavior characteristic parameters in the data set, and perform Pearson correlation analysis on the data features selected from the data set; S4. Establish a vehicle following model based on the improved DDPG algorithm and set up a reinforcement learning environment for the vehicle following scenario, including the state space, action space and reward function. S5. Set the state space and select the longitudinal vehicle velocity v FV , relative speed and the position on the longitudinal trajectory The DDPG agent can control the vehicle's trajectory by controlling the speed of the following vehicle. The DDPG algorithm is used to obtain the acceleration value of the vehicle during the following process. The state update state is calculated by the classical kinematic model. S6. Setting the action space: The output of the action space is the acceleration value of the vehicle, and then the future speed, relative speed and relative distance are iteratively generated according to the motion point equation; S7, the reward function R comprehensively considers three factors, which are used to evaluate the behavior of the vehicle and provide feedback based on these behaviors, so as to promote the model to develop in the desired direction. The main factors considered are safety, efficiency and comfort; S8. Initialize the participant network parameters θ μ and the critic network parameters θ Q , initialize the participant target network θ μ′ and the critic target network θ Q′ , initialize the experience pool D 1 and experience pool D 2 , where the participant network is responsible for iteratively updating the network parameters and is responsible for updating the network parameters according to the current state S t Select the current action a t , which is used to interact with the environment, and the target actor network is responsible for replaying the next state S sampled in the pool based on the experience t+1 ) Select the best next action a t+1 , the critic network is responsible for iteratively updating the value network parameters and calculating the current action value q(s t , a t ), the critic target network is responsible for calculating the target action value q(s t+1 , a t+1 )part; S9, start training, observe the current state S t , select action a according to the policy network t , perform action a t Get Rewards t And observe the new state S t+1 ; S10, according to the feedback of the reward value, {s t , a t ,s t+1 , r t The four-tuple experience data of} is divided into positive samples and negative samples, and stored in two experience replay pools respectively. The positive samples are samples that complete the car-following task and samples that neither complete the car-following task nor collide with each other. The negative samples are samples that collide with each other. S11, sampling from the experience pool, first from the positive sample experience pool D 1 After a fixed delay, the sample is then proportionally sampled from D 1 Experience pool and D 2 Random sampling in the experience pool and then updating the network S12. Use the critic target network to calculate the Q value of the target: y t′ =r t′ +γQ(s t′ ,a t′ |θ Q ) Among them, y t′ represents the target Q value, which is used to update the critic network, r t′ At time step t i The immediate reward obtained, γ represents the discount factor, and γ is between 0 and 1, indicating the importance of future rewards, Q(s t′ , a t′ |θ Q ) represents the critic execution network with parameters θ Q Based on the state t′ and action a t′ , the calculated Q value; S13. Update the parameters θ of the critic network by minimizing the loss function Q , the loss function L takes the form of mean square error, which is the sum of the squares of the differences between the actual Q value and the target Q value, and then takes the average: Among them, L is the value of the loss function; N is the batch size, that is, the number of samples used in each parameter update, y i represents the target Q value of the i-th sample, which is obtained by step S12. i -a i θ Q ) represents the state s of the Q network for the i-th sample i and action a i The predicted Q value of S14. Update the parameters θ of the participant network using policy gradient μ ; S15. Update the parameters of the critic target network based on the updated parameters of the critic network, and update the parameters of the participant target network based on the updated parameters of the participant network; update in a soft update manner: Among them, ← represents assignment, τ is a preset hyperparameter, a small value between 0 and 1, which is used to control the speed of updating the target network parameters, θμ is the participant network parameter, θμ′ is the participant target network parameter, and θ Q is the critic network parameter, θ Q ′ is the critic target network parameter; S16: When the strategy converges or reaches the preset number of training rounds, terminate the training process.

2. A vehicle following behavior decision method based on an improved DDPG algorithm as claimed in claim 1, It is characterized in that The rules for filtering out vehicle-following events include: S1.

1. To facilitate the analysis of vehicle-following events, the vehicle type in the data set is selected as a sedan, excluding motorcycles and large vehicles; S1.

2. The vehicle number identified by the radar remains unchanged, indicating that the front and rear vehicles in the car-following event are a unique group of vehicles; S1.

3. All following vehicles are in the same lane; S1.

4. The duration of following the vehicle must be >15s and continuous to ensure that there is enough data for analysis during the following time.

3. The vehicle following behavior decision method based on the improved DDPG algorithm as claimed in claim 1, It is characterized in that The THW and TTCi are expressed as follows: THW(t)=d FV,LV (t) / v FV (t) TTCi(t)=Δv rel (t) / d FV,LV (t) THW is the time from the front center of vehicle FV to the front center of the preceding vehicle LV, d FV,LV (t) The distance between the following vehicle FV and the leading vehicle LV at time t, v FV (t) represents the speed of the following vehicle at time t; TTC refers to the time required for two vehicles to collide during the following process, also known as the minimum collision time, which ensures the minimum distance between vehicles under dangerous braking. When the relative speed of the vehicles is 0, TTC will have no solution, so the deformation of TTC is TTCi, that is, the inverse of TTC, Δv rel (t) represents the relative speed between the following vehicle and the preceding vehicle at time t, d FV,LV (t) represents the distance between the following vehicle and the preceding vehicle at time t.

4. The vehicle following behavior decision method based on the improved DDPG algorithm as claimed in claim 1, It is characterized in that The Pearson correlation analysis calculation formula: Among them, ρ(x, y) is the Pearson correlation coefficient, which indicates the linear correlation between variables x and y, cov(x, y) indicates the covariance between variables x and y, σ(x) is the standard deviation of variable x, σ(y) is the standard deviation of variable y, E[(x-μ x )(y-μ y )] is the covariance calculation formula of variables x and y, E represents the expected value, μ x is the mean of variable x, μ y is the mean of the variable y, where the correlation coefficient π(x, y) ranges from [-1, 1]. Positive and negative values ​​indicate positive and negative correlations between the two variables, respectively. When the absolute value of the correlation coefficient between the two variables is greater than 0.8, the two variables are extremely correlated. When the absolute value of the correlation coefficient is less than 0.4, the correlation between the two variables is weak.

5. The vehicle following behavior decision method based on the improved DDPG algorithm as claimed in claim 1, It is characterized in that The reward function R is calculated as follows: R total =w 1 R 1 +w 2 R 2 +w 3 R 3 +w 4 R 4 +w 5 R 5 Among them, R 1 It is expressed as a safety reward function, which is measured by the inverse of the minimum collision time TTCi to ensure that the vehicle travels within a specific threshold. The safety reward function considers the minimum collision time of the vehicle and defines the reward value in the form of a logarithmic function to encourage the vehicle to maintain a safe distance during the following process to avoid collision risks. Its design ensures that when the vehicle is close to or below the safe time distance threshold, the agent will be punished, and when it is outside the safe distance, the reward value is zero, which maintains the smoothness and rationality of the reward function; The safety reward function is: TTCi represents the inverse value of the current minimum collision time. * represents the safety threshold, that is, under this threshold, the current vehicle distance is considered safe. If the current headway is less than this threshold, it is considered that there is a collision risk. α represents the weight parameter; R 2 It is expressed as an efficiency reward function, which is measured by the headway time width (THW) in the car-following scenario. The efficiency reward function considers the headway time width (THW) and combines it with kernel density estimation to enable the agent to maintain an efficient and safe following distance during the car-following process. The function gives a higher reward when the headway time width is small, and gradually reduces the reward value when the headway time width is too large, ensuring that the agent can make the best decision in various situations. The efficiency reward function is: THW(t) represents the current headway, that is, the time interval between the current vehicle and the preceding vehicle, Threshold * is the threshold. When the headway is greater than this threshold, the efficiency reward will be penalized. β is an adjustment parameter used to control the speed at which the reward value decays. f KDE is the THW distribution function value calculated based on kernel density estimation, K is the kernel function used for kernel density estimation, h is the bandwidth parameter used to control the smoothness of kernel density estimation, x is the current THW value, and x i are independently distributed sampling points, n is the amount of data; R 3 Expressed as a comfort reward function, the comfort reward function considers the vehicle's jerk and combines the benchmark value J value , encourages the agent to maintain a stable driving behavior during the vehicle following process, thereby improving the comfort of passengers. The function gives negative rewards when the jerk is large, which drives the agent to optimize its control strategy to reduce acceleration changes, thereby improving driving comfort; The comfort function is: Jerk represents the vehicle's jerk, that is, the rate of change of acceleration, defined as J value Expressed as a baseline value, it is defined as the sum of the squares of the acceleration changes over a period of time, a max Indicates the maximum acceleration of the vehicle within a period of time ΔT, a min Indicates the minimum acceleration of the vehicle within a period of time ΔT.

6. A vehicle following behavior decision method based on an improved DDPG algorithm as claimed in claim 5, It is characterized in that The R 4 and R 5 It is used for collision and early stopping scenarios in vehicle following behavior decision making, and defines the penalty values ​​for collision and early stopping respectively; R 4 By penalizing the relative speed difference when a collision occurs, the agent is encouraged to avoid collisions during the vehicle following process. Since collision is an undesirable event, the value of this function is usually negative. Through the logarithmic function, the penalty value can be smoothly increased to make it logarithmically related to the relative speed difference. R 5 By penalizing the relative distance difference when stopping early, the agent is encouraged to avoid early stops during the following process. This function is usually used to encourage vehicles to keep a distance when following. In order to prevent vehicles from stopping early frequently, its value is usually negative to avoid unreasonable early stops. The formula is: δ and η represent weight parameters, which are used to adjust the size of the reward value. Indicates the relative speed between the front and rear vehicles. iscollision is a Boolean variable that is true when a collision occurs. Indicates the relative distance between the front and rear vehicles. isstop is a Boolean variable, which is true when the vehicle stops early.

7. The vehicle following behavior decision method based on the improved DDPG algorithm as claimed in claim 1, It is characterized in that In S14, the policy gradient is used to update the parameters θμ of the participant network: in, Represents the gradient of the strategy, which represents the gradient of the objective function J that we want to maximize with respect to the strategy parameter θμ. By maximizing the expected reward J, we can optimize the strategy network parameter θμ. It means averaging N samples in a batch, Q(s, aθ Q ) represents the critic network used to estimate the future cumulative reward obtained by performing action a in state s, with parameter θ Q , μ(s; θμ) represents the output action a of the participant network in a given state s, with parameters θμ, s i is the state of the i-th sample, μ(s i ) indicates that the policy network is in state s i The output action is: Denotes the Q function Q(s,aθ Q ) The gradient of action a, evaluated at state s i Use the strategy μ to calculate the action μ(s i ), represents the policy function μ(s; θ μ ) for the policy parameter θ μ The gradient of , evaluated at state s i The action μ(s) output by the next policy network i ) for the parameter θ μ gradient.

8. The vehicle following behavior decision method based on the improved DDPG algorithm as claimed in claim 1, It is characterized in that The participant network and the critic network are both composed of an input layer, a hidden layer and an output layer, and the hidden layers of the participant network and the critic network are 30 neurons each. The input layer of the participant network has three inputs, namely, the longitudinal vehicle speed v FV , relative speed and the relative position on the longitudinal trajectory The output layer is the acceleration a value, which will also serve as the input of the critic network. Therefore, the critic network has 4 inputs and the output layer is the Q value.

Citation Information

Cited By

  • Control method considering safety of intelligent suspension, medium and equipment

    CN120269980A