Long and large downhill road section driving speed control method based on depth Q network

Abstract

The invention discloses a long and large downhill road section driving speed control method based on a depth Q network, and the method comprises the following steps: building an environment model interacting with the depth Q network based on a long and large downhill road section scene, and defining a condition space, an action space and a reward function; constructing a deep Q network model, carrying out iterative interaction with the environment model to obtain a data set, and calculating to obtain a predicted Q value and a target Q value; constructing a loss function based on the predicted Q value, the target Q value and a reward function, and optimizing the loss function by using a gradient descent mode to realize training of a deep Q network model; and after multiple iterations, based on the trained deep Q network model, realizing control of the long and large downhill driving speed. The method is mainly based on data driving, comprehensively uses the driving safety, high efficiency and comfort indexes to represent the total income brought by speed control, is beneficial for avoiding bad driving behaviors in the driving process of the long and large downhill road section, and improves the traffic safety level.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent traffic information processing, and in particular is a method for controlling vehicle speed on a long downhill section based on a deep Q network. Background Art

[0002] Due to the constraints of geomorphology and topography, high-risk sections such as long downhill sections are inevitably seen in highway construction, especially in rugged plateaus and mountainous areas. On long downhill sections, drivers need to frequently step on the brakes to maintain the speed, which naturally leads to the continuous decline of vehicle braking performance or even brake failure. In response to this type of risk prevention and control, existing research focuses on formulating road and vehicle control measures. In terms of roads, it involves the setting of fixed-position service facilities such as cooling pools and traffic safety signs, as well as the matching design of slope and slope length; in terms of vehicles, it focuses on the implementation of macro-road management measures such as dynamic speed limit, passenger and cargo separation, and overload limit. Although the above measures have improved the driving safety level of long downhill sections to a certain extent, traffic accidents still occur from time to time. The fundamental reason is that the rule-driven, fixed-point, uniform, and non-difference intervention mode is difficult to effectively avoid the occurrence of bad driving behaviors (such as speeding and high-intensity braking) during driving.

[0003] In essence, the occurrence of bad driving behavior is jointly influenced by one's own characteristics (driving experience, physiology and psychology, etc.) and the external environment. Long downhill sections of highways usually adopt the design mode of "steep slope + gentle slope + steep slope". Drivers have different perceptions of the risks brought about by changes in slope and road environment, and their driving behaviors show heterogeneity accordingly. In addition, due to the psychological effect of drivers "pulling more and running faster" and their lack of sensitivity to slope, it is very easy to cause a large difference between the actual vehicle speed and the driver's perceived speed, which in turn induces bad driving behavior, reduces driving comfort, and accelerates the accumulation of driving risks.

[0004] Therefore, how to achieve safe, efficient and comfortable driving speed control has become an explorable path to further improve the level of active prevention and control of traffic safety on long downhill sections. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention proposes a method for controlling the driving speed on a long downhill section based on a deep Q network. This method regards the speed control of a long downhill section as a sequential decision-making process, and reinforcement learning provides a paradigm for solving such problems. The reward function is constructed by comprehensively considering driving safety, efficiency and comfort, and the driving speed is optimized by iterative interactive mapping of driving speed-risk state to maximize the cumulative reward.

[0006] A method for controlling vehicle speed on a long downhill section based on a deep Q network comprises the following steps:

[0007] Step 1: Based on the long downhill road scene, build an environment model that interacts with the deep Q network and define the state space, action space, and reward function;

[0008] Step 2: Build a deep Q network model and iteratively interact with the environment model to obtain a data set, and calculate the predicted Q value and target Q value at the same time;

[0009] Step 3: Based on the predicted Q value, target Q value and reward function, construct the loss function and optimize it using gradient descent to train the deep Q network model.

[0010] Step 4: After multiple iterations, the speed control of the vehicle on the long downhill slope is realized based on the trained deep Q network model.

[0011] Furthermore, the condition space in step 1 includes driving speed V, road slope I, downhill distance L, downhill time T, and initial wheel hub temperature Tmp. For the current time period t, the condition space is recorded as S(t)=(V, I, L, T, Tmp).

[0012] Furthermore, in step 1, the action space only includes acceleration a, and the action space of the current time period t is represented by a(t).

[0013] Furthermore, the reward function in step 1 comprehensively considers the driving safety index F safe , high efficiency index F eff and comfort index F cmf , then the reward function r is calculated as:

[0014]

[0015] Among them, ω1, ω2 and ω3 are the custom weights corresponding to the three indicators, and their value range is (0,1]; F safe Reflects the thermal degradation performance of the vehicle brake hub, and is quantified by the proximity of the vehicle hub temperature to the temperature threshold; F eff Characterized by the average driving speed, using the driving speed and the safety speed threshold V s quantify the closeness of cmf Quantitative characterization is based on the derivative of acceleration.

[0016] Furthermore, the construction process of the data set D′ in step 2 is:

[0017] Step 21: According to the initial state S(0) of the environment model, randomly select the action space a(0), and then obtain the next cycle state S(1). Through the iterative mapping of the state space-action space, the data sequence D={<S(0),a(0)> ,<S(1),a(1)> ,...,<S(t),a(t)> , (S(t+1), a(t+1)), ...};

[0018] Step 22: Follow<S(t),a(t),S(t+1)> The structure of the data sequence D is reconstructed to obtain D′={<S(0),a(0),S(1)> ,(S(1),a(1),S(2)),..., <S(t),a(t),S(t+1)),...}。

[0019] Furthermore, the specific implementation method of step 3 is:

[0020] Step 31: Construct the loss function Loss based on the predicted Q value, the target Q value and the calculated reward function r from the data sample, expressed as:

[0021] Loss=(r+γ*max a(t+1) Q(S(t+1), a(t+1); θ′)-Q(S(t), a(t); θ)) 2 (2)

[0022] Among them, max a(t+1) Q(·) represents the maximum Q(·) output obtained by selecting different a(t+1) in the next state S(t+1); γ∈(0,1] is the discount factor, which is used to measure the impact on the rewards obtained in the future time period;

[0023] Step 32: Use gradient descent to optimize the loss function Loss, and update the parameters θ of the Q network and the parameters θ′ in the target Q network. After multiple iterations, the speed control is optimized. For a single training process, F safe >1 or the downhill distance L reaches the total length of the long downhill section as the judgment condition for stopping training

[0024] The advantages of the present invention are:

[0025] (1) The most significant feature of the present invention is that it is data-driven and regards the speed control on long downhill sections as a sequential decision-making process, thereby optimizing the driving process with the help of deep Q-network modeling, thus breaking the traditional driving requirements constrained by rules.

[0026] (2) The main purpose of the present invention is to construct a reward function that comprehensively considers safety, efficiency and comfort evaluation indicators, so that the driving speed control is closer to that of an excellent driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0028] Figure 2 This is a schematic diagram of the longitudinal section of the long downhill section in the area;

[0029] Figure 3 This is a curve chart showing the average cumulative reward changes during the algorithm training process;

[0030] Figure 4 It is the change diagram of the safety index during driving process;

[0031] Figure 5 It is the change diagram of the high efficiency index of the driving process;

[0032] Figure 6 This is a diagram showing changes in comfort indicators during driving. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention is a data-driven long downhill driving speed control optimization method. Firstly, a deep Q network environment model is established based on a real long downhill road section scene, and the state and action in the environment model are determined; secondly, safety, efficiency and comfort evaluation indicators are extracted from the state and action variables, and a reward function is constructed in a weighted form; then, the state-action is iterated to maximize the cumulative reward training model; finally, the driving speed output by the model and the actual driving speed are compared in combination with the evaluation indicators to verify the model effect.

[0035] refer to Figure 1 The present invention provides a method for controlling vehicle speed on a long downhill section based on a deep Q network, comprising the following steps:

[0036] 1) Construction of reinforcement learning environment model. Based on the long downhill road scene, an environment model that interacts with the deep Q network is constructed. The environment model is used to describe the dynamic characteristics of the long downhill scene, interact with the reinforcement learning agent, and generate data. It mainly includes three modules: state space, action space, and reward function, which are defined as follows.

[0037] (1) Definition of state space. The driving state of a long downhill slope is affected by many factors. The state space includes the driving speed V, the road slope I, the downhill distance L, the downhill time T, and the initial wheel hub temperature Tmp. For the current time period t, the state space is recorded as S(t) = (V, I, L, T, Tmp).

[0038] (2) Definition of action space. Acceleration a is an indicator that directly reflects the change in speed, so the action space of the current time period t is represented by a(t).

[0039] (3) Reward function definition. First, define the driving safety index F safe , high efficiency index F eff and comfort index F cmf ,The detailed description of each performance index is as follows.

[0040] F safe The thermal decay performance of the vehicle brake hub is characterized and quantified by the proximity of the vehicle hub temperature to the temperature threshold (260°C), as shown in formula (1):

[0041] F safe =1-(0.005*I*L+0.992*L / V 5.244 +Tmp) / 260 (1)

[0042] Among them, F safe ∈[0, 1), the larger the value, the smaller the driving risk.

[0043] F eff Characterized by driving speed, using driving speed and safety speed threshold (V s ) is quantified, as shown in formula (2):

[0044] F eff =V / V s (2)

[0045] Among them, F eff ∈[0, 1], the larger the value, the faster the vehicle can leave the long downhill section.

[0046] F cmf The quantization is performed based on the derivative of acceleration, as shown in formula (3):

[0047]

[0048] Among them, Norm(·) is the normalization function, making F cmf ∈[0, 1], the larger the value, the higher the driving comfort.

[0049] Based on the above three indicators, use F safe, F eff and F cmf The weighted sum of and F is used as the reward function, as shown in Equation (4):

[0050] r = ω1F safe + ω2F eff + ω3F cmf (4)

[0051] where ω1, ω2, and ω3 are the custom weights corresponding to the three metrics, and the value range is (0, 1].

[0052] 2) Construction of the reinforcement learning network. Since the action space a is a continuous variable, a deep Q-network is used to build a deep Q-network model, which iteratively interacts with the environment model to obtain a dataset, and at the same time calculates and obtains the predicted Q value and the target Q value. This model includes two sub-networks: the Q-network and the target Q-network. The network structures of the two are the same and are composed of a double-layer convolutional structure.

[0053] (1) Training of the Q-network. Use the current state S(t) and action a(t) to train the Q-network to obtain the "predicted Q value" Q(S(t), a(t); θ), which represents the value of choosing the action a(t), where θ is the hyperparameter of the Q-network.

[0054] (2) Training of the target Q-network. Given the state data S(t + 1) of the next cycle, use the target Q-network to predict the maximum Q value from all actions executed in the state S(t + 1), that is, the "target Q value" max a(t+1) Q(S(t + 1), a(t + 1); θ'), where θ' is the hyperparameter of the target Q-network.

[0055] The construction process of the dataset D' is as follows:

[0056] Step 21: According to the initial state S(0) of the environment model, randomly select the action space a(0), and then obtain the state S(1) of the next cycle. Through the iterative mapping of the state space - action space, obtain the data sequence D = {<S(0), a(0)>, <S(1), a(1)),..., <S(t), a(t)>, <S(t + 1), a(t + 1)),...};

[0057] Step 22: Reconstruct the data sequence D according to the structure of <S(t), a(t), S(t + 1)), and obtain D' = {<S(0), a(0), S(1)>, (S(1), a(1), S(2)),..., <S(t), a(t), S(t + 1)),...}.

[0058] 3) Driving speed control optimization. Based on the training of the deep Q network, the model can eventually output the best action according to the current state of the vehicle. The specific process is divided into the following two steps.

[0059] (1) Construct the loss function Loss based on the predicted Q value, the target Q value and the reward r calculated from the data sample, as shown in formula (5):

[0060] Loss=(r+γ*max a(t+1) Q(S(t+1), a(t+1); θ′)-Q(S(t), a(t); θ)) 2 (5)

[0061] Among them, γ∈(0, 1] is a discount factor, which is used to measure the impact on the rewards obtained in future time periods.

[0062] (2) Use gradient descent to optimize the loss function Loss and update the parameters θ of the Q network and the parameters θ′ in the target Q network. After multiple iterations, the speed control is optimized and the training of the deep Q network model is realized. For a single training process, F safe >1 (i.e., vehicle brake failure) or the downhill distance L reaches the total length of the long downhill section as the judgment conditions for stopping this training.

[0063] 4): After multiple iterations, the speed control of the vehicle on the long downhill slope is realized based on the trained deep Q network model.

[0064] The biggest feature of the present invention is that it is data-driven and regards the speed control of long downhill sections as a sequential decision-making process, thereby optimizing the driving process with the help of deep Q network modeling, thereby breaking the traditional driving requirements constrained by rules. The main purpose of the present invention is to construct a reward function that comprehensively considers safety, efficiency and comfort evaluation indicators, so that the driving speed control is closer to that of an excellent driver.

[0065] Example

[0066] A method for controlling vehicle speed on a long downhill section based on a deep Q network is as follows:

[0067] 1) Instantiate the environment model. Figure 2 As shown in the figure, a long downhill section in a certain area is 11400m long, and the slope range of the section is [0.6%~4.0%]. Based on this, the instantiated environment model is as follows:

[0068] (1) Determination of the state space. Since driving speeds that are too high or too low pose safety risks, the value range of V is [40 km / h, 90 km / h], that is, the speed threshold V s= 90 Km / h; slope I ∈ [0.6%, 4.0%]; downhill distance L ∈ [0, 11400]; driving time T with a step size of 0.1 second each time, that is, the deep Q network and the environment model interact every 0.1 second; the initial wheel temperature Tmp of the vehicle before entering the slope is set to 130 °C.

[0069] (2) Determination of the action space. The action space involves the acceleration and deceleration of the vehicle. Given the general performance of the vehicle, the value range of the acceleration a is set to [-3m / s 2 ~3m / s 2 .

[0070] (3) Determination of the reward function. Let ω1, ω2, and ω3 all be 1, indicating that the importance of driving safety, efficiency, and comfort is the same, that is, r = F safe +F eff +F cmf .

[0071] 2) Construct a deep Q network and iteratively interact with the environment model.

[0072] (1) According to the initial state S(0) of the environment model, randomly select the action a(0), and then obtain the next-cycle state S(1). Through the iterative mapping of state-action, obtain the data sequence D = {<S(0), a(0)>, <S(1), a(1)),..., <S(t), a(t)>, <S(t + 1), a(t + 1)),...}.

[0073] (2) Since the Q network takes <S(t), a(t)> as the input, while the target Q network takes S(t + 1) as the input, reconstruct the data sequence D to get D′ = {<S(0), a(0), S(1)>, <S(1), a(1), S(2)),..., <S(t), a(t), S(t + 1)),...}, and then obtain the "predicted Q value" and the "target Q value".

[0074] 3) Model training and verification of test effects.

[0075] (1) Calculate the loss function according to formula (5) and optimize it using gradient descent. Set the number of training rounds to 200, train 10 times in a single round, and draw the curve of the average cumulative reward obtained in each training round, as Figure 3 shown. It can be seen that after 45 training rounds, the average cumulative reward converges to about 400, and the final average cumulative reward is 408.11, indicating that the model has convergence.

[0076] (2) For the trained deep Q network (usually called the "agent"), conduct a long downhill driving speed control experiment with V(0) = 80 Km / h, and separately from the safety index Fsafe , high efficiency index F eff and comfort index F cmf The results of the human driving experiment are compared and analyzed at the same level. Figure 4 , Figure 5 and Figure 6 As shown. Figure 4 It can be seen that as the downhill distance increases, the F safe Both continue to decrease, but the F safe Slightly higher. Figure 5 It can be seen that the F eff The overall trend of change is similar, but each time the speed needs to be reduced, the agent will reduce the speed in advance, and in the case of speed operation, the speed of the agent driving will be slightly higher than that of a natural person. Figure 6 It can be seen that in the case of natural person driving decision-making, sudden deceleration or acceleration occurs more frequently, and the comfort is lower than that in the intelligent agent driving decision-making process.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling vehicle speed on a long downhill section based on a deep Q network, characterized in that: The steps include: Step 1: Based on the long downhill road scene, build an environment model that interacts with the deep Q network and define the state space, action space, and reward function; Step 2: Build a deep Q network model and iteratively interact with the environment model to obtain a data set, and calculate the predicted Q value and target Q value at the same time; Step 3: Based on the predicted Q value, target Q value and reward function, construct the loss function and optimize it using gradient descent to train the deep Q network model. Step 4: After multiple iterations, the speed control of the vehicle on the long downhill slope is realized based on the trained deep Q network model.

2. The method for controlling vehicle speed on a long downhill section based on a deep Q network according to claim 1, characterized in that: The condition space in step 1 includes driving speed V, road slope I, downhill distance L, downhill time T, and initial wheel hub temperature Tmp. For the current time period t, the condition space is recorded as S(t)=(V, I, L, T, Tmp).

3. The method for controlling vehicle speed on a long downhill section based on a deep Q network according to claim 2, characterized in that: In step 1, the action space only includes acceleration a, and the action space of the current time period t is represented by a(t).

4. The method for controlling vehicle speed on a long downhill section based on a deep Q network according to claim 3 is characterized in that: The reward function in step 1 comprehensively considers the driving safety index F safe , high efficiency index F eff and comfort index F cmf , then the reward function r is calculated as: Among them, ω1, ω2 and ω3 are the custom weights corresponding to the three indicators, and their value range is (0,1]; F safe Reflects the thermal degradation performance of the vehicle brake hub, and is quantified by the proximity of the vehicle hub temperature to the temperature threshold; F eff Characterized by the average driving speed, using the driving speed and the safety speed threshold V s quantify the closeness of cmf Quantitative characterization is based on the derivative of acceleration.

5. The method for controlling vehicle speed on a long downhill section based on a deep Q network according to claim 4, characterized in that: The construction process of the data set D' in step 2 is: Step 21: According to the initial state S(0) of the environment model, randomly select the action space a(0), and then obtain the next cycle state S(1). Through the iterative mapping of the state space-action space, the data sequence D={<S(0),a(0)> ,<S(1),a(1)> ,…,<S(t),a(t)> ,<S(t+1),a(t+1)> ,…}; Step 22: Follow<S(t),a(t),S(t+1)> The structure of the data sequence D is reconstructed to obtain D'={<S(0),a(0),S(1)> ,<S(1),a(1),S(2)> ,…,<S(t),a(t),S(t+1)> ,…}.

6. The method for controlling vehicle speed on a long downhill section based on a deep Q network according to claim 5, characterized in that: The specific implementation method of step 3 is: Step 31: Construct the loss function Loss based on the predicted Q value, the target Q value and the calculated reward function r from the data sample, expressed as: Loss=(r+γ*max a(t+1) Q(S(t+1),a(t+1);θ')-Q(S(t),a(t);θ)) 2 (2) among them, max a(t+1) Q(·) represents the maximum Q(·) output obtained by selecting different a(t+1) in the next state S(t+1); γ∈(0,1] is the discount factor, which is used to measure the impact on the rewards obtained in the future time period; Step 32: Use gradient descent to optimize the loss function Loss, and update the parameters θ of the Q network and the parameters θ' in the target Q network. After multiple iterations, the speed control is optimized. For a single training process, F safe >1 or the downhill distance L reaches the total length of the long downhill section as the judgment condition for stopping training.

Images