Decision-making method for autonomous driving vehicles based on implicit time-scale reinforcement learning

Abstract

The present invention discloses a decision-making method for autonomous vehicles based on implicit time-scale reinforcement learning. The method comprises an implicit time-scale decision model, an improved reinforcement learning exploration method, a rule-based and learning-based mobility method, and an intelligent environment vehicle model. The implicit time-scale decision model implicitly incorporates time-scale information into a state-value table, thereby improving the continuity of decision-making behavior. The improved reinforcement learning exploration method improves the exploration efficiency of reinforcement learning by using a preset strategy-based exploration method, a backpropagation-based value table update method, and a momentum-based local optimum breakthrough. The rule-based and learning-based mobility method and the intelligent environment vehicle model can establish a simulation environment to verify the effectiveness and continuity of the decision-making of the present invention. The present invention implicitly incorporates time information into the state-value table, improving the continuity and safety of vehicle decision-making.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving vehicle decision-making and relates to an autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning. Background Art

[0002] With the rapid development of computer and communication technologies, autonomous vehicles have garnered widespread attention, and pilot deployments have already begun in some Chinese cities. However, according to current research, autonomous vehicles still face incompatibilities between driving safety, effectiveness, and behavioral rationality. The vehicle decision-making module, as a lower-level application of the vehicle perception module and a higher-level guidance for the vehicle trajectory planning and control modules, plays a crucial role in the safety and efficiency of autonomous vehicles.

[0003] With the continuous advancement of computing power in autonomous vehicles, learning-based decision-making methods have become one of the mainstream approaches for autonomous vehicle decision-making. Compared to the neural network value tables of deep reinforcement learning algorithms, conventional reinforcement learning algorithms have greater research value due to their less black-box nature, ease of integration with rule-based methods, and strong flexibility.

[0004] Faced with complex traffic environments, intelligent vehicles must not only avoid obstacles to reach their destination but also interact with other traffic participants to maintain safety, efficiency, and other requirements. The key issue is how to efficiently and correctly interact with traffic.

[0005] In summary, learning-based vehicle behavior decision-making research methods are currently a hot research topic, but the application of these methods mostly requires rapid updates, the decision-making continuity is insufficient, and methods based on rapid updates may not be able to support real-car applications of large models. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide an autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning, which hides time information in the state value table without significantly increasing the computational difficulty, thereby improving the decision-making continuity and safety of the vehicle.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A decision-making method for autonomous vehicles based on implicit time-scale reinforcement learning, comprising:

[0009] S1, building a decision model based on implicit time scale;

[0010] S2, based on the decision-making model established in S1, uses reinforcement learning to make vehicle decisions and improves the exploration efficiency of reinforcement learning;

[0011] S3, after the exploration phase, in order to improve the safety of the designed model during movement, a rule-based and learning-based movement method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision.

[0012] The present invention also includes the following technical features:

[0013] Specifically, in S1, a state value table is constructed based on implicit time scale information to establish a vehicle decision model; specifically, the following steps are included:

[0014] S1.1, building a decision model: Discretize the continuous highway into discrete units of equal length. The width of each discrete unit is equal to the lane width. Each discrete unit represents a state S. For each state S, there are four actions as shown in the following formula:

[0015]

[0016] In the above formula, i = 1, 2, 3 or 4, moving one unit up or down corresponds to lane changing, and moving one or two units to the right corresponds to moving at different speeds;

[0017] S1.2, import implicit time scale: Based on the above decision model, the time scale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state. The implicit time scale is assigned by moving the environment vehicle.

[0018] Specifically, in S2, the exploration efficiency of reinforcement learning is improved by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, and a reinforcement learning exploration strategy is designed.

[0019] Specifically, the exploration method based on the preset strategy includes:

[0020] In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random move strategy is used to explore the environment as much as possible. After 100 explorations, a greedy strategy is used for action selection, as shown in the following formula:

[0021]

[0022] Where a(S) represents the action selection in state S; Q(S,:) represents the value table of each action in state S; n represents the number of exploration steps;

[0023] During the random movement phase, to increase the number of explorations at the far end of the large map, a preset exploration strategy is used as shown in the following formula to adjust the probabilities between different actions to achieve pseudo-directional exploration:

[0024]

[0025] p(S,a i ) means that in state S, action a is selected i The probability that i = 1, 2, 3 or 4.

[0026] Specifically, the value table updating method based on back propagation includes:

[0027] In the exploration phase, when the autonomous vehicle agent reaches the destination, it performs reverse value propagation on the path it has taken. After multiple backpropagations, the state and corresponding action of the state that is always on the path to the destination will be higher than other states, thus better finding the optimal feasible path;

[0028] The back propagation update method is as follows:

[0029]

[0030] Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

[0031] Specifically, the breakthrough of the local optimum based on the momentum method includes: for the designed decision model, according to the movement characteristics of the vehicle, a momentum method based on vehicle speed maintenance is designed, as shown in the following formula:

[0032] S n =S n-2

[0033] a(S n )=a(S n-2 )

[0034] a(S n-1 )=a(S n-2 )

[0035] Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to judge whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, a breakthrough will be made based on the vehicle speed direction.

[0036] Specifically, the design of the reinforcement learning exploration strategy includes: designing an exploration method based on the fusion of rules and learning, adopting an exploration mechanism that integrates rules and learning, setting that when the vehicle agent explores a high negative reward, the probability of selecting the action will be adjusted to 0, and a second exploration will be performed in the original state. For exploratory reinforcement learning, this method can significantly improve the exploration efficiency of each exploration cycle, as shown in the following formula:

[0037]

[0038] in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

[0039] Specifically, the S3 includes:

[0040] S3.1. After the exploration phase, the autonomous vehicle follows the optimal feasible path found by the decision model. The following rules are designed to improve the safety of the movement:

[0041] Rule 1 is to use the rules to reselect the action; Rule 2 is that when any action in the state will result in a collision or miss the target point, the ABS mechanism will be triggered, causing the vehicle to stop in the current state, and the ABS triggers the failure of the corresponding decision; as shown in the following formula:

[0042] Rule 1:

[0043] Rule 2:

[0044] Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of; set2 represents the state collection that violates security;

[0045] S3.2, Probability of Adoption Corresponding to the different driving styles of vehicles in the intelligent environment, different behavioral rules are established based on different driving styles:

[0046] Intelligent environment vehicles have different longitudinal customary following distances according to different driving styles, as shown in the following formula:

[0047]

[0048] When the actual following distance is not within the customary following distance range, the vehicle brakes and accelerates accordingly to meet the needs of safety and traffic efficiency, as shown in the following formula:

[0049]

[0050] Among them, d f represents the habitual following distance, P is the driving style, round is the rounding function, d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to the vehicle speed respectively;

[0051] S3.3. Design the lateral lane-changing behavior of intelligent environment vehicles, which is divided into two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by driving goals; the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are established. The lane-changing behavior will only occur when the overtaking lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met.

[0052] Specifically, the overtaking lane change intention is established based on the MOBIL principle and actual traffic regulations. The overtaking intention is generated by a nonlinear function based on the difference between the vehicle conditions in the adjacent lane and the current lane. The vehicle conditions in the adjacent lane and the current lane are represented by the difference between the current following distance and the following distance after the lane change, as shown in the following formula:

[0053] x d =(d rf / d rf -d r )>0

[0054]

[0055] Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake and change lanes to the left, d lf Iten is the following distance after changing lanes to the left lane. c Intention to change lanes for overtaking;

[0056] The lane change intention caused by the driving goal is expressed as follows: the closer the vehicle is to the goal point, the stronger the lane change intention. The lane change intention setting makes the behavior of the surrounding vehicles more consistent with the real vehicles when merging into or exiting the highway, as shown in the following formula:

[0057] x m =mb ji -i

[0058]

[0059] Among them, x mis the distance between the driving target endpoint and the current position of the environment vehicle, mb ji is the driving goal of the environment vehicle located in the i unit of lane j, Iten m For driving goal intention.

[0060] Specifically, the total lane change intention (TI) is defined as the sum of the two lane change intentions. A vehicle will only change lanes when the total lane change intention exceeds a threshold. This lane change intention ensures that intelligent vehicles in an environment do not repeatedly change lanes, potentially endangering traffic, while also acknowledging the differences between vehicles with different driving styles. The total lane change intention is expressed as follows:

[0061] Iten=Iten c +Iten m

[0062]

[0063]

[0064] Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined by driving style;

[0065] When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation:

[0066]

[0067] Among them, d rr ,d rf They represent the distance between the front and rear vehicles before changing lanes to the right lane. Lane changing will only occur when TI and TC are both equal to 1.

[0068] Compared with the prior art, the present invention has the following technical effects:

[0069] (I) The present invention improves the exploration efficiency of reinforcement learning by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, effectively improving the exploration efficiency of the exploration stage.

[0070] (II) The present invention imports time scale information into the state value table in an implicit way, thereby improving the continuity of decision-making behavior and providing effective decision-making behavior for subsequent planning and control of the vehicle.

[0071] (III) The present invention designs a corresponding rule-based and learning-based mobility method and intelligent environment vehicle model to build a simulation environment to verify the effectiveness and continuity of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1Flowchart of the invention.

[0073] Figure 2 Decision environment diagram.

[0074] Figure 3 Set up a graph for implicit timescale rewards. DETAILED DESCRIPTION

[0075] The present invention provides an autonomous vehicle decision-making method based on implicit timescale reinforcement learning, comprising an implicit timescale decision-making model, an improved reinforcement learning exploration method, a rule-based and learning-based mobility method, and an intelligent environment vehicle model. The implicit timescale decision-making model implicitly imports timescale information into a state-value table, thereby improving the continuity of decision-making behavior. The improved reinforcement learning exploration method improves the efficiency of reinforcement learning exploration by using a preset strategy-based exploration method, a backpropagation-based value table update method, and a momentum-based local optimum breakthrough. The rule-based and learning-based mobility method and the intelligent environment vehicle model can establish a simulation environment to verify the effectiveness and continuity of the decision-making of the present invention.

[0076] Specific embodiments of the present invention are given below. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent modifications made on the basis of the technical solution of this application fall within the protection scope of the present invention.

[0077] Example:

[0078] This embodiment provides a decision-making method for an autonomous driving vehicle based on implicit time-scale reinforcement learning, such as Figure 1 As shown, including:

[0079] S1, establishing a decision model based on implicit time scale; specifically, constructing a state value table based on implicit time scale information and establishing a vehicle decision model;

[0080] S2: Based on the decision-making model established in S1, use reinforcement learning to make vehicle decisions and improve the exploration efficiency of reinforcement learning. Specifically, use a preset strategy-based exploration method, a back-propagation-based value table update method, and a momentum-based local optimal point breakthrough to improve the exploration efficiency of reinforcement learning. Then, design the reinforcement learning exploration strategy.

[0081] S3, after the exploration phase, in order to improve the safety of the designed model during movement, a rule-based and learning-based movement method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision.

[0082] More specifically, it includes the following aspects and steps:

[0083] First, establish a decision model based on implicit time scale.

[0084] S1.1, build a decision model: discretize the continuous highway into discrete units of 5m in length, and the width of each discrete unit is equal to the lane width, such as Figure 2 Each discrete unit represents a state S. For each state S, there are four actions as shown below:

[0085]

[0086] In this formula, i = 1, 2, 3, or 4; moving one unit up or down corresponds to changing lanes, and moving one or two units to the right corresponds to moving at different speeds.

[0087] S1.2, Implicit time scale: Based on the above decision model, the time scale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state, such as Figure 3 As shown. The implicit time scale is assigned by moving the vehicle in the environment. When the autonomous vehicle is in the exploration stage of the environment, each exploration of the vehicle is accompanied by the movement of the surrounding environment, which means that the vehicle will receive different reward feedback when exploring the same location at different exploration moments. However, due to the characteristics of highway driving, that is, the vehicle will only move forward at different speeds and will not move backward, so after the vehicle has explored a certain number of times, a feasible path with implicit time scale information will be formed. This path may have a collision when observed from the static environment, but when the vehicle drives to the static collision area, the vehicle that actually caused the collision has already left.

[0088] Secondly, we will improve the exploration efficiency of reinforcement learning and design exploration strategies.

[0089] The update formula of the traditional Bellman optimal equation in policy iteration is shown below. For the environment constructed by the present invention, each state S is represented by the corresponding coordinates (x, y).

[0090] Q n (S,a)=(1-α)Q n-1 (S,a)+α(r+γ*max(Q(S a ,:)))

[0091] Where Q n (S,a) represents the value of action a in state S at exploration time n, α is the learning rate, γ is the decay rate, which are 0.1 and 0.9 respectively for this embodiment, r is the reward value in this state, S a is the new state reached after action a in state S, max(Q(S a ,:)) is state Sa The maximum value of all action values.

[0092] S2.1, using preset strategies to improve the exploration method: In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random move strategy is used to explore the environment as much as possible. After 100 explorations, a greedy strategy is used for action selection, as shown in the following formula:

[0093]

[0094] Among them, a(S) represents the action selection in state S; Q(S,:) represents the value table of each action in state S; n represents the exploration step.

[0095] During the random movement phase, to increase the number of explorations at the far end of the large map, a preset exploration strategy is used as shown in the following formula to adjust the probabilities between different actions to achieve pseudo-directional exploration:

[0096]

[0097] Among them, p(S,a i ) means that in state S, action a is selected i The probability of; more specifically, as follows:

[0098]

[0099] S2.2, Design of a value table update method based on backpropagation: In large maps, due to the existence of learning rate and decay rate, it is difficult for autonomous vehicles to find the optimal path by comparing the difference in action value when they are far away from the target. To further improve the exploration efficiency of the reinforcement learning algorithm, this paper proposes a reinforcement learning update method based on backpropagation. This method performs backpropagation of the value of the path passed by the autonomous vehicle agent when it reaches the destination during the exploration phase. After multiple backpropagations, the state and corresponding action of the state that are always on the path to the target point will be higher than those of other states, thus better finding the optimal feasible path;

[0100] The back propagation update method is as follows:

[0101]

[0102] Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

[0103] S2.3, Designing a Breakthrough for Local Optimal Points Based on Momentum Methods: For the decision-making model designed by this invention, when there are many vehicles in the traffic environment, it is easy to fall into a local optimal point by moving along the maximum state action value curve, causing the vehicle to move back and forth between two states or fall into a closed movement loop. To improve the ability of the reinforcement learning algorithm to escape the local optimal point, this invention designs a momentum method based on vehicle speed maintenance based on the characteristics of vehicle movement, as shown in the following formula:

[0104] S n =S n-2

[0105] a(S n )=a(S n-2 )

[0106] a(S n-1 )=a(S n-2 )

[0107] Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to judge whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, a breakthrough will be made based on the vehicle speed direction.

[0108] S2.4, design an exploration method based on the fusion of rules and learning: Then use the improved reinforcement learning algorithm mentioned above to explore, but when there are many vehicles in the traffic environment, the negative rewards of the environmental vehicles will gradually generate negative value partition areas as the number of explorations increases. And for the state value table of the implicit time scale, negative rewards at different times can easily constitute regional negative rewards during the exploration process. Therefore, the present invention borrows the exploration mechanism of the fusion of rules and learning, and sets that when the vehicle agent explores a high negative reward, the selection probability of the action will be adjusted to 0, and a second exploration will be performed in the original state. For exploratory reinforcement learning, this method can significantly improve the exploration efficiency of each exploration cycle, as shown in the following formula:

[0109]

[0110] in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

[0111] Thirdly, a rule-based and learning-based mobility method and a corresponding intelligent environment vehicle are established to build a simulation environment to verify the effectiveness and continuity of the present invention.

[0112] S3.1. After the exploration phase ends, the vehicle begins to move forward along the optimal feasible path found by the decision model. The decision frequency is set to 1 Hz. However, in a highway environment, surrounding vehicles and autonomous vehicles can travel an average of more than 15 meters (determined by the minimum speed limit of the highway) within 1 second. Therefore, the present invention designs corresponding rules to improve the safety of movement.

[0113] There are two types of rules: one is to use rules to reselect actions, and the other is that when any action in the state will result in a collision or miss the target point, the ABS mechanism will be triggered, causing the vehicle to stop in the current state. The triggering of ABS also means the failure of decision-making. As shown in the following formula:

[0114] Rule 1:

[0115] Rule 2:

[0116] Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of set2 represents a collection of states that violate security.

[0117] S3.2, in addition to the autonomous driving vehicle of the present invention, the accurate description of the surrounding vehicles is also important for the effectiveness of the simulation. For each individual intelligent environment vehicle, it has different driving styles, different driving goals and random starting positions. Corresponding to different driving styles, different behavioral codes are established based on different driving styles.

[0118] Based on relevant research, we hypothesize that driving style has a uniform impact on different behaviors. The smaller P, the more conservative the driving style, the more stringent the conditions for overtaking and lane changing. When these conditions are met, the probability of execution decreases, and the following distance increases.

[0119] In the constructed simulation environment, the intelligent environment vehicles have different longitudinal habitual following distances according to different driving styles, as shown in the following formula:

[0120]

[0121] When the actual following distance is not within the customary following distance range, the vehicle will brake and accelerate accordingly to meet the needs of safety and traffic efficiency, as shown in the following formula:

[0122]

[0123] Among them, d frepresents the habitual following distance, P is the driving style, round is the rounding function, d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to vehicle speeds of 54, 72, and 90 km / h respectively.

[0124] S3.3. Next, the lateral lane-changing behavior of the intelligent environment vehicle is designed. There are two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by the driving goal. To unify the impact of different situations, the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are established. The lane-changing behavior will only occur when the lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met.

[0125] The overtaking lane change intention is based on the MOBIL principle and actual traffic regulations. The overtaking intention is generated by a nonlinear function based on the difference between the traffic conditions in the adjacent lane and the current lane. The traffic conditions in the adjacent lane and the current lane are expressed as the difference between the current following distance and the following distance after the lane change, as shown in the following formula:

[0126] x d =(d rf / d lf -d r )>0

[0127]

[0128] Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake and change lanes to the left, d lf Iten is the following distance after changing lanes to the left lane. c Changing lanes with the intention of overtaking.

[0129] The lane change intention caused by the driving goal can be expressed as follows: the closer the vehicle is to the goal point, the stronger the lane change intention. The setting of the lane change intention makes the behavior of the surrounding vehicle more consistent with the real vehicle when merging or exiting the highway, as shown in the following formula:

[0130] x m =mb ji -i

[0131]

[0132] Among them, x m is the distance between the driving target endpoint and the current position of the environment vehicle, mb jiis the driving goal of the environment vehicle located in the i unit of lane j, Iten m For driving goal intention.

[0133] The total lane change intention (TI) is defined as the sum of the two lane change intentions. The vehicle will only change lanes when the total lane change intention is greater than a threshold. The establishment of lane change intention ensures that intelligent environment vehicles will not repeatedly change lanes and endanger traffic, while also expressing the differences between vehicles with different driving styles.

[0134] Iten=Iten c +Iten m

[0135]

[0136]

[0137] Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined by driving style.

[0138] When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation:

[0139]

[0140] Among them, d rr ,d rf They represent the distance between the front and rear vehicles before changing lanes to the right lane. Lane changing will only occur when TI and TC are both equal to 1.

[0141] It should be noted that the adjacent lane and lane change intention calculations in the formula are calculated separately based on the differences between left and right lane changes.

[0142] To sum up, this is the process of the decision-making method for autonomous driving vehicles on highways based on implicit time-scale reinforcement learning proposed by the present invention. Through the design of the present invention, the exploration efficiency of the algorithm can be accelerated and the effectiveness and continuity of vehicle decision-making can be improved.

[0143] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the scope of protection of the present invention.

[0144] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. To avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0145] In addition, the various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A decision-making method for autonomous driving vehicles based on implicit time-scale reinforcement learning, characterized in that: include: S1, building a decision model based on implicit time scale; S2, based on the decision-making model established in S1, uses reinforcement learning to make vehicle decisions and improves the exploration efficiency of reinforcement learning; S3, after the exploration phase, to improve the safety of the designed model during movement, a rule-based and learning-based movement method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision; In S1, a state value table is constructed based on implicit time scale information to establish a vehicle decision model; specifically, the following steps are performed: S1.1, building a decision model: Discretize the continuous highway into discrete units of equal length. The width of each discrete unit is equal to the lane width. Each discrete unit represents a state S. For each state S, there are four actions as shown in the following formula: In the above formula, i = 1, 2, 3 or 4, moving one unit up or down corresponds to lane changing, and moving one or two units to the right corresponds to moving at different speeds; S1.2, Importing Implicit Timescale: Based on the above decision model, the timescale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state. The implicit timescale is assigned by moving the surrounding vehicle. In S2, the exploration efficiency of reinforcement learning is improved by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, and a reinforcement learning exploration strategy is designed; The S3 includes: S3.

1. After the exploration phase, the autonomous vehicle follows the optimal feasible path found by the decision model. The following rules are designed to improve the safety of the movement: Rule 1 is to use the rules to reselect the action; Rule 2 is that when any action in the state will result in a collision or miss the target point, the ABS mechanism will be triggered, causing the vehicle to stop in the current state, and the ABS triggers the failure of the corresponding decision; as shown in the following formula: Rule 2: Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of; set2 represents the state collection that violates security; S3.2, Probability of Adoption Corresponding to the different driving styles of vehicles in the intelligent environment, different behavioral rules are established based on different driving styles: Intelligent environment vehicles have different longitudinal customary following distances according to different driving styles, as shown in the following formula: When the actual following distance is not within the customary following distance range, the vehicle brakes and accelerates accordingly to meet the needs of safety and traffic efficiency, as shown in the following formula: Among them, d f represents the habitual following distance, P is the driving style, round is the rounding function, d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to the vehicle speed respectively; S3.

3. Design the lateral lane-changing behavior of intelligent environment vehicles, which is divided into two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by driving goals; the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are established. The lane-changing behavior will only occur when the overtaking lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met.

2. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The exploration method based on the preset strategy includes: In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random move strategy is used to explore the environment as much as possible. After 100 explorations, a greedy strategy is used for action selection, as shown in the following formula: Where a(S) represents the action selection in state S; Q(S,:) represents the value table of each action in state S; n represents the number of exploration steps; During the random movement phase, to increase the number of explorations at the far end of the large map, a preset exploration strategy is used as shown in the following formula to adjust the probabilities between different actions to achieve pseudo-directional exploration: p(S,a i ) means that in state S, action a is selected i The probability that i = 1, 2, 3 or 4.

3. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The value table updating method based on back propagation includes: In the exploration phase, when the autonomous vehicle agent reaches the destination, it performs reverse value propagation on the path it has taken. After multiple backpropagations, the state and corresponding action of the state that is always on the path to the destination will be higher than other states, thus better finding the optimal feasible path; The back propagation update method is as follows: Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

4. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The breakthrough of the local optimum point based on the momentum method includes: for the designed decision model, according to the movement characteristics of the vehicle, a momentum method based on vehicle speed maintenance is designed, as shown in the following formula: S n =S n-2 a(S n )=a(S n-2 ) a(S n-1 )=a(S n-2 ) Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to judge whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, a breakthrough will be made based on the vehicle speed direction.

5. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 2, characterized in that: The design of the reinforcement learning exploration strategy includes: designing an exploration method based on the fusion of rules and learning, adopting an exploration mechanism that integrates rules and learning, setting that when the vehicle agent explores a high negative reward, the selection probability of the action will be adjusted to 0, and a second exploration will be performed in the original state. For exploratory reinforcement learning, it is shown in the following formula: in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

6. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The overtaking lane change intention is established based on the MOBIL principle and actual traffic regulations. The overtaking intention is generated by a nonlinear function based on the difference between the vehicle conditions in the adjacent lane and the current lane. The vehicle conditions in the adjacent lane and the current lane are represented by the difference between the current following distance and the following distance after the lane change, as shown in the following formula: x d =(d rf / d rf -d r )>0 Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake and change lanes to the left, d lf Iten is the following distance after changing lanes to the left lane. c Intention to change lanes for overtaking; The lane change intention caused by the driving goal is expressed as follows: the closer the vehicle is to the goal point, the stronger the lane change intention. The lane change intention setting makes the behavior of the surrounding vehicles more consistent with the real vehicles when merging into or exiting the highway, as shown in the following formula: x m =mb ji -i Among them, x m is the distance between the driving target endpoint and the current position of the environment vehicle, mb ji is the driving goal of the environment vehicle located in the i unit of lane j, Iten m For driving goal intention.

7. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The total lane change intention (TI) is defined as the sum of the two lane change intentions. A vehicle will only change lanes when the total lane change intention exceeds a threshold. This lane change intention ensures that intelligent vehicles do not repeatedly change lanes, potentially endangering traffic, while also acknowledging the differences between vehicles with different driving styles. The total lane change intention is expressed as follows: Food=Food c +Food m Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined by driving style; When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation: Among them, d rr ,d rf They represent the distance between the front and rear vehicles before changing lanes to the right lane. Lane changing will only occur when TI and TC are both equal to 1.

Images