Automatic driving vehicle decision-making method based on implicit time scale reinforcement learning

Abstract

The invention discloses an automatic driving vehicle decision-making method based on implicit time scale reinforcement learning. The method comprises a decision-making model based on implicit time scales, an improved reinforcement learning exploration method, a moving method based on rules and learning and an intelligent environment vehicle model. Time scale information is imported into a state value table by an implicit method through a decision model based on implicit time scales, so that the continuity of decision behaviors is improved; according to the improved reinforcement learning exploration method, an exploration method based on a preset strategy, a value table updating method based on back propagation and a local optimal point breakthrough based on a momentum method are used for improving the exploration efficiency of reinforcement learning; the moving method based on rules and learning and the intelligent environment vehicle model can establish a simulation environment to verify the effectiveness and continuity of the decision. According to the invention, the time information is hidden in the state value table, and the decision continuity and safety of the vehicle are improved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving vehicle decision-making, and relates to an autonomous driving vehicle decision-making method based on implicit time scale reinforcement learning. Background Art

[0002] With the rapid development of computer and communication technology, autonomous vehicles have received widespread attention, and pilot deployments of autonomous vehicles have been carried out in some cities in China. However, according to current research by relevant scholars, autonomous vehicles still have incompatibilities in driving safety, effectiveness, and behavioral rationality. Among them, the vehicle decision module, as the lower-level application of the vehicle perception module and the upper-level guidance of the vehicle trajectory planning and control module, plays an important role in the safety and efficiency of autonomous vehicles.

[0003] With the continuous development of computing power of self-driving cars, learning-based decision-making methods have become one of the mainstream methods for studying self-driving vehicle decision-making. Compared with the neural network value table of deep reinforcement learning algorithms, conventional reinforcement learning algorithms have greater research value due to their weaker black box characteristics, easy integration with rule-based methods, and strong plasticity of the original algorithm.

[0004] Faced with complex traffic environments, intelligent driving vehicles not only need to avoid obstacles to reach their destination, but also need to interact with other traffic participants to maintain traffic safety, efficiency or other requirements. How to efficiently and correctly interact with traffic has become a more important issue.

[0005] In summary, the current research methods of vehicle behavior decision-making based on learning are the current research hotspots, but the application of these methods mostly requires rapid updates, the continuity of decisions is insufficient, and the methods based on rapid updates may not be able to support the actual vehicle application of large models. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide an autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning, which hides time information in the state value table without significantly increasing the computational difficulty, thereby improving the decision-making continuity and safety of the vehicle.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions to achieve the above problems:

[0008] A decision-making method for an autonomous driving vehicle based on implicit time-scale reinforcement learning, comprising:

[0009] S1, building a decision model based on implicit time scale;

[0010] S2, based on the decision model established in S1, uses reinforcement learning to make vehicle decisions and improves the exploration efficiency of reinforcement learning;

[0011] S3, after the exploration phase, in order to improve the safety of the designed model when moving, a rule-based and learning-based mobility method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision.

[0012] The present invention also includes the following technical features:

[0013] Specifically, in S1, a state value table is constructed based on implicit time scale information to establish a vehicle decision model; specifically, the following steps are included:

[0014] S1.1, build a decision model: discretize the continuous highway into discrete units of equal length. The width of each discrete unit is equal to the lane width. Each discrete unit represents a state S. For each state S, there are four actions as shown in the following formula:

[0015]

[0016] In the above formula, i = 1, 2, 3 or 4, moving one unit up or down corresponds to lane changing, and moving one unit or two units to the right corresponds to moving at different speeds;

[0017] S1.2, import implicit time scale: Based on the above decision model, the time scale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state. The implicit time scale is assigned by moving the environmental vehicle.

[0018] Specifically, in S2, the exploration efficiency of reinforcement learning is improved by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, and a reinforcement learning exploration strategy is designed.

[0019] Specifically, the exploration method based on the preset strategy includes:

[0020] In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random movement strategy is used to explore the environment as much as possible. After 100 explorations, a greedy strategy is used for action selection, as shown in the following formula:

[0021]

[0022] Where a(S) represents the action selection in state S; Q(S,:) represents the value table of each action in state S; n represents the exploration step;

[0023] In the random movement phase, in order to increase the number of explorations at the far end of the large map, the preset exploration strategy shown in the following formula is used to adjust the probabilities between different actions to achieve pseudo-directional exploration:

[0024]

[0025] p(S,a i ) means that in state S, action a is selected i The probability that i=1, 2, 3 or 4.

[0026] Specifically, the value table updating method based on back propagation includes:

[0027] In the exploration phase, when the autonomous driving vehicle agent reaches the destination, reverse value propagation is performed on the path it has passed. After multiple reverse propagations, the state and corresponding action that are always on the path to the target point will be higher than other states, so as to better find the optimal feasible path;

[0028] The back propagation update method is as follows:

[0029]

[0030] Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

[0031] Specifically, the breakthrough of the local optimum based on the momentum method includes: for the designed decision model, according to the moving characteristics of the vehicle, a momentum method based on vehicle speed maintenance is designed, as shown in the following formula:

[0032] S n =S n-2

[0033] a(S n )=a(S n-2 )

[0034] a(S n-1 )=a(S n-2 )

[0035] Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to determine whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, it will break through based on the direction of the vehicle speed.

[0036] Specifically, the design of the reinforcement learning exploration strategy includes: designing an exploration method based on rule and learning fusion, adopting an exploration mechanism that integrates rule and learning, setting that when the vehicle agent explores a high negative reward, the selection probability of the action will be adjusted to 0, and a secondary exploration will be performed in the original state. For exploratory reinforcement learning, this method can significantly improve the exploration efficiency of each exploration cycle, as shown in the following formula:

[0037]

[0038] in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

[0039] Specifically, S3 includes:

[0040] S3.1, after the exploration phase is over, the autonomous vehicle moves forward according to the optimal feasible path explored by the decision model, and designs the following rules to improve the safety of movement:

[0041] Rule 1 is to use the rules to reselect actions; Rule 2 is that when any action in this state will result in a collision or miss the target point, the ABS mechanism will be triggered to stop the vehicle in the current state, and the ABS triggers the failure of the corresponding decision; as shown in the following formula:

[0042] Rule 1:

[0043] Rule 2:

[0044] Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of; set2 represents the state collection that violates security;

[0045] S3.2, Probability of adoption Corresponding to the different driving styles of intelligent environment vehicles, different behavioral rules are established according to different driving styles:

[0046] Intelligent environment vehicles have different longitudinal customary following distances according to different driving styles as shown in the following formula:

[0047]

[0048] When the actual following distance is not within the customary following distance range, the vehicle performs corresponding braking and acceleration to meet the needs of safety and traffic efficiency, as shown in the following formula:

[0049]

[0050] Among them, d f represents the habitual following distance, P is the driving style, round is the rounding function, and d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to the vehicle speed respectively;

[0051] S3.3, the lateral lane-changing behavior of intelligent environment vehicles is designed, which is divided into two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by driving goals; the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are set. When the overtaking lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met, the lane-changing behavior will occur.

[0052] Specifically, the establishment of the overtaking lane change intention is based on the MOBIL principle and actual traffic rules. The overtaking intention is generated by a nonlinear function established according to the difference between the vehicle conditions of the adjacent lane and the current lane; the vehicle conditions of the adjacent lane and the current lane are represented by the difference between the current following distance and the following distance after the lane change, as shown in the following formula:

[0053] x d =(d rf / d rf -d r )>0

[0054]

[0055] Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake to the left, d lf Iten is the following distance after changing lanes to the left lane. c Changing lanes with the intention of overtaking;

[0056] The lane change intention caused by the driving goal is expressed as follows: the closer the vehicle is to the target point, the stronger the lane change intention is; the setting of the lane change intention makes the behavior of the environment vehicle more consistent with the real vehicle when merging into or exiting the highway, as shown in the following formula:

[0057] x m =mb ji -i

[0058]

[0059] Among them, x mis the distance between the driving target endpoint and the current position of the environment vehicle, mb ji is the driving target of the environment vehicle located in the i unit of lane j, Iten m For driving purpose intention.

[0060] Specifically, the total lane change intention TI is defined as the sum of the two lane change intentions. When the total lane change intention is greater than the threshold, the vehicle will change lanes. The establishment of the lane change intention ensures that the intelligent environment vehicle will not change lanes repeatedly and endanger traffic, while expressing the differences between vehicles with different driving styles. The total lane change intention is as follows:

[0061] Iten=Iten c +Iten m

[0062]

[0063]

[0064] Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined for driving style;

[0065] When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation:

[0066]

[0067] Among them, d rr ,d rf They represent the front and rear vehicle distances when changing lanes to the right lane. When TI and TC are both equal to 1, lane changing will occur.

[0068] Compared with the prior art, the present invention has the following technical effects:

[0069] (I) The present invention improves the exploration efficiency of reinforcement learning by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, thereby effectively improving the exploration efficiency of the exploration stage.

[0070] (II) The present invention imports the time scale information into the state value table in an implicit way, thereby improving the continuity of the decision-making behavior and providing effective decision-making behavior for the subsequent planning and control of the vehicle.

[0071] (III) The present invention designs a corresponding rule-based and learning-based mobility method and intelligent environment vehicle model to build a simulation environment to verify the effectiveness and continuity of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1The invention flow chart is shown in FIG.

[0073] Figure 2 Decision environment diagram.

[0074] Figure 3 Set up a graph for implicit timescale rewards. DETAILED DESCRIPTION

[0075] The present invention provides an automatic driving vehicle decision-making method based on implicit time scale reinforcement learning, including a decision model based on implicit time scale, an improved reinforcement learning exploration method, a mobile method based on rules and learning, and an intelligent environment vehicle model. The decision model based on implicit time scale imports time scale information into a state value table in an implicit way, thereby improving the continuity of decision-making behavior; the improved reinforcement learning exploration method uses an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method to improve the exploration efficiency of reinforcement learning; the mobile method based on rules and learning and the intelligent environment vehicle model can establish a simulation environment to verify the effectiveness and continuity of the decision of the present invention.

[0076] Specific embodiments of the present invention are given below. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent changes made on the basis of the technical solution of this application fall within the protection scope of the present invention.

[0077] Example:

[0078] This embodiment provides a decision-making method for an autonomous driving vehicle based on implicit time-scale reinforcement learning, such as Figure 1 As shown, including:

[0079] S1, establishing a decision model based on implicit time scale; specifically, constructing a state value table based on implicit time scale information to establish a vehicle decision model;

[0080] S2, based on the decision model established in S1, uses the reinforcement learning method to make vehicle decisions and improves the exploration efficiency of reinforcement learning; specifically, uses the exploration method based on preset strategies, the value table update method based on back propagation, and the local optimal point breakthrough based on the momentum method to improve the exploration efficiency of reinforcement learning; then designs the reinforcement learning exploration strategy;

[0081] S3, after the exploration phase, in order to improve the safety of the designed model when moving, a rule-based and learning-based mobile method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision.

[0082] More specifically, it includes the following aspects and steps:

[0083] First, a decision-making model based on implicit time scale is established.

[0084] S1.1, build a decision model: discretize the continuous highway into discrete units of 5m in length, and the width of each discrete unit is equal to the lane width, such as Figure 2 Each discrete unit represents a state S. For each state S, there are four actions as shown below:

[0085]

[0086] In this formula, i = 1, 2, 3 or 4; moving one unit up or down corresponds to changing lanes, and moving one unit or two units to the right corresponds to moving forward at different speeds.

[0087] S1.2, Implicit time scale: Based on the above decision model, the time scale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state, such as Figure 3 As shown. The implicit time scale is assigned by moving the vehicle in the environment. When the autonomous driving vehicle is in the exploration stage of the environment, each exploration of the vehicle will be accompanied by the movement of the surrounding environment, which means that the vehicle will get different reward feedback when exploring the same location at different exploration times. However, due to the characteristics of highway driving, that is, the vehicle will only move forward at different speeds and will not move backward, so after the vehicle explores a certain number of times, a feasible path with implicit time scale information will be formed. This path may have a collision when observed from a static environment, but when the vehicle drives to the static collision area, the vehicle that actually caused the collision has left.

[0088] Secondly, we improve the exploration efficiency of reinforcement learning and design exploration strategies.

[0089] The update formula of the traditional Bellman optimal equation in policy iteration is shown below. For the environment constructed by the present invention, each state S is represented by the corresponding coordinates (x, y).

[0090] Q n (S,a)=(1-α)Q n-1 (S,a)+α(r+γ*max(Q(S a ,:)))

[0091] In the formula, Q n (S, a) represents the value of action a in state S at exploration time n, α is the learning rate, γ is the decay rate, and the values ​​for this embodiment are 0.1 and 0.9 respectively, r is the reward value in this state, S a is the new state reached after action a in state S, max(Q(S a ,:)) is state Sa The maximum value of all action values.

[0092] S2.1, using preset strategies to improve the exploration method: In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random movement strategy is used to explore the environment as much as possible. After 100 times, a greedy strategy is used for action selection, as shown in the following formula:

[0093]

[0094] Among them, a(S) represents the action selection under state S; Q(S,:) represents the value table of each action under state S; n represents the exploration step.

[0095] In the random movement phase, in order to increase the number of explorations at the far end of the large map, the preset exploration strategy shown in the following formula is used to adjust the probabilities between different actions to achieve pseudo-directional exploration:

[0096]

[0097] Among them, p(S,a i ) means that in state S, action a is selected i The probability of; More specifically, as follows:

[0098]

[0099] S2.2, design a value table update method based on back propagation: In a large map, due to the existence of learning rate and decay rate, it will be difficult for the autonomous driving vehicle to find the optimal path through the difference in action value when it is far away from the target. In order to further improve the exploration efficiency of the reinforcement learning algorithm, the present invention proposes a reinforcement learning update method based on back propagation. This method will perform reverse value propagation on the path passed when the autonomous driving vehicle agent arrives at the end point during the exploration phase. After multiple back propagations, the state and corresponding action that can always be in the path to the target point will be higher than other states, so as to better find the optimal feasible path;

[0100] The back propagation update method is as follows:

[0101]

[0102] Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

[0103] S2.3, design a breakthrough of local optimal point based on momentum method: For the decision model designed by the present invention, when there are many vehicles in the traffic environment, it is easy to fall into the local optimal point along the maximum state action value curve, causing the vehicle to move back and forth between two states or fall into a moving closed loop. In order to improve the ability of the reinforcement learning algorithm to jump out of the local optimal point, the present invention designs a momentum method based on vehicle speed maintenance according to the movement characteristics of the vehicle, as shown in the following formula:

[0104] S n =S n-2

[0105] a(S n )=a(S n-2 )

[0106] a(S n-1 )=a(S n-2 )

[0107] Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to determine whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, it will break through based on the direction of the vehicle speed.

[0108] S2.4, design an exploration method based on the fusion of rules and learning: Then use the above-mentioned improved reinforcement learning algorithm to explore, but when there are many vehicles in the traffic environment, the negative rewards of the environmental vehicles will gradually generate negative value partition areas as the number of explorations increases. And for the state value table of the implicit time scale, negative rewards at different times can easily constitute regional negative rewards during the exploration process. Therefore, the present invention borrows the exploration mechanism of the fusion of rules and learning, and sets that when the vehicle agent explores a high negative reward, the selection probability of the action will be adjusted to 0, and a second exploration will be performed in the original state. For exploratory reinforcement learning, this method can significantly improve the exploration efficiency of each exploration cycle, as shown in the following formula:

[0109]

[0110] in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

[0111] Thirdly, a rule-based and learning-based mobility method and a corresponding intelligent environment vehicle are established to build a simulation environment to verify the effectiveness and continuity of the present invention.

[0112] S3.1, when the exploration phase is over, the vehicle starts to move forward according to the optimal feasible path explored by the decision model, and the decision frequency is set to 1 Hz. However, in the highway environment, surrounding vehicles and autonomous driving vehicles can travel an average of more than 15 meters (determined by the minimum speed limit of the highway) within 1 second. Therefore, the present invention designs corresponding rules to improve the safety of movement.

[0113] There are two types of rules: one is to use rules to reselect actions, and the other is that when any action in this state will result in a collision or miss the target point, the ABS mechanism will be triggered, causing the vehicle to stop in the current state. The triggering of ABS also means the failure of decision-making. As shown in the following formula:

[0114] Rule 1:

[0115] Rule 2:

[0116] Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of; set2 represents a collection of states that violate security.

[0117] S3.2, in addition to the autonomous driving vehicle decided by the present invention, the accurate description of the surrounding vehicles is also important for the effectiveness of the simulation. For each individual intelligent environment vehicle, it has a different driving style, different driving goals and a random starting position. Corresponding to different driving styles, different behavioral codes are constructed based on different driving styles.

[0118] Based on relevant research, it is assumed that the influence of driving style on different behaviors is uniform. The smaller P is, the more conservative the driving style is, the more stringent the conditions for overtaking and lane changing are, and when the conditions are met, the execution probability is smaller and the following distance is longer.

[0119] In the constructed simulation environment, the intelligent environment vehicle has different longitudinal habitual following distances according to different driving styles as shown in the following formula:

[0120]

[0121] When the actual following distance is not within the customary following distance range, the vehicle will brake and accelerate accordingly to meet the needs of safety and traffic efficiency, as shown in the following formula:

[0122]

[0123] Among them, d frepresents the habitual following distance, P is the driving style, round is the rounding function, and d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to vehicle speeds of 54, 72, and 90 km / h respectively.

[0124] S3.3, then the lateral lane-changing behavior of the intelligent environment vehicle is designed, which is divided into two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by the driving goal; in order to unify the influence of different situations, the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are set. When the lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met, the lane-changing behavior will occur.

[0125] The establishment of overtaking lane change intention is based on the MOBIL principle and actual traffic rules. The overtaking intention is generated by a nonlinear function based on the difference between the vehicle conditions of the adjacent lane and the current lane. The vehicle conditions of the adjacent lane and the current lane are expressed by the difference between the current following distance and the following distance after lane change, as shown in the following formula:

[0126] x d =(d rf / d lf -d r )>0

[0127]

[0128] Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake to the left, d lf Iten is the following distance after changing lanes to the left lane. c Changing lanes with the intention of overtaking.

[0129] The lane change intention caused by the driving goal can be expressed as follows: the closer the vehicle is to the target point, the stronger the lane change intention. The setting of the lane change intention makes the behavior of the environment vehicle more consistent with the real vehicle when merging into or exiting the highway, as shown in the following formula:

[0130] x m =mb ji -i

[0131]

[0132] Among them, x m is the distance between the driving target endpoint and the current position of the environment vehicle, mb jiis the driving target of the environment vehicle located in the i unit of lane j, Iten m For driving purpose intention.

[0133] The total lane change intention TI is defined as the sum of the two lane change intentions. The vehicle will change lanes only when the total lane change intention is greater than the threshold. The establishment of the lane change intention ensures that the intelligent environment vehicle will not change lanes repeatedly and endanger traffic, while expressing the differences between vehicles with different driving styles.

[0134] Iten=Iten c +Iten m

[0135]

[0136]

[0137] Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined by driving style.

[0138] When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation:

[0139]

[0140] Among them, d rr ,d rf They represent the front and rear vehicle distances when changing lanes to the right lane. When TI and TC are both equal to 1, lane changing will occur.

[0141] It should be noted that the adjacent lane and lane change intention calculations in the formula are calculated separately based on the differences between left and right lane changes.

[0142] To sum up, this is the process of a highway autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning proposed by the present invention. The design of the present invention can accelerate the exploration efficiency of the algorithm and improve the effectiveness and continuity of vehicle decision-making.

[0143] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0144] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0145] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A decision-making method for an autonomous driving vehicle based on implicit time-scale reinforcement learning, characterized in that: include: S1, building a decision model based on implicit time scale; S2, based on the decision-making model established in S1, uses the reinforcement learning method to make vehicle decisions and improves the exploration efficiency of reinforcement learning; S3, after the exploration phase, in order to improve the safety of the designed model when moving, a rule-based and learning-based mobility method and an intelligent environmental vehicle model are designed to establish a simulation environment to verify the effectiveness and continuity of the decision.

2. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: In S1, a state value table is constructed based on implicit time scale information to establish a vehicle decision model; specifically, the following steps are performed: S1.1, build a decision model: discretize the continuous highway into discrete units of equal length. The width of each discrete unit is equal to the lane width. Each discrete unit represents a state S. For each state S, there are four actions as shown in the following formula: In the above formula, i = 1, 2, 3 or 4, moving one unit up or down corresponds to lane changing, and moving one unit or two units to the right corresponds to moving forward at different speeds; S1.2, import implicit time scale: Based on the above decision model, the time scale information is implicitly imported into its state value table. The state value table records the corresponding values ​​of the four actions of the vehicle in each state. The implicit time scale is assigned by moving the environmental vehicle.

3. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: In S2, the exploration efficiency of reinforcement learning is improved by using an exploration method based on a preset strategy, a value table update method based on back propagation, and a local optimal point breakthrough based on the momentum method, and a reinforcement learning exploration strategy is designed.

4. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning as claimed in claim 3, characterized in that: The exploration method based on the preset strategy includes: In order to strike a balance between the exploration efficiency and exploration effect of reinforcement learning, a corresponding exploration action selection strategy is designed. In the first 100 explorations, a random movement strategy is used to explore the environment as much as possible. After 100 explorations, a greedy strategy is used for action selection, as shown in the following formula: Where a(S) represents the action selection in state S; Q(S,:) represents the value table of each action in state S; n represents the exploration step; In the random movement phase, in order to increase the number of explorations at the far end of the large map, the preset exploration strategy shown in the following formula is used to adjust the probabilities between different actions to achieve pseudo-directional exploration: p(S,a i ) means that in state S, action a is selected i The probability that i=1, 2, 3 or 4.

5. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning as claimed in claim 3, characterized in that: The value table updating method based on back propagation includes: In the exploration phase, when the autonomous driving vehicle agent reaches the destination, reverse value propagation is performed on the path it has passed. After multiple reverse propagations, the state and corresponding action that are always on the path to the target point will be higher than other states, so as to better find the optimal feasible path; The back propagation update method is as follows: Among them, Q(s,a) n-t is the state and action value at time t before reaching the target point, and Reward is the reward value of the target point.

6. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning as claimed in claim 3, characterized in that: The local optimum breakthrough based on the momentum method includes: for the designed decision model, according to the moving characteristics of the vehicle, a momentum method based on vehicle speed maintenance is designed, as shown in the following formula: S n =S n-2 a(S n )=a(S n-2 ) a(S n-1 )=a(S n-2 ) Among them, a(S n-1 ) is in state S n-1 The maximum value action selection, S n-1 is the state at the previous moment; this formula is used to determine whether the decision falls into the local optimal point, that is, whether it will return to the original state after two exploration moves. When it falls into the local optimal point, it will break through based on the vehicle speed direction.

7. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning as claimed in claim 3, characterized in that: The design of the reinforcement learning exploration strategy includes: designing an exploration method based on rule and learning fusion, adopting an exploration mechanism based on rule and learning fusion, setting that when the vehicle agent explores a high negative reward, the selection probability of the action will be adjusted to 0, and a secondary exploration will be performed in the original state. For exploratory reinforcement learning, this method can significantly improve the exploration efficiency of each exploration cycle, as shown in the following formula: in, For S n Next action a i The state after moving, set1 is the state collection of high negative rewards.

8. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 1, characterized in that: The S3 includes: S3.1, after the exploration phase is over, the autonomous vehicle moves forward according to the optimal feasible path explored by the decision model, and designs the following rules to improve the safety of movement: Rule 1 is to use the rules to reselect actions; Rule 2 is that when any action in this state will result in a collision or miss the target point, the ABS mechanism will be triggered to stop the vehicle in the current state, and the ABS triggers the failure of the corresponding decision; as shown in the following formula: Rule 1: Rule 2: Among them, values(S n-1 ,a i ) is state S n-1 The next action a i The value of; set2 represents the state collection that violates security; S3.2, Probability of adoption Corresponding to the different driving styles of intelligent environment vehicles, different behavioral rules are established according to different driving styles: Intelligent environment vehicles have different longitudinal customary following distances according to different driving styles as shown in the following formula: When the actual following distance is not within the customary following distance range, the vehicle performs corresponding braking and acceleration to meet the needs of safety and traffic efficiency, as shown in the following formula: Among them, d f represents the habitual following distance, P is the driving style, round is the rounding function, and d r is the actual following distance, d m is the moving distance, and the moving unit numbers 3, 4, and 5 correspond to the vehicle speed respectively; S3.3, the lateral lane-changing behavior of intelligent environment vehicles is designed, which is divided into two types: one is the overtaking lane-changing behavior based on traffic efficiency, and the other is the target lane-changing behavior guided by driving goals; the generation of lane-changing behavior is divided into the generation of lane-changing intention and the permission of lane-changing conditions, and corresponding thresholds and conditions are set. When the overtaking lane-changing intention exceeds the lane-changing intention threshold and the lane-changing conditions are met, the lane-changing behavior will occur.

9. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning as claimed in claim 8, characterized in that: The establishment of the overtaking lane change intention is based on the MOBIL principle and actual traffic rules. The overtaking intention is generated by a nonlinear function established according to the difference between the vehicle conditions of the adjacent lane and the current lane; the vehicle conditions of the adjacent lane and the current lane are represented by the difference between the current following distance and the following distance after the lane change, as shown in the following formula: x d =(d rf / d rf -d r )>0 Among them, x d is the vehicle condition difference variable, d rf is the following distance after changing lanes to the right lane, d rf Can also be replaced by d lf Used to calculate the intention to overtake to the left, d lf Iten is the following distance after changing lanes to the left lane. c Changing lanes with the intention of overtaking; The lane change intention caused by the driving goal is expressed as follows: the closer the vehicle is to the target point, the stronger the lane change intention is; the setting of the lane change intention makes the behavior of the environment vehicle more consistent with the real vehicle when merging into or exiting the highway, as shown in the following formula: x m =mb ji -i Among them, x m is the distance between the driving target endpoint and the current position of the environment vehicle, mb ji is the driving target of the environment vehicle located in the i unit of lane j, Iten m For driving purpose intention.

10. The autonomous driving vehicle decision-making method based on implicit time-scale reinforcement learning according to claim 8, characterized in that: The total lane change intention TI is defined as the sum of the two lane change intentions. When the total lane change intention is greater than the threshold, the vehicle will change lanes. The establishment of lane change intention ensures that the intelligent environment vehicle will not change lanes repeatedly and endanger traffic, while expressing the differences between vehicles with different driving styles. The total lane change intention is as follows: Food=Food c +Food m Among them, Iten is the total lane change intention, I Y Lane change intention threshold determined for driving style; When the lane-changing intention is greater than the lane-changing threshold, the lane-changing condition of the intelligent environment vehicle must also be satisfied as shown in the following equation: Among them, d rr ,d rf They represent the front and rear vehicle distances when changing lanes to the right lane. When TI and TC are both equal to 1, lane changing will occur.

Images