Automatic driving decision-making method based on offline and online fusion reinforcement learning

Abstract

The invention relates to an automatic driving decision-making method based on off-line and on-line fusion reinforcement learning, which relates to the field of automatic driving and artificial intelligence and comprises two stages of off-line training and on-line strategy expansion. And initializing offline and online reinforcement learning strategies and an offline value network. In the off-line training stage, data in the static data set are put into an off-line experience pool, static data are obtained through random sampling from the off-line experience pool, and an off-line strategy and an off-line value network are updated. In the online strategy expansion stage, firstly freezing the offline strategies obtained in the offline training stage, combining the offline strategies with the online strategies to form an expansion strategy set, selecting strategies from the expansion strategy set according to a certain probability to interact with the environment, and putting interaction data into an online experience pool; and randomly sampling from the online experience pool to obtain static data, and updating an online strategy and an online value network. According to the method, the decision-making agent can benefit from further online learning while the safety of offline reinforcement learning is kept.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and artificial intelligence, and in particular to an autonomous driving decision-making method based on offline-online fusion reinforcement learning. Background Art

[0002] In the past two decades, autonomous driving has attracted great research interest because it is expected to free drivers from the tiring and boring driving process. Although autonomous driving research has made great progress in the fields of path planning, perception and control, high-level decision-making remains a challenge, especially in mixed traffic with complex dynamic driving environments. Reinforcement learning is a powerful data-driven control method with wide applications in high-level decision-making in autonomous driving. Existing reinforcement learning algorithms for autonomous driving decision-making can be divided into two categories: offline reinforcement learning and online reinforcement learning. Online reinforcement learning algorithms train decision-making agents by interacting with simulation environments or directly with driving environments, while offline reinforcement learning algorithms train decision-making agents through collected static datasets for autonomous driving.

[0003] However, pure online or offline reinforcement learning algorithms are not competent for autonomous driving tasks with complex environments. Specifically, online reinforcement learning algorithms learn control strategies through trial-and-error interactions with the environment, which may lead to catastrophic consequences such as collisions and derailment when decision-making agents are tested in real-world traffic. Offline reinforcement learning can learn effective strategies from previously collected static autonomous driving datasets without the need for active interaction, and has higher safety, but it usually leads to rather conservative strategies with limited decision-making efficiency. Summary of the invention

[0004] In order to overcome the inability of pure online or offline reinforcement learning algorithms to cope with complex environments such as autonomous driving, the present invention provides an autonomous driving decision-making method based on offline and online fusion reinforcement learning, which bridges offline and online reinforcement learning by adaptively selecting online interaction strategies, and further trains online strategies while retaining useful behaviors learned in the offline training phase and retaining the safety of offline reinforcement learning to improve the performance of the decision-making agent; at the same time, the offline strategy and the online strategy are decoupled by setting a strategy set, providing more design flexibility. While retaining the safety of offline reinforcement learning, the present invention allows the decision-making agent to benefit from further online learning.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] An autonomous driving decision-making method based on offline and online fusion reinforcement learning, including an offline training phase and an online strategy expansion phase, includes the following steps;

[0007] Step S1: Initialize offline strategy , offline value network , offline experience pool , Online Strategy , online value network , online experience pool ,in represents the empty set;

[0008] The offline training stage includes steps S2 to S4:

[0009] Step S2: store the data in the format of a quad The data in the static data set is put into the offline experience pool In which s t Represents the current state, a t represents the current action, r t Represents the current moment reward, s t+1 represents the state at the next moment, and t represents time;

[0010] Step S3: From the offline experience pool Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )};

[0011] Step S4: According to the static data d env Update offline strategy and offline value networks ;

[0012] Repeat steps S3-S4 until the offline strategy is convergence;

[0013] The line strategy expansion phase includes steps S5 to S11:

[0014] Step S5: Offline experience pool The data in is put into the online experience pool B o middle;

[0015] Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategy Combined into an extended policy set P: P =[ , ];

[0016] Step S7: From the online experience pool B o Random sampling to obtain static data denv , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )};

[0017] Step S8: According to the static data d env Calculate the state-action pair (s t ,a f ) and (s t ,a o );a f Indicates offline action, a o Indicates online actions;

[0018] Step S9: According to the state action pair (s t ,a f ) and (s t ,a o ) Calculate the online strategy selection probability P o ;

[0019] Step S10: Select probability P according to online strategy o Select an interactive strategy from the extended strategy set P to interact with the driving environment and obtain a four-tuple {(s t ,a t ,r t ,s t+1 )} and put it into the online experience pool B o middle;

[0020] Step S11: From the online experience pool B o The static data d is obtained by sampling env , and use it to update online strategies and online value network ;

[0021] Repeat steps S7 to S11 until the online strategy is convergence.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention bridges offline and online reinforcement learning by adaptively selecting online interaction strategies, further training online strategies while retaining the useful behaviors learned in the offline training phase and the safety of offline reinforcement learning, thereby improving the performance of the decision-making agent while ensuring safety;

[0024] (2) The present invention reuses offline strategies by constructing strategy sets in the online interaction process and uses them to hot-start the learning of online strategies. This can decouple offline strategies and online strategies and provide more design flexibility.

[0025] (3) The present invention discloses an autonomous driving decision-making method based on offline-online fusion reinforcement learning, which can be combined with any existing offline and online reinforcement learning algorithms and has strong universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of an autonomous driving decision-making method based on offline and online fusion reinforcement learning according to Example 1 of the present invention. DETAILED DESCRIPTION

[0027] Existing reinforcement learning algorithms applied to autonomous driving decision-making can be divided into two categories: offline reinforcement learning and online reinforcement learning. Online reinforcement learning algorithms train decision-making agents by interacting with simulation environments or directly with driving environments, while offline reinforcement learning algorithms train decision-making agents through collected static data sets for autonomous driving. However, pure online reinforcement learning algorithms learn control strategies through trial-and-error interactions with the environment, which may lead to catastrophic consequences such as collisions and derailment when tested in real-world traffic. Although pure offline reinforcement learning does not require active interaction and has higher safety, it usually leads to quite conservative strategies and limited decision-making efficiency. Therefore, in order to preserve the safety of offline reinforcement learning while allowing the decision-making agent to benefit from further online learning, the present invention proposes an autonomous driving decision-making method based on offline and online fusion reinforcement learning. The method bridges offline and online reinforcement learning by adaptively selecting online interaction strategies, further training online strategies while ensuring the reuse of offline strategies to improve the performance of decision-making agents while ensuring safety, and providing more design flexibility.

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0029] Embodiment 1:

[0030] like Figure 1 As shown, an automatic driving decision-making method based on offline and online fusion reinforcement learning of the present invention includes two stages: offline training and online strategy expansion;

[0031] Step S1: Initialize offline strategy , offline value network , Offline Experience Pool , Online Strategy , online value network , Online Experience Pool ,in represents the empty set;

[0032] The offline training stage includes steps S2 to S4:

[0033] Step S2: store the data in the format of a quad The data in the static data set is put into the offline experience pool Among them Represents the current state. Represents the current action. Represents the current moment reward, represents the state at the next moment, and t represents time;

[0034] Step S3: From the offline experience pool Random sampling to obtain static data , static data Quadruple Form representation;

[0035] Step S4: Based on static data Update offline strategy and offline value networks ,include:

[0036] S41: Based on static data , update the offline strategy through formula (1) , where max represents maximization;

[0037] (1)

[0038] That is to say, Take the maximum value ;

[0039] S42: Based on static data , update the offline value network through formula (2) ,in Indicates the next action to be performed. Indicates offline strategy Predicted current moment execution action, represents the weight coefficient, , represents the attenuation factor, , max represents maximization, min represents minimization, Representative from Randomly pick out . Indicates random extraction.

[0040] (2)

[0041] Repeat steps S3-S4 until the offline strategy is convergence;

[0042] The online strategy expansion stage includes steps S5 to S11:

[0043] Step S5: Offline experience pool The data in the online experience pool ;

[0044] Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategies Combined into an extended policy set ;

[0045] Step S7: From the online experience pool Random sampling to obtain static data , static data Quadruple Form representation;

[0046] Step S8: Based on static data Calculate the state-action pair and ,include:

[0047] S81: Static data The current state of Enter the extended policy set The two strategies in get offline actions and online actions ;

[0048] S82: Offline Action , Online Action and the current state Combine to get state-action pair and ;

[0049] Step S9: According to the state action and Calculate the online strategy selection probability ,include:

[0050] S91: Change the state action to and Enter the online value network , get the offline Q value and online Q value ;

[0051] S92: Based on offline Q value and online Q value , the online strategy selection probability is calculated by formula (3) , where m represents the annealing coefficient, .

[0052] (3)

[0053] Among them, exp() represents the exponential function.

[0054] Step S10: Select probability according to online strategy From the extended policy set Select an interactive strategy to interact with the driving environment and obtain a four-tuple Interaction data in the form of Including:

[0055] S101: From the extended policy set Select an interaction strategy ,in Select probability with online strategy The probability of being selected as the online strategy ,by The probability of being selected as the offline strategy ;

[0056] S102: Interaction Strategy According to the current state Select current moment action ;

[0057] S103: Execute the current action Interact with the driving environment, and the environment returns the current moment reward and the next moment state , get the interaction data ;

[0058] S104: Interaction data Quadruple Deposit it into the online experience pool middle.

[0059] Step S11: From the online experience pool Sampling to get static data , and use it to update online strategies and online value network ,include:

[0060] S111: From the online experience pool Random sampling to obtain static data , static data Quadruple Form representation;

[0061] S112: Based on static data , update the online strategy through formula (4) ;

[0062] (4)

[0063] S113: Based on static data , update the online value network through formula (5) ,in Indicates the next action to be performed. represents the weight coefficient, , represents the attenuation factor, .

[0064] (5)

[0065] In the offline training stage, the present invention learns a relatively safe offline reinforcement learning decision strategy from a static data set. In the online strategy expansion stage, offline and online reinforcement learning are bridged by adaptively selecting online interaction strategies. The decision agent can further train the online strategy while retaining the useful behaviors learned in the offline training stage, so as to improve the performance of the decision agent while ensuring safety. At the same time, by constructing a strategy set to decouple the offline strategy and the online strategy, more design flexibility is provided.

[0066] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. An autonomous driving decision-making method based on offline and online fusion reinforcement learning, characterized in that: It includes an offline training phase and an online strategy expansion phase, including the following steps; Step S1: Initialize offline strategy , offline value network , offline experience pool , Online Strategy , online value network , online experience pool ,in represents the empty set; The offline training stage includes steps S2 to S4: Step S2: store the data in the format of a quad The data in the static data set is put into the offline experience pool In which s t Represents the current state, a t represents the current action, r t Represents the current moment reward, s t+1 represents the state at the next moment, and t represents time; Step S3: From the offline experience pool Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )}; Step S4: According to the static data d env Update offline strategy and offline value networks ; Repeat steps S3-S4 until the offline strategy is convergence; The line strategy expansion phase includes steps S5 to S11: Step S5: Offline experience pool The data in is put into the online experience pool B o middle; Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategy Combined into an extended policy set P: P =[ , ]; Step S7: From the online experience pool B o Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )}; Step S8: According to the static data d env Calculate the state-action pair (s t ,a f ) and (s t ,a o );a f Indicates offline action, a o Indicates online actions; Step S9: According to the state action pair (s t ,a f ) and (s t ,a o ) Calculate the online strategy selection probability P o ; Step S10: Select probability P according to online strategy o Select an interactive strategy from the extended strategy set P to interact with the driving environment and obtain a four-tuple {(s t ,a t ,r t ,s t+1 )} and put it into the online experience pool B o middle; Step S11: From the online experience pool B o The static data d is obtained by sampling env , and use it to update online strategies and online value network ; Repeat steps S7 to S11 until the online strategy is convergence.

2. The automatic driving decision-making method based on offline and online fusion reinforcement learning according to claim 1 is characterized in that: The step S4 comprises: S41: Based on static data , update the offline strategy through formula (1) : （1） Among them, max represents maximization, that is, getting Take the maximum value ; S42: Based on static data , update the offline value network through formula (2) , where a t+1 Indicates the next action to be performed. Indicates offline strategy The action to be performed at the current moment of prediction, a represents the weight coefficient, , g represents the attenuation factor, , max represents maximization, min represents minimization, Represents static data Randomly pick out ; Indicates random selection; （2）。 3. The automatic driving decision-making method based on offline and online fusion reinforcement learning according to claim 2 is characterized in that: The step S8 comprises: S81: Static data The state s in t Input two strategies in the extended strategy set P and get the offline action and online actions ; S82: Offline action a f , online action o and the current state s t Combined to get the state-action pair (s t ,a f ) and (s t ,a o ).

4. The automatic driving decision-making method based on offline and online fusion reinforcement learning according to claim 3 is characterized in that: The step S9 comprises: S91: Set the state action pair (s t ,a f ) and (s t ,a o ) Input online value network , get the offline Q value (s t ,a f ) and online Q value (s t ,a o ); S92: Based on offline Q value (s t ,a f ) and online Q value (s t ,a o ), the online strategy selection probability P is calculated by formula (3) o： （3） Among them, exp() represents the exponential function, m represents the annealing coefficient, ; Intermediate parameters .

5. The automatic driving decision-making method based on offline and online fusion reinforcement learning according to claim 4 is characterized in that: The step S10 comprises: S101: Select an interaction strategy from the extended strategy set P , where the interaction strategy P o The probability of being selected as the online strategy , with 1-P o The probability of being selected as the offline strategy ; S102: Interaction Strategy According to the current state s t Select Action ; S103: Execute the action at the current moment Interact with the driving environment and return the current moment reward and the next moment state , get the interaction data ; S104: Interaction data Quadruple Deposit it into the online experience pool middle.

6. The automatic driving decision-making method based on offline and online fusion reinforcement learning according to claim 5 is characterized in that: The step S11 comprises: S111: From the online experience pool Random sampling to obtain static data , static data Quadruple : S112: Based on static data , update the online strategy through formula (4) ; （4） S113: Based on static data , update the online value network through formula (5) : （5） in, Indicates the next action to be performed. represents the weight coefficient, , represents the attenuation factor, .

Images