An autonomous driving decision-making method based on offline and online fusion reinforcement learning

Abstract

The present invention relates to an autonomous driving decision-making method based on offline-online fusion reinforcement learning, which relates to the fields of autonomous driving and artificial intelligence, and includes two stages: offline training and online strategy expansion. Initialize the strategies and offline value networks of offline and online reinforcement learning. In the offline training stage, the data in the static data set are placed in an offline experience pool, static data are randomly sampled from the offline experience pool, and the offline strategy and offline value network are updated. In the online strategy expansion stage, the offline strategy obtained in the offline training stage is first frozen and combined with the online strategy into an extended strategy set, with a certain probability, strategies are selected from the extended strategy set to interact with the environment and the interaction data are placed in the online experience pool, static data are randomly sampled from the online experience pool, and the online strategy and online value network are updated. While retaining the security of offline reinforcement learning, the present invention allows decision-making agents to benefit from further online learning.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and artificial intelligence, and in particular to an autonomous driving decision-making method based on offline-online fusion reinforcement learning. Background Art

[0002] Over the past two decades, autonomous driving has attracted great research interest because it promises to free drivers from the tedious and tedious process of driving. Although autonomous driving research has made great progress in the fields of path planning, perception, and control, high-level decision-making remains a challenge, especially in mixed traffic with complex and dynamic driving environments. Reinforcement learning is a powerful data-driven control method with wide applications in high-level decision-making in autonomous driving. Existing reinforcement learning algorithms for autonomous driving decision-making can be divided into two categories: offline reinforcement learning and online reinforcement learning. Online reinforcement learning algorithms train decision-making agents by interacting with a simulation environment or directly with the driving environment, while offline reinforcement learning algorithms train decision-making agents using collected static autonomous driving datasets.

[0003] However, purely online or offline reinforcement learning algorithms are inadequate for autonomous driving tasks in complex environments. Specifically, online reinforcement learning algorithms learn control policies through trial-and-error interaction with the environment, which can lead to catastrophic consequences such as collisions and derailment when tested in real-world traffic. Offline reinforcement learning can learn effective policies from previously collected static autonomous driving datasets without requiring active interaction, offering greater safety. However, it often results in relatively conservative policies and limited decision-making efficiency. Summary of the Invention

[0004] To overcome the inability of purely online or offline reinforcement learning algorithms to cope with the complex environment of autonomous driving, the present invention provides an autonomous driving decision-making method based on offline-online fusion reinforcement learning. This method bridges offline and online reinforcement learning by adaptively selecting an online interaction strategy. While retaining the useful behaviors learned during the offline training phase and the safety of offline reinforcement learning, it further trains the online strategy to improve the performance of the decision-making agent. Furthermore, by setting a strategy set, the offline and online strategies are decoupled, providing greater design flexibility. This method preserves the safety of offline reinforcement learning while enabling the decision-making agent to benefit from further online learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An autonomous driving decision-making method based on offline-online fusion reinforcement learning, including an offline training phase and an online strategy expansion phase, includes the following steps;

[0007] Step S1: Initialize offline strategy , offline value network , offline experience pool , Online Strategy , online value network , online experience pool ,in represents the empty set;

[0008] The offline training stage includes steps S2 to S4:

[0009] Step S2: convert the storage format into a quad The data in the static dataset is put into the offline experience pool In which s t Represents the current state, a t Represents the current action, r t Represents the current moment reward, s t+1 Represents the state at the next moment, and t represents time;

[0010] Step S3: From the offline experience pool Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )};

[0011] Step S4: According to the static data d env Update offline strategy and offline value networks ;

[0012] Repeat steps S3-S4 until the offline strategy is convergence;

[0013] The line strategy expansion phase includes steps S5 to S11:

[0014] Step S5: Offline experience pool The data in the online experience pool B o middle;

[0015] Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategies Combined into an extended policy set P: P = [ , ];

[0016] Step S7: From the online experience pool B o Random sampling to obtain static data denv , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )};

[0017] Step S8: According to the static data d env Calculate the state action pair (s t ,a f ) and (s t ,a o );a f Indicates offline action, a o Indicates online actions;

[0018] Step S9: According to the state action pair (s t ,a f ) and (s t ,a o ) Calculate the online strategy selection probability P o ;

[0019] Step S10: Select probability P according to online strategy o Select an interactive strategy from the extended strategy set P to interact with the driving environment and obtain a four-tuple {(s t ,a t ,r t ,s t+1 )} and put it into the online experience pool B o middle;

[0020] Step S11: From the online experience pool B o Sampling to get static data d env , and use it to update online strategies and online value networks ;

[0021] Repeat steps S7-S11 until the online strategy convergence.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention bridges offline and online reinforcement learning by adaptively selecting online interaction strategies. While retaining the useful behaviors learned in the offline training phase and preserving the safety of offline reinforcement learning, the online strategy is further trained, thereby improving the performance of the decision-making agent while ensuring safety.

[0024] (2) The present invention reuses offline strategies by constructing strategy sets during online interaction and hot-starting online strategy learning. This decouples offline strategies from online strategies and provides more design flexibility.

[0025] (3) The autonomous driving decision-making method based on offline-online fusion reinforcement learning disclosed in the present invention can be combined with any existing offline and online reinforcement learning algorithms and has strong universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of an autonomous driving decision-making method based on offline-online fusion reinforcement learning according to Example 1 of the present invention. DETAILED DESCRIPTION

[0027] Existing reinforcement learning algorithms for autonomous driving decision-making can be categorized as offline and online. Online reinforcement learning algorithms train decision-making agents through interaction with a simulated environment or directly with the driving environment, while offline reinforcement learning algorithms train decision-making agents using collected static autonomous driving data. However, pure online reinforcement learning algorithms learn control policies through trial-and-error interaction with the environment, which can lead to catastrophic consequences such as collisions and derailment when tested in real-world traffic. While pure offline reinforcement learning does not require active interaction and offers higher safety, it often results in relatively conservative policies and limited decision-making efficiency. Therefore, to preserve the safety of offline reinforcement learning while allowing the decision-making agent to benefit from further online learning, this paper proposes an autonomous driving decision-making method based on offline-online fusion reinforcement learning. This method bridges offline and online reinforcement learning by adaptively selecting an online interaction strategy. While reusing the offline strategy, it further trains the online strategy to improve the performance of the decision-making agent while ensuring safety, while also providing greater design flexibility.

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementation and in conjunction with the accompanying drawings.

[0029] Example 1:

[0030] like Figure 1 As shown, an autonomous driving decision-making method based on offline and online fusion reinforcement learning of the present invention includes two stages: offline training and online strategy expansion;

[0031] Step S1: Initialize offline strategy , offline value network , offline experience pool , online strategy , online value network , online experience pool ,in represents the empty set;

[0032] The offline training stage includes steps S2 to S4:

[0033] Step S2: convert the storage format into a quad The data in the static dataset is put into the offline experience pool Among them Represents the current state. Represents the current moment action, Represents the current moment reward, Represents the state at the next moment, and t represents time;

[0034] Step S3: From the offline experience pool Random sampling to obtain static data , static data Quadruple formal representation;

[0035] Step S4: Based on static data Update offline strategy and offline value networks ,include:

[0036] S41: Based on static data , update the offline strategy through formula (1) , where max stands for maximization;

[0037] (1)

[0038] That is to say, Take the maximum value ;

[0039] S42: Based on static data , update the offline value network through formula (2) ,in Indicates the next moment to perform an action. Indicates offline strategy Predict the action to be performed at the current moment, represents the weight coefficient, , represents the attenuation factor, , max represents maximization, min represents minimization, Representatives from Randomly pick out . Indicates random sampling.

[0040] (2)

[0041] Repeat steps S3-S4 until the offline strategy is convergence;

[0042] The online strategy expansion phase includes steps S5 to S11:

[0043] Step S5: Offline experience pool The data in the online experience pool ;

[0044] Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategies Combined into an extended policy set ;

[0045] Step S7: From the online experience pool Random sampling to obtain static data , static data Quadruple formal representation;

[0046] Step S8: Based on static data Calculate the state-action pair and ,include:

[0047] S81: Static data The current state of Entering an extended policy set The two strategies in get offline action and online actions ;

[0048] S82: Offline action , online action and the current state Combine to get state-action pair and ;

[0049] Step S9: According to the state action and Calculate the online strategy selection probability ,include:

[0050] S91: Change the state action to and Enter online value network , get the offline Q value and online Q value ;

[0051] S92: Based on offline Q value and online Q value , the online strategy selection probability is calculated by formula (3) , where m represents the annealing coefficient, .

[0052] (3)

[0053] Among them, exp() represents the exponential function.

[0054] Step S10: Select probability according to online strategy From the extended policy set Select the interaction strategy to interact with the driving environment and obtain the four-tuple Interaction data expressed in a formal way and put into the online experience pool Including:

[0055] S101: From the extended policy set Select an interaction strategy ,in Probability of online strategy selection The probability of being selected as the online strategy ,by The probability of being selected as the offline strategy ;

[0056] S102: Interaction Strategy According to the current state Select the current moment action ;

[0057] S103: Execute the current action Interact with the driving environment, and the environment returns the current moment reward and the next moment state , get interaction data ;

[0058] S104: Interaction data Quadruple Deposit it into the online experience pool middle.

[0059] Step S11: From the online experience pool Sampling to get static data , and use it to update online strategies and online value networks ,include:

[0060] S111: From the online experience pool Random sampling to obtain static data , static data Quadruple formal representation;

[0061] S112: Based on static data , update the online strategy through formula (4) ;

[0062] (4)

[0063] S113: Based on static data , update the online value network through formula (5) ,in Indicates the next moment to perform the action, represents the weight coefficient, , represents the attenuation factor, .

[0064] (5)

[0065] During the offline training phase, this invention learns a relatively safe offline reinforcement learning decision-making strategy from a static dataset. During the online strategy expansion phase, it bridges offline and online reinforcement learning by adaptively selecting an online interaction strategy. This allows the decision-making agent to further train the online strategy while retaining the useful behaviors learned during the offline training phase, thereby improving the decision-making agent's performance while ensuring safety. Furthermore, by constructing a strategy set, the offline and online strategies are decoupled, providing greater design flexibility.

[0066] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. An autonomous driving decision-making method based on offline and online fusion reinforcement learning, characterized in that: It includes an offline training phase and an online strategy expansion phase, and includes the following steps; Step S1: Initialize offline strategy , offline value network , offline experience pool , Online Strategy , online value network , online experience pool ,in represents the empty set; The offline training stage includes steps S2 to S4: Step S2: convert the storage format into a quad The data in the static dataset is put into the offline experience pool In which s t Represents the current state, a t Represents the current action, r t Represents the current moment reward, s t+1 Represents the state at the next moment, and t represents time; Step S3: From the offline experience pool Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )}; Step S4: According to the static data d env Update offline strategy and offline value networks ; Repeat steps S3-S4 until the offline strategy is convergence; The online strategy expansion phase includes steps S5 to S11: Step S5: Offline experience pool The data in the online experience pool B o middle; Step S6: Freeze the offline strategy obtained during the offline training phase and combine it with online strategies Combined into an extended policy set P: P = [ , ]; Step S7: From the online experience pool B o Random sampling to obtain static data d env , static data d env With the four-tuple {(s t ,a t ,r t ,s t+1 )}; Step S8: According to the static data d env Calculate the state action pair (s t ,a f ) and (s t ,a o );a f Indicates offline action, a o Indicates online actions; Step S9: According to the state action pair (s t ,a f ) and (s t ,a o ) Calculate the online strategy selection probability P o ; include: S91: Set the state action pair (s t ,a f ) and (s t ,a o ) Input online value network , get the offline Q value (s t ,a f ) and online Q value (s t ,a o ); S92: Based on offline Q value (s t ,a f ) and online Q value (s t ,a o ), the online strategy selection probability P is calculated by formula (3) o： （3） Among them, exp() represents the exponential function, represents the annealing coefficient, ; Step S10: Select probability P according to online strategy o Select an interactive strategy from the extended strategy set P to interact with the driving environment and obtain a four-tuple {(s t ,a t ,r t ,s t+1 )} and put it into the online experience pool B o middle; Step S11: From the online experience pool B o Sampling to get static data d env , and use it to update online strategies and online value networks ; Repeat steps S7-S11 until the online strategy convergence.

2. The autonomous driving decision-making method based on offline and online fusion reinforcement learning according to claim 1, characterized in that: The step S4 comprises: S41: Based on static data , update the offline strategy through formula (1) : （1） Among them, max represents maximization, that is, getting Take the maximum value ; S42: Based on static data , update the offline value network through formula (2) , where a t+1 Indicates the next moment to perform an action. Indicates offline strategy The action to be performed at the current moment of prediction, a represents the weight coefficient, a (0,1), represents the attenuation factor, (0,1), max represents maximization, min represents minimization, Represents static data Randomly pick out ; Indicates random sampling; （2）。 3. The autonomous driving decision-making method based on offline and online fusion reinforcement learning according to claim 2, characterized in that: The step S8 comprises: S81: Static data The state s in t Input two strategies in the extended strategy set P and get the offline action and online actions ; S82: Offline action a f , online action a o and the current state s t Combined to get state action pair (s t ,a f ) and (s t ,a o ).

4. The autonomous driving decision-making method based on offline and online fusion reinforcement learning according to claim 3 is characterized in that: The step S10 includes: S101: Select an interaction strategy from the extended strategy set P , where the interaction strategy P o The probability of being selected as the online strategy , with 1-P o The probability of being selected as the offline strategy ; S102: Interaction Strategy According to the current state s t Select Action ; S103: Execute the action at the current moment Interact with the driving environment and return the current moment reward and the next moment state , get interaction data ; S104: Interaction data Quadruple Deposit it into the online experience pool middle.

5. The autonomous driving decision-making method based on offline and online fusion reinforcement learning according to claim 4 is characterized in that: The step S11 includes: S111: From the online experience pool Random sampling to obtain static data , static data Quadruple The form of expression; S112: Based on static data , update the online strategy through formula (4) ; （4） S113: Based on static data , update the online value network through formula (5) : （5） in, Indicates the next moment to perform an action. represents the weight coefficient, , represents the attenuation factor, .

Images