Reinforced learning method and system

Abstract

The invention discloses a reinforcement learning method and system, and relates to the field of reinforcement learning, and the method comprises the steps: obtaining suboptimal demonstration data in a target application scene; performing imitation learning through the intelligent agent according to the suboptimal demonstration data to obtain an initial strategy; determining a plurality of behavior sequences based on the initial strategy; performing iterative training on the reward model according to the plurality of behavior sequences to obtain a trained reward model; based on the trained reward model, introducing a regular term into a contrast reinforcement learning algorithm of KL divergence to obtain a contrast reinforcement learning loss function with the regular term; and according to the plurality of behavior sequences, based on the trained reward model and the contrast reinforcement learning loss function with the regular term, performing reinforcement learning to obtain a final reinforcement learning strategy in the target application scene. According to the method, the reward model and the loss function of contrast reinforcement learning are improved, and after the agent simulates and learns the suboptimal demonstration data, a better strategy can be quickly learned.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning, and in particular to a reinforcement learning method and system. Background Art

[0002] Reinforcement learning is a machine learning method that allows computer systems to learn optimal behavior strategies through trial and error. In reinforcement learning, the agent learns how to take actions to maximize a certain cumulative reward through interaction with the environment. Although reinforcement learning has demonstrated its potential in many fields, how to effectively use environmental feedback to improve the learning efficiency and performance of the agent remains a challenge. Summary of the invention

[0003] The purpose of this application is to provide a reinforcement learning method and system, which can enable an intelligent agent to quickly learn a better strategy after imitating and learning suboptimal demonstration data.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a reinforcement learning method, comprising:

[0006] Obtain suboptimal demonstration data for the target application scenario;

[0007] The intelligent agent performs imitation learning based on the suboptimal demonstration data to obtain an initial strategy;

[0008] Based on the initial strategy, determining a plurality of action sequences;

[0009] Iteratively train the reward model according to multiple behavior sequences to obtain a trained reward model;

[0010] Based on the trained reward model, a regularization term is introduced into the contrastive reinforcement learning algorithm of KL divergence to obtain the contrastive reinforcement learning loss function with a regularization term.

[0011] According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrastive reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario.

[0012] In a second aspect, the present application provides a reinforcement learning system, comprising:

[0013] A suboptimal data acquisition module, used to acquire suboptimal demonstration data in a target application scenario;

[0014] An initial strategy determination module, used for obtaining an initial strategy by an intelligent agent performing imitation learning according to the suboptimal demonstration data;

[0015] A behavior sequence determination module, used to determine a plurality of behavior sequences based on the initial strategy;

[0016] A reward model training module, used to iteratively train the reward model according to multiple behavior sequences to obtain a trained reward model;

[0017] A loss determination module, which is used to introduce a regularization term into the contrastive reinforcement learning algorithm of KL divergence based on the trained reward model to obtain a contrastive reinforcement learning loss function with a regularization term;

[0018] The reinforcement learning module is used to perform reinforcement learning according to multiple behavior sequences, based on the trained reward model and the contrast reinforcement learning loss function with a regularization term, so as to obtain the final reinforcement learning strategy in the target application scenario.

[0019] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0020] The present application provides a reinforcement learning method and system, wherein an intelligent agent obtains an initial strategy through imitation learning based on suboptimal demonstration data, determines multiple behavior sequences based on the initial strategy, and iteratively trains a reward model based on multiple behavior sequences, thereby reducing the data flow required for training the reward model. Further, based on the trained reward model, a regularization term is introduced into the contrastive reinforcement learning algorithm of KL divergence to obtain a contrastive reinforcement learning loss function with a regularization term; reinforcement learning is performed based on multiple behavior sequences based on the trained reward model and the contrastive reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario. By improving the reward model and the loss function of contrastive reinforcement learning, the intelligent agent can quickly learn a better strategy after imitating and learning suboptimal demonstration data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 This is an application environment diagram of a reinforcement learning method in one embodiment of the present application;

[0023] Figure 2 A flowchart of a reinforcement learning method provided in one embodiment of the present application;

[0024] Figure 3 A schematic diagram of functional modules of a reinforcement learning system provided in one embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The reward function is an important component of guiding the agent to explore and utilize the interaction data of the environment. For a reinforcement learning system, a stable reward model must be constructed first. However, the construction of the reward model is mostly based on the experience of engineers. As for whether an excellent agent can be trained well, it requires multiple experiments to adjust the reward model and hyperparameters. In the field of trajectory planning of agents where rewards are difficult to define and the field of artificial intelligence content generation, methods such as inverse reinforcement learning (IRL) and behavior cloning (BC) are used to simplify or even eliminate reward models, but these two methods cannot significantly outperform the demonstrator. Because IRL usually seeks a reward function that makes the demonstrator's behavior look close to the best, rather than inferring the demonstrator's potential intention to execute the best strategy in the environment. BC directly supervises the behavior strategy of the demonstrator. This method will change the state distribution at the time of testing due to the compound error of action prediction, resulting in the strategy being unable to recover from the unknown state. Therefore, a good reward model needs to understand the demonstrator's intentions, rather than simply making the agent imitate the demonstration behavior that may be suboptimal.

[0028] Daniel S. Brown tried to train a reward model by comparing the pros and cons of trajectories. The reward model can infer the potential intentions of the demonstrator, and finally conduct reinforcement learning training to learn better strategies. This method is theoretically feasible, but it is extremely difficult to train a reward model with a granularity of state-action pairs based on comparing the pros and cons of the entire trajectory, and it is also difficult to construct sufficient data. In addition, the reward model trained by the comparison method is difficult to avoid reward deviation. For example, in the same environment, the score for trajectory 1 is 100 and the score for trajectory 2 is 101, while in another environment, the score for trajectory 1 is 0 and the score for trajectory 2 is 1. In the comparison, trajectory 2 is obviously better than trajectory 1 in the two environments. However, in reinforcement learning, both trajectories in the former environment are regarded as better trajectories, which will greatly affect the training effect. Google also trained a reward model with intention representation based on the energy model, but this reward model also has the disadvantage of being difficult to train successfully.

[0029] After obtaining an ideal reward model, reinforcement learning training is required. However, many researchers use the trained model for original proximal policy optimization (PPO) training or off-policy reinforcement learning training, such as the Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) algorithms. If the scene is complex and changeable, the trained reward model may not be generalized to such a wide state-action space, which will directly lead to training failure.

[0030] Therefore, the present application provides a reinforcement learning method, which enables the generated trajectory or content to surpass the suboptimal demonstration data through reinforcement learning from observation. After the intelligent agent imitates and learns the suboptimal demonstrator strategy, it quickly learns a better strategy, which mainly includes the design and training of a reward model, and the use of the reward model for comparative reinforcement learning.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] The reinforcement learning method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send suboptimal demonstration data in the target application scenario to the server 104. After the server 104 receives the suboptimal demonstration data in the target application scenario, the initial strategy is obtained by the intelligent agent through imitation learning based on the suboptimal demonstration data, and multiple behavior sequences are determined based on the initial strategy. According to the multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regular term to obtain the final reinforcement learning strategy in the target application scenario. The server 104 can feedback the final reinforcement learning strategy obtained to the terminal 102. In addition, in some embodiments, the reinforcement learning method can also be implemented by the server 104 or the terminal 102 alone.

[0033] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0034] In an exemplary embodiment, Figure 2 As shown, a reinforcement learning method is provided, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 206. Among them:

[0035] Step 201, obtain suboptimal demonstration data in the target application scenario. Here, the robot trajectory generation scenario is used for reasoning, and it can also be extrapolated to decision planning, artificial intelligence generated content (AIGC), Atari series games, OpenAI Gym arena and other scenarios.

[0036] Step 202, the intelligent agent performs imitation learning based on the suboptimal demonstration data to obtain an initial strategy.

[0037] In a specific example, the following formula is used for imitation learning:

[0038]

[0039] in, is the parameter adjusted in the imitation learning process, E is the expectation, (s, a) is the state-action pair, s is the state, a is the action, and B init is the distribution of strategies based on suboptimal demonstration data, is the loss function for supervised learning, For state s, the parameters The actions generated by the defined strategy, To minimize the loss function during the learning process The value of The defined strategy is the initial strategy. Explore the minimum value of the variable, and finally get the training The value is Corresponding strategies Defined as the initial strategy π Initial .

[0040] Step 203: Determine multiple behavior sequences based on the initial strategy.

[0041] Specifically, according to the initial strategy π Initial , multiple trajectories (i.e., multiple behavior sequences) are sampled at a certain sampling temperature and ranked to train the reward model.

[0042] Step 204, iteratively train the reward model according to the multiple behavior sequences to obtain a trained reward model.

[0043] In a preferred embodiment, the loss function when training the reward model is:

[0044]

[0045] Among them, loss(θ) is the loss function value, E is the expectation, x is the starting state, y is the 1 and 2 are two different behavior sequences, i.e., the equal-length trajectories obtained by randomly downsampling the initial strategy, containing a state sequence of length n (s i ,s i+1 ,…s i+n-1 ) or a sequence of state-action pairs (s i ,a i ,s i+1 ,a i+2 ,…s i+n-1 ,a i+n-1 ), D is the data distribution of the initial strategy, r θ() is the reward model, θ is the parameter adjusted during the training process of the reward model, and σ() is the sigmoid function, which is used to convert the score value into a probability value with a value range of 0 to 1.

[0046] Here we assume that y 1 Better than y 2 According to the loss function of the above reward model, we can get a reward r that can score the entire trajectory. θ (x,y), and conduct comparative reinforcement learning based on the trajectory scores.

[0047] Step 205 , based on the trained reward model, a regularization term is introduced into the contrastive reinforcement learning algorithm of KL divergence to obtain a contrastive reinforcement learning loss function with a regularization term.

[0048] The goals of reinforcement learning are: Among them, object(φ) is the target value of reinforcement learning, φ is the variable, Represents the strategy π φ The starting state x and trajectory y sampled from the relevant data distribution D, From strategy π φ The data set sampled by the reinforcement learning process in step 204. θ (x, y), and explore the maximum value of object(φ) by changing φ, which is the reinforcement learning process.

[0049] In order to solve the problem that the target strategy deviates too far from the initial strategy during training and collapses, this application chooses to use a contrastive reinforcement learning algorithm with controllable KL divergence, whose goals are:

[0050]

[0051] Among them, β is the Lagrange multiplier of the KL divergence term, According to the strategy π φ The probability of choosing action y in state x, π Initial (y|x) means according to the strategy π Initial The probability of choosing action y in state x, represents the reinforcement learning strategy determined by the parameter φ, and KL represents the KL divergence.

[0052] The following describes the reasoning process of the comparative reinforcement learning algorithm:

[0053] (1) Construct the Lagrangian function:

[0054]

[0055] Among them, α is the set of weight coefficients of the constraint terms of the probability distribution, ensuring that the action distribution of the strategy π for each state x is normalized. X is the set of all states, α x is the weight coefficient of the constraint term corresponding to each state x, and α and β correspond to the Lagrange multipliers.

[0056] (2) Based on the above Lagrangian function, the objective of the contrastive reinforcement learning algorithm is simplified to the following formula. To simplify the representation, the reinforcement learning strategy is represented as π(y|x):

[0057]

[0058] Its constraints are

[0059] in, represents the unnormalized discounted state distribution when the strategy is π, Indicates that in the initial strategy π Initial The state value function under ∈ represents a small positive number, D KL ( ) represents the divergence, π(·|x) represents the probability distribution of different actions y selected by strategy π in state x, where · represents the probability distribution of all possible actions y, π Initial (·|x) means that in state x, the initial strategy n Initial The probability distribution of choosing different actions y.

[0060] (3) Taking the derivative of π(y|x) we get:

[0061]

[0062] (4) The goal of reinforcement learning optimization is to obtain the extreme value of the function, so let We can get:

[0063]

[0064] (5) Due to The optimal strategy for constrained reinforcement learning is:

[0065]

[0066] Among them, π * The * in (y|x) represents the goal of reinforcement learning, that is, the optimal strategy.

[0067] (7) Let the partition function Z(x) be:

[0068]

[0069] Then the implicit reward function is:

[0070]

[0071] make:

[0072]

[0073] When θ (x,y 1 )>r θ (x,y 2 ), according to the Bradley-Terry model:

[0074]

[0075] in, Indicates that in state x, the action (behavior sequence) y 1 Relative to action (behavior sequence) y 2 Better probability.

[0076] At this point, we can get the loss function L(π φ ; rπ Initial ):

[0077]

[0078] Among them, π(φ k ) is φ k The strategy determined for the parameter.

[0079] The loss function is proved below.

[0080] In the Plackett-Luce preference framework, especially the Bradley-Terry preference framework, two reward functions from the same equivalence class induce the same preference distribution. The proof is as follows:

[0081] If for some function f(x), r θ '(x,y)=r θ (x,y)+f(x), then the two reward models r θ (x,y) and r θ '(x,y) are from the same equivalence class. The Plackett-Luce model is considered (the Bradley-Terry model is a special case with K=2), and the specific reward model r θ The ranking probability distribution caused by (x,y) is denoted as p r For any state x, action y 1 , …, y KAnd ranking τ, the following formula can be listed:

[0082]

[0083] The theory was confirmed.

[0084] In constrained reinforcement learning problems, two reward models from the same equivalence class can induce the same optimal policy. The proof is as follows: Consider two reward models from the same class, such as r θ '(x,y)=r θ (x,y)+f(x), according to this reward model, the optimal strategy is solved by the formula listed above:

[0085]

[0086]

[0087] Integrate y on both sides of the equation and use Solving the optimal solution of constraint reinforcement learning is:

[0088]

[0089] This formula is about π * The implicit function of (y|x) is the same as the optimal strategy of reinforcement learning above, which means r θ (x,y) and r θ '(x,y)The optimal strategy obtained is the same, and the theory is proven.

[0090] The reinforcement learning strategy π(y|x) can be calculated based on the above loss function L(π φ ; π Initial ) is updated, but this loss function can easily lead to overfitting and still needs further improvement.

[0091] At this time, the estimation of KL divergence still has a certain error. In order to reduce the negative impact of the error, you can multiply it by the comparison reward difference:

[0092]

[0093] Since the hidden meaning of the formula is to increase the probability of generating good trajectories and reduce the probability of generating poor trajectories, it is easy to cause overfitting. The regularization term can be used to suppress overfitting. Finally, the contrast reinforcement learning loss function with the regularization term is:

[0094]

[0095] Among them, L(π φ ; π Initial ) is the contrast reinforcement learning loss function value with regularization term, π φTo compare the strategies generated during reinforcement learning, π Initial is the initial strategy, E is the expectation, x is the starting state, y 1 and 2 For two different behavior sequences, π(φ k ) is φ k The strategy for parameter determination, r θ ( ) is the reward model, ε 1 for y 1 The corresponding intermediate quantity, ε 2 for y 2 The corresponding intermediate quantity, β is the Lagrange multiplier of the KL divergence term, π(y 1 |x) is the behavior sequence y selected in the starting state x according to the strategy π 1 The probability of π(y 2 |x) is the behavior sequence y in the starting state x according to the strategy π 2 The probability of Initial (y 1 |x) is based on the initial strategy π Initial Selecting action sequence y in starting state x 1 The probability of Initial (y 2 |x) is based on the initial strategy π Initial Selecting action sequence y in starting state x 2 The probability of σ( ) is the sigmoid function, which is used to convert the score value into a probability value with a range of 0 to 1. γ and α are the weight coefficients of the regularization term. θ (y 1 |x) is the behavior sequence y selected in the starting state x 1 The reward, r θ (y 2 |x) is the behavior sequence y selected in the starting state x 2 The reward, π θ (y 1 |x) is based on the strategy π θ Selecting action sequence y in starting state x 1 The probability of θ (y 2 |x) is based on the strategy π θ Selecting action sequence y in starting state x 2 probability.

[0096] The final reinforcement learning strategy π(y|x) can be updated according to this contrastive reinforcement learning loss function with a regular term to stabilize the training and obtain a better trajectory.

[0097] Step 206 , performing reinforcement learning based on the trained reward model and the contrast reinforcement learning loss function with a regularization term according to the multiple behavior sequences, so as to obtain a final reinforcement learning strategy in the target application scenario.

[0098] This application provides four target application scenarios, as follows:

[0099] (1) Vehicle lane-changing trajectory planning scenario. That is, the target application scenario is the vehicle lane-changing trajectory planning scenario.

[0100] The suboptimal demonstration data is a suboptimal lane-changing trajectory. The multiple behavior sequences are multiple trajectory sequences obtained by randomly sampling the suboptimal lane-changing trajectory. The form of each trajectory sequence is (s i ,a i ,s i+1 ,a i+2 ,…s i+n-1 ,a i+n-1 ), where s i is the i-th state, a i is the ith action, n is the length of the trajectory sequence, the state includes the distance of the vehicle from the center line of the lane, the speed of the vehicle and the heading angle of the vehicle, and the action includes the lateral acceleration of the vehicle and the front wheel turning angle of the vehicle. The final reinforcement learning strategy is the optimal lane changing strategy.

[0101] According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: selecting two trajectory sequences from multiple trajectory sequences, using the trained reward model to determine the reward values ​​of the two trajectory sequences. According to the reward values ​​of each of the two trajectory sequences in the multiple trajectory sequences, reinforcement learning is performed based on the contrast reinforcement learning loss function with a regularization term to obtain the optimal lane changing strategy.

[0102] Specifically, a policy network is initialized first, a certain number of suboptimal lane-changing trajectories are collected, and an initial policy is obtained by imitating the suboptimal lane-changing trajectories. According to the above suboptimal lane-changing trajectories, two trajectories are randomly selected each time, and two trajectory sequences with a horizon length of n are selected from the two trajectories. The trajectories are ranked according to certain human driver lane-changing trajectory feature standards, and then the reward model calculates the reward value, selects good trajectory sequences and bad trajectory sequences, and finally calculates the loss according to the contrastive reinforcement learning loss function with a regular term, and performs gradient descent to obtain a new optimal lane-changing strategy that meets the characteristics of human drivers.

[0103] (2) Vehicle driving scenario. That is, the target application scenario is the vehicle driving scenario.

[0104] The initial strategy is the initial driving strategy. The multiple behavior sequences are multiple trajectory sequences obtained by randomly sampling the initial driving strategy. The final reinforcement learning strategy is the optimal driving strategy.

[0105] According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: selecting two trajectory sequences from multiple trajectory sequences, using the trained reward model, and determining the reward values ​​of the two trajectory sequences. According to the reward values ​​of each two trajectory sequences in the multiple trajectory sequences, reinforcement learning is performed based on the contrast reinforcement learning loss function with a regularization term to obtain the optimal driving strategy.

[0106] Specifically, if there is already a rule-based initial strategy, collect the safe driving data of the vehicle under the initial strategy, and obtain the initial strategy through imitation learning. Since the reward for decision-making can be defined based on safety and efficiency, the reward model provided by this application is generally not used to learn the reward function, so all driving data (s i ,a i ,r i ,s i+1 ) Perform offline reinforcement learning to obtain the initial strategy; r i is the reward value of the i-th trajectory sequence.

[0107] In terms of the specific experimental settings, it is assumed that the Markov decision process in this scenario is completely observable. According to the above formula, it can be proved that the sequence x before the sequence y can be eliminated without affecting the reward and target strategy, that is, r θ (x),y)=r θ (y) = r θ (τ π (s)), τ π (s) is the trajectory with the starting state s. In this way, each sampling only needs to select a pair of state-action sequences with a certain horizon for comparison and reinforcement. Considering that the total reward of the trajectory sequence with different starting states s for vehicle decision making is quite different, each dimension of the scene state can be divided into N equal parts, and the size of each part is: (max(s i )-min(s i ) / N. When selecting data pairs, only the data with the same initial state need to be considered. A large amount of data can be added, deleted, modified and checked using the R-tree to improve efficiency. In order to increase the number of training samples, partial trajectory pairs with a certain horizon length n are trained instead of full trajectory pairs in one round. This data enhancement method can obtain contrast enhancement of many partial decision sequences.

[0108] In this trajectory, the reward value is the sum of all discounted rewards, i.e., rθ (τ π (s)). By strategy π φ Collect trajectory samples for training reinforcement learning policies:

[0109]

[0110] Among them, the dataset D(s,a) saves the state, action, ε and total reward of all trajectories.

[0111] The specific operation is divided into two steps: data collection and offline update:

[0112] 1) Collect enough data. First, copy the initial strategy π Initial Get the reinforcement learning strategy π φ , according to π φ The trajectory sequence is obtained by sampling. When sampling, the temperature sampling can be performed according to the inverse of the normalized cumulative discounted reward to reasonably increase the exploration rate. The initial r θ (τ π (s i )) are all set to 1.

[0113] Calculate the discount cumulative reward r according to the above formula θ (τ π (s i ), since the decision only considers the cumulative reward of action a after state s, ε can be defined as Put the data into D(s,a).

[0114] 2) Traverse the states in the data, query and find all tuples with the same initial state, and calculate the cumulative discount reward r θ Sort, find the maximum and minimum values, normalize and get the loss, and finally π φ Perform gradient descent and update offline.

[0115] By continuously iterating the above two processes, a better driving strategy can be obtained.

[0116] (3) Large model alignment scenario. That is, the target application scenario is the large model alignment scenario.

[0117] The initial strategy is the pre-trained large model. The multiple behavior sequences are multiple word-unit sequences generated after inputting multiple prompt words into the pre-trained large model respectively. Each prompt word generates two word-unit sequences; the final reinforcement learning strategy is the aligned large model.

[0118] According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: selecting two word-unit sequences from multiple word-unit sequences, using the trained reward model to determine the reward values ​​of the two word-unit sequences. According to the reward values ​​of every two word-unit sequences in the multiple word-unit sequences, reinforcement learning is performed based on the contrast reinforcement learning loss function with regularization term to obtain the aligned large model.

[0119] Specifically, based on the pre-trained large model, a certain amount of prompt words are selected according to the task, and each prompt word is input into the pre-trained large model to generate a pair of token sequences, which are ranked according to certain standards, and then the reward model is learned to obtain the reward model r θ (x,y). When training reinforcement learning, first copy the pre-trained large model to get π φ , and then input these prompt words into π φ In the example, each prompt word generates two token sequences, and then according to r θ (x, y) selects good token sequences and bad token sequences, and finally calculates the loss according to the contrast reinforcement learning loss function with a regular term, and performs gradient descent to obtain the new aligned large model π φ .

[0120] (4) Battery charging and discharging strategy optimization scenario. That is, the target application scenario is the battery charging and discharging strategy optimization scenario.

[0121] The initial strategy is the initial battery charge and discharge strategy curve. The multiple behavior sequences are multiple charge and discharge trajectories obtained by randomly sampling the initial battery charge and discharge strategy curve. Among them, the charge and discharge trajectory is the whole process curve from the beginning of battery charging to the end of battery discharging. Each charge and discharge trajectory includes a state space and an action space. The state space is the current state of the battery, and the action space includes a charging strategy and a discharging strategy. The final reinforcement learning strategy is the optimal battery charge and discharge strategy.

[0122] According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: selecting two charging and discharging trajectories from multiple charging and discharging trajectories, using the trained reward model to determine the reward values ​​of the two charging and discharging trajectories. According to the reward values ​​of every two charging and discharging trajectories in the multiple charging and discharging trajectories, reinforcement learning is performed based on the contrast reinforcement learning loss function with regularization term to obtain the optimal battery charging and discharging strategy.

[0123] Specifically, the state space is the current state of the battery, such as power, voltage, temperature, charge / discharge rate, etc. The action space is the charging strategy (such as current size, charging time) and the discharging strategy (such as discharge rate, sleep strategy). The reward function can be designed as an indicator of battery life, such as cycle life or remaining capacity. High rewards correspond to long life and stable battery performance. The generated strategy is the curve of the entire process from the beginning of battery charging to the end of discharging. Each curve contains state, action and result. Reward modeling uses Bradley-Terry and Luce-Shephard models to model these charge and discharge curves, sort each curve by preference according to battery life or other performance indicators, and train the reward model. An initial battery charge and discharge strategy curve is calculated through a simple battery charge and discharge model, and its strategy is continuously optimized later. Define an actor model to learn and select the charge and discharge strategy. Initially, you can use random or existing strategies. At the same time, define a training reward model to evaluate the pros and cons of each charge and discharge trajectory, provide feedback, and help the actor improve the strategy. After training and verification, the reward model and strategy optimization algorithm are adjusted according to the verification results to further improve the effectiveness of the strategy, and ultimately a better battery charging and discharging strategy model can be obtained.

[0124] This application uses an agent to imitate and learn a fixed initial actor model after the suboptimal demonstrator behavior, and copies the model to be used as an actor model for reinforcement learning. The entire trajectory is scored by a trained reward model to guide the training of the actor model, and the coefficient of the KL divergence can be used to adjust the closeness of the actor model to the initial model. The training of the reward model is simplified, and the data flow required to train the reward model is reduced. The reinforcement learning training is stabilized because the accuracy of the reward model evaluation is improved. The actor obtained by training can also be adjusted by adjusting the relevant hyperparameters to determine whether it is close to the initial actor. In the case of large models, video memory can be saved and training speed can be accelerated.

[0125] In order to better understand the solution of this application, the following is an overall introduction to the reinforcement learning process of this application:

[0126] 1. Perform agent imitation learning training based on the existing suboptimal demonstration data, so that the agent can learn the demonstrator's behavior strategy and prepare for the subsequent training of the reward model.

[0127] 2. Design a reward model. The reward model is trained based on the idea of ​​comparative learning intention. First use the actor model trained in the previous step, then sample multiple trajectories for preference sorting, and use the Bradley-Terry and Luce-Shephard preference models to model the reward model. However, this application does not use rewards with state-action granularity, but uses a reward model with trajectory granularity. Based on the selection of good and bad pairs for the entire trajectory, the trajectory pairs enter the neural network for training together, and then the entire trajectory is scored. This training method is relatively simple and convenient, and the training is more stable.

[0128] 3. Improve reinforcement learning algorithms.

[0129] The trained reward model can be understood as an actor-based preference model, that is, accuracy can be guaranteed without deviating too far from the initial actor, so the actor's update must be restricted so that it is updated near the initial actor. Therefore, the proximal policy optimization algorithm is a suitable method, but the original proximal policy optimization algorithm still cannot ensure that the actor is distributed similarly to the initial actor. For this reason, this application gives a deviation penalty term based on the reward output by the reward model, so that the reward model can correctly evaluate the generated samples of the actor.

[0130] Furthermore, in order to alleviate the problem of slow training of the Proximal Policy Optimization (PPO) algorithm and the reward model being the reward for the entire trajectory, this application changes the algorithm to a greedy algorithm, which not only speeds up the training speed and facilitates program design, but also alleviates the pressure on the video memory when the model used is large.

[0131] 4. Large model contrast reinforcement training. Since this algorithm draws on the off-policy reinforcement learning method, when a large model occupies too much video memory and the training speed is slow during on-policy contrast reinforcement learning training, you can generate contrast trajectories in batches, score them, and then iterate the training according to this algorithm in rounds, which can also achieve good results. If the reward model is difficult to train, you can train in batches.

[0132] 5. Parameter adjustment details. Compared with the original PPO parameters, this algorithm needs to adjust fewer hyperparameters, namely β, γ, and α. β and α are hyperparameters that control the degree of deviation between the reinforcement learning strategy and the original strategy. If it deviates too far from the original strategy, it can be appropriately increased. However, if the training speed is too slow, β can be appropriately reduced. If instability or overfitting occurs during training, α can be appropriately reduced. γ is a regularization degree term. If instability or overfitting occurs during training, it can be appropriately increased. If the training speed is too slow, it can be appropriately reduced.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a reinforcement learning system for implementing the reinforcement learning method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more reinforcement learning system embodiments provided below can refer to the limitations of the reinforcement learning method above, and will not be repeated here.

[0134] In an exemplary embodiment, Figure 3 As shown, a reinforcement learning system is provided, including: a suboptimal data acquisition module 301, an initial strategy determination module 302, a behavior sequence determination module 303, a reward model training module 304, a loss determination module 305 and a reinforcement learning module 306.

[0135] The suboptimal data acquisition module 301 is used to acquire suboptimal demonstration data in a target application scenario.

[0136] The initial strategy determination module 302 is used to obtain an initial strategy by having an intelligent agent perform imitation learning based on the suboptimal demonstration data.

[0137] The behavior sequence determination module 303 is used to determine multiple behavior sequences based on the initial strategy.

[0138] The reward model training module 304 is used to iteratively train the reward model according to multiple behavior sequences to obtain a trained reward model.

[0139] The loss determination module 305 is used to introduce a regularization term into the contrastive reinforcement learning algorithm of KL divergence based on the trained reward model to obtain a contrastive reinforcement learning loss function with a regularization term.

[0140] The reinforcement learning module 306 is used to perform reinforcement learning according to multiple behavior sequences, based on the trained reward model and the contrast reinforcement learning loss function with a regularization term, so as to obtain the final reinforcement learning strategy in the target application scenario.

[0141] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store suboptimal demonstration data under the target application scenario. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a reinforcement learning method is implemented.

[0142] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0143] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0144] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0146] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0147] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A reinforcement learning method, characterized in that: The reinforcement learning method comprises: Obtain suboptimal demonstration data for the target application scenario; The intelligent agent performs imitation learning based on the suboptimal demonstration data to obtain an initial strategy; Based on the initial strategy, determining a plurality of action sequences; Iteratively train the reward model according to multiple behavior sequences to obtain a trained reward model; Based on the trained reward model, a regularization term is introduced into the contrastive reinforcement learning algorithm of KL divergence to obtain the contrastive reinforcement learning loss function with a regularization term. According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrastive reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario.

2. The reinforcement learning method according to claim 1, characterized in that: The following formula is used for imitation learning: in, is the parameter adjusted in the imitation learning process, E is the expectation, (s, a) is the state-action pair, s is the state, a is the action, and B init is the distribution of strategies based on suboptimal demonstration data, is the loss function for supervised learning, For state s, the parameters The actions generated by the defined strategy, To minimize the loss function during the learning process The value of The defined policy is the initial policy.

3. The reinforcement learning method according to claim 1, characterized in that: The loss function when training the reward model is: Among them, loss(θ) is the loss function value, E is the expectation, x is the starting state, y1 and y2 are two different behavior sequences, D is the data distribution of the initial strategy, and r θ ( ) is the reward model, θ is the parameter adjusted during the reward model training process, and σ( ) is the sigmoid function, which is used to convert the score value into a probability value ranging from 0 to 1.

4. The reinforcement learning method according to claim 1, characterized in that: The loss function of contrastive reinforcement learning with regularization is: Among them, L(π φ ; π Initial ) is the contrast reinforcement learning loss function value with regularization term, π φ To compare the strategies generated during reinforcement learning, π Initial is the initial strategy, E is the expectation, x is the starting state, y1 and y2 are two different behavior sequences, π(φ k ) is φ k The strategy for parameter determination, r θ ( ) is the reward model, ε1 is the intermediate quantity corresponding to y1, ε2 is the intermediate quantity corresponding to y2, β is the Lagrange multiplier of the KL divergence term, π(y1|x) is the probability of selecting the behavior sequence y1 in the starting state x according to the strategy π, π(y2|x) is the probability of the behavior sequence y2 in the starting state x according to the strategy π, Initial (y1|x) is the initial strategy π Initial The probability of selecting the action sequence y1 in the starting state x, π Initial (y2|x) is based on the initial strategy π Initial The probability of selecting the behavior sequence y2 in the starting state x, σ( ) is the sigmoid function, which is used to convert the score value into a probability value with a range of 0 to 1, γ and α are the weight coefficients of the regularization term, r θ (y1|x) is the reward for selecting the behavior sequence y1 in the starting state x, r θ (y2|x) is the reward for selecting the behavior sequence y2 in the starting state x, π θ (y1|x) is based on the strategy π θ The probability of selecting the action sequence y1 in the starting state x, π θ (y2|x) is based on the strategy π θ The probability of choosing action sequence y2 given starting state x.

5. The reinforcement learning method according to claim 1, characterized in that: The target application scenario is a vehicle lane-changing trajectory planning scenario; the suboptimal demonstration data is a suboptimal lane-changing trajectory; the multiple behavior sequences are multiple trajectory sequences obtained by randomly sampling the suboptimal lane-changing trajectory; and the final reinforcement learning strategy is the optimal lane-changing strategy; According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: Select two trajectory sequences from multiple trajectory sequences, and use the trained reward model to determine the reward values ​​of the two trajectory sequences; According to the reward values ​​of every two trajectory sequences in multiple trajectory sequences, reinforcement learning is performed based on the contrastive reinforcement learning loss function with a regularization term to obtain the optimal lane changing strategy.

6. The reinforcement learning method according to claim 5, characterized in that: Each trajectory sequence is in the form of (s i ,a i ,s i+1 ,a i+2 ,…s i+n-1 ,a i+n-1 ), where s i is the i-th state, a i is the ith action, n is the length of the trajectory sequence, the state includes the distance between the vehicle and the centerline of the lane, the speed of the vehicle and the heading angle of the vehicle, and the action includes the lateral acceleration of the vehicle and the front wheel turning angle of the vehicle.

7. The reinforcement learning method according to claim 1, characterized in that: The target application scenario is a vehicle driving scenario; the initial strategy is an initial driving strategy; the multiple behavior sequences are multiple trajectory sequences obtained by randomly sampling the initial driving strategy; the final reinforcement learning strategy is the optimal driving strategy; According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: Select two trajectory sequences from multiple trajectory sequences, and use the trained reward model to determine the reward values ​​of the two trajectory sequences; According to the reward values ​​of every two trajectory sequences in multiple trajectory sequences, reinforcement learning is performed based on the contrastive reinforcement learning loss function with a regular term to obtain the optimal driving strategy.

8. The reinforcement learning method according to claim 1, characterized in that: The target application scenario is a large model alignment scenario; the initial strategy is a pre-trained large model; the multiple behavior sequences are multiple word-unit sequences generated after inputting multiple prompt words into the pre-trained large model respectively; wherein each prompt word generates two word-unit sequences; the final reinforcement learning strategy is the aligned large model; According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: Select two word-gram sequences from multiple word-gram sequences, and use the trained reward model to determine reward values ​​of the two word-gram sequences; According to the reward value of every two word-gram sequences in multiple word-gram sequences, reinforcement learning is performed based on the contrastive reinforcement learning loss function with a regular term to obtain the aligned large model.

9. The reinforcement learning method according to claim 1, characterized in that: The target application scenario is a battery charge and discharge strategy optimization scenario; the initial strategy is an initial battery charge and discharge strategy curve; the multiple behavior sequences are multiple charge and discharge trajectories obtained by randomly sampling the initial battery charge and discharge strategy curve; wherein the charge and discharge trajectory is a curve of the entire process from the start of battery charging to the end of battery discharging, and each charge and discharge trajectory includes a state space and an action space, the state space is the current state of the battery, and the action space includes a charging strategy and a discharging strategy; the final reinforcement learning strategy is the optimal battery charge and discharge strategy; According to multiple behavior sequences, reinforcement learning is performed based on the trained reward model and the contrast reinforcement learning loss function with a regularization term to obtain the final reinforcement learning strategy in the target application scenario, specifically including: Select two charging and discharging trajectories from multiple charging and discharging trajectories, and use the trained reward model to determine the reward values ​​of the two charging and discharging trajectories; According to the reward values ​​of every two charging and discharging trajectories among multiple charging and discharging trajectories, reinforcement learning is performed based on the contrastive reinforcement learning loss function with a regular term to obtain the optimal battery charging and discharging strategy.

10. A reinforcement learning system, applied to the reinforcement learning method according to any one of claims 1 to 9, characterized in that: The reinforcement learning system comprises: A suboptimal data acquisition module, used to acquire suboptimal demonstration data in a target application scenario; An initial strategy determination module, used for obtaining an initial strategy by an intelligent agent performing imitation learning according to the suboptimal demonstration data; A behavior sequence determination module, used to determine a plurality of behavior sequences based on the initial strategy; A reward model training module, used to iteratively train the reward model according to multiple behavior sequences to obtain a trained reward model; A loss determination module, which is used to introduce a regularization term into the contrastive reinforcement learning algorithm of KL divergence based on the trained reward model to obtain a contrastive reinforcement learning loss function with a regularization term; The reinforcement learning module is used to perform reinforcement learning according to multiple behavior sequences, based on the trained reward model and the contrast reinforcement learning loss function with a regularization term, so as to obtain the final reinforcement learning strategy in the target application scenario.

Images