A reinforcement learning intelligent decision-making method based on a multi-prior strategy

Abstract

The application relates to the technical field of machine learning and reinforcement learning, and discloses a reinforcement learning intelligent decision-making method based on multiple priori strategies, which solves the decision-making difficulty problem caused by sparse rewards, low sample utilization rate and environment overfitting when the existing technology uses reinforcement learning for intelligent decision-making. The method uses a target strategy network and a priori strategy to guide an agent to make decisions, and puts the interaction data generated by the decisions into a replay buffer; then data sampling is performed on the replay buffer to obtain a training set for target strategy network training; then a comprehensive loss function is calculated according to the training data, and the target strategy network is updated by using the gradient descent method; finally, the sampling probability of the training data in the replay buffer is updated; in the intelligent decision-making process, the environment state observed by the agent is input into the target strategy network, an action is executed according to the network output, and state transition is completed, and then the above observation-decision process is repeated until the decision-making task is completed.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and reinforcement learning, specifically to a reinforcement learning intelligent decision-making method based on multiple prior policies. Background Technology

[0002] Reinforcement learning, as an algorithmic framework for solving sequential decision-making problems, has been widely applied in many popular fields such as autonomous driving, robot control, and competitive gaming. It utilizes the interaction between an agent and its environment, learning policies through continuous trial and error during the interaction process. With the development of artificial intelligence methods and the rapid improvement of hardware computing power, reinforcement learning models have become increasingly powerful, continuously improving their problem-solving abilities and achieving excellent results in many real-world decision-making tasks. However, due to problems such as sparse rewards, low sample utilization, and overfitting to the environment, reinforcement learning suffers from low learning efficiency in certain scenarios, leading to poor decision-making performance.

[0003] Reinforcement learning methods based on multiple prior policies refer to intelligent decision-making through reinforcement learning based on multiple existing prior policies, thus avoiding the process of learning policies from scratch and making decision-making more efficient. In 2016, Rusu et al. successfully applied knowledge distillation techniques to reinforcement learning, proposing a method for rapidly learning the target policy based on multiple prior policies, effectively achieving rapid decision-making in multiple decision tasks, and subsequently, several other similar methods emerged. However, these methods all have certain limitations, namely, when the target decision task changes, the target policy obtained by the method is difficult to guarantee the quality of the decision, thus failing to meet the intelligent decision-making needs in complex and ever-changing real-world scenarios. Summary of the Invention

[0004] Based on the DQN (Deep Q-Network) algorithm, this invention proposes a reinforcement learning intelligent decision-making method based on multiple prior policies, which solves the problems of reward sparsity, low sample utilization and environment overfitting encountered in existing intelligent decision-making using reinforcement learning.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A reinforcement learning-based intelligent decision-making method based on multiple prior policies includes the following steps:

[0007] A. Training process:

[0008] A1. Initialize the target policy network, import multiple existing prior policies, and initialize the corresponding replay buffer for each prior policy;

[0009] A2. Select a prior policy in sequence, use the target policy network and the prior policy to guide the agent to make decisions, and put the interaction data (s,a,r,s′) generated by the decision into the replay buffer;

[0010] A3. Sample data from the replay buffer to obtain a training set {s} consisting of states, actions, rewards, and the next state. i ,a i ,r i ,s i+1}, used for training the target policy network;

[0011] A4. Obtain the prediction term based on the behavior network in the target policy network, obtain the supervision term based on the target network and the prior policy in the target policy network, and combine the distillation loss of the prior policy to calculate the loss function. Then, update the target policy network using the gradient descent method.

[0012] A5. Update the sampling probability of the training data used and put it back into the replay buffer;

[0013] B. Intelligent decision-making process:

[0014] The agent observes the current environmental state S and inputs it into the target policy network. Based on the network output, it selects an action, executes the action, and completes the state transition. This process is repeated until the decision-making task is completed.

[0015] The beneficial effects of this invention are:

[0016] (1) By effectively utilizing prior strategies in the target decision-making task, the reinforcement learning decision-making model can be trained quickly while reducing training costs.

[0017] (2) During the distillation of the prior strategy, the target agent is still allowed to interact with the current environment, which makes full use of the approximation capability of the target decision network and can still effectively utilize the prior strategy when the target task changes.

[0018] (3) During the sampling of training data, the priority of training data can be set by using the idea of ​​prior experience playback, which can effectively improve the data utilization rate. At the same time, monitoring the reward distribution in the environment can enable the rapid correction of prior strategies. Attached Figure Description

[0019] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating the decision-making interaction process during model training in an embodiment of the present invention.

[0021] Figure 3 This is a flowchart of the loss calculation process during model training in an embodiment of the present invention;

[0022] Figure 4 This is a flowchart illustrating the sampling probability update process of training data during model training in this embodiment of the invention.

[0023] Figure 5 This is a flowchart of the intelligent decision-making process of the model in an embodiment of the present invention. Detailed Implementation

[0024] This invention aims to propose a method for effectively utilizing prior policies in reinforcement learning for intelligent decision-making, addressing the problems of reward sparsity, low sample utilization, and environment overfitting encountered in existing reinforcement learning decision-making processes. The method includes a training process and an intelligent decision-making process. During training, firstly, the prior policy is distilled in the target policy network to achieve rapid transfer of the prior policy. Secondly, the prior policy is used to guide the action of the target agent, helping it interact with the environment and learn the policy through reward feedback. Subsequently, using the idea of ​​priority experience replay, the priority of the training data is set according to the TD-Error, and the sampling probability of the data is adjusted according to the priority. This allows for rapid policy correction when the prior policy performs poorly, while improving sample utilization. Finally, the parameters of the target policy network are updated using gradient descent, combining the policy distillation loss and the agent training loss. During the intelligent decision-making process, the state observed by the agent is input into the target policy network. Action decisions are made based on the network output, and the state is updated after each action is executed, until the decision is complete.

[0025] Example:

[0026] In this embodiment, the reinforcement learning intelligent decision-making method based on multiple prior strategies includes two parts: a training process and an intelligent decision-making process. Taking image navigation task as the application scenario, the following is a detailed description with reference to the accompanying drawings:

[0027] See Figure 1 The training process includes:

[0028] S1. Initialize the target policy network. The target policy network mainly consists of two parts: a behavior network and a target network. The two networks have identical structures, each consisting of 3 convolutional layers and 2 fully connected layers. Both networks are initialized using randomly set network parameters. The network input is the state information observed by the agent, and the network output is the state-action value vector, i.e., the Q-vector, composed of all executable actions.

[0029] S2. Import the existing N prior policy models. The network structure of the prior policy models is the same as that of the target policy network. Initialize a replay buffer for each prior policy.

[0030] S3. Select a priori strategy Where, π T The prior policy is referred to as 't', which represents the selected prior policy number, ranging from 1 to N. Under the combined effect of the prior policy and the target policy, the agent is guided to interact with the environment. The interaction process can be found in [reference needed]. Figure 2 It mainly consists of the following steps:

[0031] S31. Based on the image information currently observed by the agent and the global navigation view, the image data is preprocessed, mainly including image cropping, downsampling, background erasure, and conversion to grayscale. The RGB pixel matrix of the preprocessed image information is used as the current state s.

[0032] S32. Define the action selection policy π(a|s) and select the substantive guiding policy for decision-making. The policy refers to the logical mapping from the agent's observed state s to action a. π(a|s) is defined as follows:

[0033]

[0034] Where, π T (a|s) represents action selection based on a prior policy, π S (a|s) represents the action selection based on the target policy network. Furthermore, β is a parameter that gradually changes during training; it is set to a large value in the early stages of training and then gradually decreases thereafter, representing the gradual transition of the agent's action selection power from the prior policy to the target policy.

[0035] S33. If decision-making is based on a priori strategy, then π is used. T (a|s) guides the choice of action. π T (a|s) is defined as follows:

[0036]

[0037] in, Let A represent the value function network of the chosen t-th prior policy, and let A represent the action space that the agent can execute in the current state. π T (a|s) is defined as the agent choosing the action that outputs the maximum value in the t-th prior policy network;

[0038] S34. If decision-making is based on the target strategy, then π is used. S (a|s) guides the choice of action. π S (a|s) is defined as follows:

[0039]

[0040] Where ∈ is a gradually decreasing parameter, representing a gradual decrease in the randomness of action selection, Q S Let A represent the target policy value function network, |A| represent the action space size, and π represent the value function network. S The definition of (a|s) indicates the probability that the action with the largest output and other actions in the target policy network will be selected for execution.

[0041] S35. In the current state s, the agent performs the action a selected according to π(a|s), obtains the reward r, and observes the next state s′.

[0042] S4. Set the sampling probability of the interaction data (s,a,a,s′) to the current maximum value and put it into the replay buffer corresponding to the current prior policy.

[0043] S5. After the replay buffer stores a certain amount of interactive data, data is sampled from the replay buffer according to the sampling probability P of the data to form a training set minibatch{s,a,r,s′}.

[0044] S6. Train the model using the interaction data in the training set minibatch{s,a,r,s′} and calculate the loss. For details on loss calculation during training, please refer to [link to training documentation]. Figure 3 It mainly consists of the following steps:

[0045] S61. Calculating the reinforcement learning loss mainly involves the following steps:

[0046] S611, Transfer the state s from the training data i As input, the predicted Q-value of the action 'a' is obtained from the behavior network in the target policy network:

[0047]

[0048] in, The parameters represent the behavior network parameters in the target policy network; Q represents the predicted state-action value.

[0049] S612, The next state s in the training data i+1 As input, based on the target network in the target policy network and the obtained environmental reward r i And by combining the prior policy network, the supervised term Y is calculated. i :

[0050]

[0051] in, represents the target network parameters in the target policy network; γ represents the reward discount factor, indicating the agent's preference for current and long-term rewards; a * The selection method is determined by both the prior policy and the target policy, and is defined as follows:

[0052]

[0053] That is, the action that yields the greatest expected return after weighted summation is selected to estimate the supervision term Y. i ,in This represents the behavior network parameters in the target policy network. This represents the state-action value predicted by the target policy network. This represents the target network parameters in the prior policy network. The state-action value is represented by the prior policy network, and α represents the importance coefficient of the prior policy relative to the target policy.

[0054] S613. Based on the predicted Q value and the supervision term Y i Calculate reinforcement learning loss

[0055]

[0056] S62. Calculate the strategy distillation loss, which mainly involves the following steps:

[0057] S621, Transfer the state s from the training data i As input, based on the behavior network in the target policy network, the Q-value vector consisting of all executable actions is obtained.

[0058] S622, Transfer the state s from the training data i As input, based on the behavior network in the prior policy network, the Q-value vector consisting of all executable actions is obtained.

[0059] S623, according to and Calculating distillation loss using KL divergence strategy

[0060]

[0061] Where τ represents the temperature coefficient of the strategy distillation; softmax refers to the normalized exponential function;

[0062] S63, Based on reinforcement learning loss and strategy distillation loss Calculate total loss

[0063]

[0064] Among them, the importance coefficient ω of the prior policy is used to balance the importance of the two parts of the loss function; |A| represents the size of the action space of the agent's executable actions;

[0065] S7, Regarding the total loss Using gradient descent to analyze the network parameters θ of the target policy network S The parameters are updated so that the loss function gradually decreases along the gradient descent direction until it converges.

[0066] S8. Update the sampled probabilities of the training data used. See also: Figure 4 It mainly consists of the following steps:

[0067] S81. Calculate the TD-Error of the training data, where TD-Error is defined as follows:

[0068]

[0069] S82. Based on the TD-Error value δ of the i-th data sample i Calculate the corresponding sampling probability P(i), which is defined as follows:

[0070]

[0071] Where z is a small positive number, used to ensure that some samples with TD-Error of 0 also have a chance to be sampled; μ is a hyperparameter that can be used to adjust the priority of TD-Error appropriately without changing the monotonicity of the priority; j corresponds to i and is the data sample number, i is specific and j is general, and its value is from 1 to M, where M is the amount of data (s,a,r,s′) in the training set. This step is to sum the weights of all samples.

[0072] S83, Transfer the training data (s) i ,a i ,r i ,s i+1 The sampling probability of ) is updated to P(i);

[0073] After the training data in the minibatch is used up (S9), return to step S3 to begin using the next prior policy to guide the agent's decision-making and training, thus achieving the network parameter θ. S Update;

[0074] See Figure 5 The intelligent decision-making process for navigation includes:

[0075] S1. The agent observes its current environment and obtains corresponding image information;

[0076] S2. Based on the currently observed image information and the global navigation view, preprocess the image data. The preprocessing process mainly includes image cropping, downsampling, background erasure, and conversion to grayscale. The preprocessed RGB image is considered as the current state s.

[0077] S3. Input the state s into the trained policy model to obtain the Q-values ​​corresponding to all executable actions;

[0078] S4. Select and execute the action with the largest Q value. The agent moves and observes new image information.

[0079] S5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

[0080] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A reinforcement learning intelligent decision-making method based on multiple prior strategies, characterized in that, This includes the training process and the intelligent decision-making process. The training process specifically includes the following steps: S1. Initialize the target policy network. The target policy network mainly consists of two parts: the behavior network and the target get network. The two networks have the same structure, both consisting of 3 convolutional layers and 2 fully connected layers. The network initialization is completed by randomly setting the network parameters. The network input is the state information observed by the current agent, and the network output is the state-action value vector, i.e., the Q vector, which is composed of all executable actions. S2. Import the existing N prior policy models. The network structure of the prior policy models is the same as that of the target policy network. Initialize a replay buffer for each prior policy. S3. Select a priori strategy Where, π T The prior policy is referred to as t, which represents the selected prior policy number and takes a value from 1 to N. Under the combined effect of the prior policy and the target policy, the agent is guided to interact with the environment, which involves the following steps: S31. Based on the image information currently observed by the agent and the global navigation view, the image data is preprocessed, mainly including image cropping, downsampling, background erasure, and conversion to grayscale. The RGB pixel matrix of the preprocessed image information is used as the current state s. S32. Define the action selection policy π(a|s) and select the substantive guiding policy for decision-making. The policy refers to the logical mapping from the agent's observed state s to action a; π(a|s) is defined as follows: Where, π T (a|s) represents action selection based on a prior policy, π S (a|s) represents the action selection based on the target policy network. β is a parameter that changes gradually during the training process. It is set to a large value in the early stage of training and then gradually decreases thereafter, which means that the agent's action selection right gradually transitions from the prior policy to the target policy. S33. If decision-making is based on a priori strategy, then π is used. T (a|s) guides action selection; π T (a|s) is defined as follows: in, Let A represent the value function network of the chosen t-th prior policy, and let A represent the action space that the agent can execute in the current state. π T (a|s) is defined as the agent choosing the action that outputs the maximum value in the t-th prior policy network; S34. If decision-making is based on the target strategy, then π is used. S (a|s) guides action selection; π S (a|s) is defined as follows: Where ∈ is a gradually decreasing parameter, representing a gradual decrease in the randomness of action selection, Q S This represents the target policy value function network, where |A| represents the action space size; S35. In the current state s, the agent performs the action a selected according to π(a|s), obtains the reward r, and observes the next state s′. S4. Set the sampling probability of the interaction data (s,a,r,s′) to the current maximum value and put it into the replay buffer corresponding to the current prior policy. S5. After the replay buffer stores a certain amount of interactive data, data is sampled from the replay buffer according to the sampling probability P of the data to form a training set minibatch{s,a,r,s′}. S6. Train the model using the interaction data in the training set minibatch{s,a,r,s′} and calculate the loss. This involves the following steps: S61. Calculating the reinforcement learning loss mainly involves the following steps: S611, Transfer the state s from the training data i As input, the predicted Q-value of the action 'a' is obtained from the behavior network in the target policy network: in, The parameters represent the behavior network parameters in the target policy network; Q represents the predicted state-action value. S612, The next state s in the training data i+1 As input, based on the target network in the target policy network and the obtained environmental reward r i And by combining the prior policy network, the supervised term Y is calculated. i : in, represents the target network parameters in the target policy network; γ represents the reward discount factor, indicating the agent's preference for current and long-term rewards; a * The selection method is determined by both the prior policy and the target policy, and is defined as follows: That is, the action that yields the greatest expected return after weighted summation is selected to estimate the supervision term Y. i ,in This represents the behavior network parameters in the target policy network. This represents the state-action value predicted by the target policy network. This represents the target network parameters in the prior policy network. The state-action value is represented by the prior policy network, and α represents the importance coefficient of the prior policy relative to the target policy. S613. Based on the predicted Q value and the supervision term Y i Calculate reinforcement learning loss S62. Calculate the strategy distillation loss, which mainly involves the following steps: S621, Transfer the state s from the training data i As input, based on the behavior network in the target policy network, the Q-value vector consisting of all executable actions is obtained. S622, Transfer the state s from the training data i As input, based on the behavior network in the prior policy network, the Q-value vector consisting of all executable actions is obtained. S623, according to and Calculating distillation loss using KL divergence strategy Where τ represents the temperature coefficient of the strategy distillation; softmax refers to the normalized exponential function; S63, Based on reinforcement learning loss and strategy distillation loss Calculate total loss Among them, the importance coefficient ω of the prior policy is used to balance the importance of the two parts of the loss function; |A| represents the size of the action space of the agent's executable actions; S7, Regarding the total loss Using gradient descent to analyze the network parameters θ of the target policy network S The parameters are updated so that the loss function gradually decreases along the gradient descent direction until it converges. S8. Update the sampling probability of the training data used, which involves the following steps: S81. Calculate the TD-Error of the training data, where TD-Error is defined as follows: S82. Based on the TD-Error value δ of the i-th data sample i Calculate the corresponding sampling probability P(i), which is defined as follows: with:p i =δ i +z Where z is a small positive number, used to ensure that some samples with TD-Error of 0 also have a chance to be sampled; μ is a hyperparameter used to adjust the priority of TD-Error appropriately without changing the monotonicity of the priority; j corresponds to i and is the data sample number, i is specific and j is general, and its value is from 1 to M, where M is the amount of data (s,a,r,s′) in the training set. This step is to sum the weights of all samples. S83, Transfer the training data (s) i ,a i ,r i ,s i+1 The sampling probability of ) is updated to P(i); After the training data in the minibatch is used up (S9), return to step S3 to begin using the next prior policy to guide the agent's decision-making and training, thus achieving the network parameter θ. S Update; The intelligent decision-making process includes the following steps: SS1: The agent observes its current environment and obtains corresponding image information; SS2. Based on the currently observed image information and global navigation view, preprocess the image data. The preprocessing process mainly includes image cropping, downsampling, background erasure, and conversion to grayscale. The preprocessed RGB image is regarded as the current state s. SS3. Input the state s into the trained policy model to obtain the Q-values ​​corresponding to all executable actions; SS4: Select and execute the action with the highest Q value. The agent moves and observes new image information. SS5. Repeat steps SS1 to SS4 until the navigation task is completed and the intelligent decision-making ends.

Images