A reinforcement learning intelligent decision-making method based on priority experience replay

Abstract

The application relates to the technical field of machine learning and reinforcement learning, and discloses a reinforcement learning intelligent decision-making method based on priority experience replay, aiming at the problem that an agent in existing reinforcement learning has difficulty in learning when facing a complex environment, and proposes a priority experience replay method based on a weight scoring network, which guides the agent to quantize the priority of data in a replay buffer, so that the agent can more efficiently learn from historical experience data. In the training process, the method calculates a comprehensive loss function according to training data, and updates a target policy network by using a gradient descent method to improve the decision-making performance of the target policy network. Through the reinforcement learning intelligent decision-making method based on priority experience replay, the application can effectively solve the problem that an agent in existing reinforcement learning has difficulty in learning when facing a complex environment, and improve the decision-making efficiency and accuracy of the agent.

Description

Technical Field

[0001] This invention belongs to the fields of machine learning and reinforcement learning, and particularly relates to a reinforcement learning intelligent decision-making method based on priority experience replay. Background Technology

[0002] One of the main goals of artificial intelligence is to endow intelligent agents with the ability to perform diverse and complex tasks. In the current field of machine learning, mainstream research often focuses on pre-collected or carefully constructed datasets with explicit labels. This static data provides the foundation for model training, enabling machine learning algorithms to learn and identify specific patterns or rules. However, reinforcement learning, as a prominent representative of a closed-loop learning paradigm, is unique in its methodology, utilizing dynamically generated data and real-time feedback labels to guide the entire learning process. Through continuous interaction with the environment, reinforcement learning algorithms can gradually optimize their behavioral strategies through trial and error, achieving self-improvement and refinement.

[0003] Reinforcement learning has achieved remarkable success in numerous sequential decision-making problems. Whether in board games, video games, or robot control tasks, reinforcement learning has demonstrated powerful problem-solving capabilities. However, it is worth noting that traditional reinforcement learning methods often simply use historical trajectory data directly for algorithm training, neglecting the potential application value inherent in this data.

[0004] In fact, these historical trajectory data not only record the agent's behavior during interaction with the environment, but also contain rich environmental information and the agent's learning state. In-depth mining and analysis of this data can not only provide richer training signals for reinforcement learning algorithms, but also reveal the agent's decision-making mechanisms and learning patterns in the task-solving process. Therefore, how to more effectively utilize historical trajectory data will become an important research direction in the field of reinforcement learning in the future.

[0005] Reinforcement learning has achieved great success in sequential decision problems. While on-policy approaches are effective, using historical experience yields better sample utilization, which is crucial in environments where samples are hard to obtain, and also further conserves resources.

[0006] Experience replay enables data reuse and is widely used in off-policy reinforcement learning. Prioritized Memory Replay (PER) considers samples with larger temporal differences in the replay buffer to be more important because the agent has a lower level of knowledge in these areas. Based on PER, researchers have proposed various sampling strategies for priority sampling, such as Prioritized Sequence Replay (PSER) and Enhanced Recent Memory (ERE). On the other hand, Distribution Correction (DisCor) argues that the data distribution updated by Approximate Dynamic Programming (ADP) directly affects the learned solution, requiring the replay buffer to be reweighted to a distribution with corrective feedback. This is because the "bootstrapping" of reinforcement learning leads to inaccurate Q-value predictions, resulting in a lack of corrective feedback in reinforcement learning. However, both algorithms only consider their individual factors and do not combine them. Summary of the Invention

[0007] Based on the Deep Q-Network algorithm, this invention proposes a reinforcement learning intelligent decision-making method based on priority experience replay, which solves the problems of low data utilization in the replay buffer and slow performance improvement in existing reinforcement learning methods.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0009] A reinforcement learning-based intelligent decision-making method based on priority-based experience replay includes a training process and a decision-making process, specifically comprising the following steps:

[0010] Training process:

[0011] A1: Model Initialization; The model includes a target policy network and a weight scoring network. Network initialization is completed by randomly setting network parameters. The network inputs are all the state information observed by the current agent. The output of the target policy network is the state-action value vector composed of all executable actions. The output of the policy scoring network is the priority weight value of the current sample. Specifically, the target policy network consists of 3 convolutional layers and 3 fully connected layers. The first 3 layers are convolutional layers and the last 3 layers are fully connected layers. The network layer structure of the weight scoring network is the same as that of the target policy network.

[0012] A2: Environmental interaction data collection. The agent's target policy network guides the agent to interact with the environment, specifically including the following steps:

[0013] A21: Acquire environmental information; Encode the image information of the environment, acquire it, and perform preprocessing.

[0014] A22: Input the image information into the target policy network, and obtain the probability distribution of each action through the forward propagation of the neural network. Here, the sum of the probabilities of all actions is 1.

[0015] A23: The obtained probability distribution is sampled to obtain the specific implementation action, which is the next action that the agent needs to perform;

[0016] A24: After the agent completes the current action a, the environment provides a reward feedback r for the action and transitions from state s to the next state s'. The transitioned state is the environmental information that the agent will input for the next decision.

[0017] A3: Use a weighted scoring network to set the priority weights of the interactive data (s,a,r,s') and put them into the replay buffer. The specific steps are as follows:

[0018] S31, Sample data; Sample data is a quadruple (s, a, r, s') formed after an agent interacts with the environment.

[0019] S32, Data Calculation: Input the environmental state information s from the sample data into the weight scoring network, and obtain the priority weight of the sample through forward propagation.

[0020] S33. Calculate the new weight values ​​for the samples, using the formula shown below, where w is the score output by the weight scoring network:

[0021] w label =0.9*w + 0.1*r

[0022] S34. Update the parameters of the weight scoring network. The loss function is as follows:

[0023] L(θ)=(w label -w) 2

[0024] A4: After the replay buffer stores a certain amount of sample data, data is sampled from the replay buffer according to the priority weight of the data to conduct one round of training of the target policy network; the priority weight of the data includes the priority weight of the interaction data generated in this iteration and the importance weight of the replay buffer after bias correction.

[0025] A5: Calculate the importance weights; use the importance weights to correct for bias, calculated as follows:

[0026]

[0027] Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of bias correction; where wj is the original weight calculated by the weighted scoring network, and wi is the corrected weight.

[0028] A6: Calculate the loss function and update the target policy network parameters. The formula for calculating the loss function is shown below:

[0029]

[0030] Where D represents the replay buffer, Q represents the target policy network, and θ represents the set of network parameters. Let E represent the parameters of the target policy network, E represent the expected value of the sample data (s,a,s',r) following the distribution of D, and γ represent the discount coefficient. For the calculated loss value, the network parameters of the target policy network are updated using the gradient descent method, so that the loss function gradually decreases along the gradient descent direction until it converges.

[0031] A7: When the selected training data is exhausted, if the set policy objective is not achieved, return to step A2 and continue the above steps; if the policy performance meets the requirements, end the training.

[0032] Decision-making process:

[0033] B1: The agent observes its current environment and acquires image information;

[0034] B2: Preprocess the image data based on the currently observed image information and the global navigation view;

[0035] B3: Input the state s into the trained model to obtain the probabilities of all executable actions.

[0036] B4: Sample the action to obtain the action performed by the agent this time. At the same time, the agent receives the reward value given by the environment and observes the changed environment.

[0037] B5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

[0038] Compared with existing technologies, the beneficial effects of the present invention are mainly reflected in the following aspects:

[0039] First, by guiding the selection of higher-quality samples through priority weights, this invention enables faster convergence of reinforcement learning decision models. This improvement not only significantly enhances the training efficiency of the model but also greatly increases the computer processing speed, reduces the time cost required for training, and saves users valuable resources and time.

[0040] Secondly, this method is not only applicable to specific off-policy methods, but also has broad applicability, and can be applied to various off-policy scenarios. This characteristic makes the invention more flexible and applicable in practical applications, and can meet the needs of different scenarios.

[0041] Finally, this invention effectively solves the problem of convergence difficulty in reinforcement learning strategies by improving sample utilization and algorithm performance. In terms of algorithm performance, this invention has significant advantages, providing users with a better experience and higher efficiency.

[0042] In summary, this invention has significant beneficial effects in the field of reinforcement learning. It not only improves the training efficiency and sample utilization of the model, but also solves the problem of difficulty in convergence of reinforcement learning strategies, bringing new breakthroughs to the development of related fields. Attached Figure Description

[0043] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention;

[0044] Figure 2 This is a flowchart illustrating the decision-making interaction process during model training in an embodiment of the present invention.

[0045] Figure 3 This is a flowchart of the priority weight calculation process during model training in an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of the intelligent decision-making process of the model in this embodiment of the invention;

[0047] Figure 5 This is a visual connection diagram of the target policy network in an embodiment of the present invention. Detailed Implementation

[0048] This method aims to address the problems of low sample utilization, long training time, and unsatisfactory training results for complex tasks in reinforcement learning by using a weight scoring network to assign weights to data in the replay buffer and perform priority sampling. During training, the weight scoring network for calculating priority weights is initialized first, along with the policy network, which guides the agent's decision-making. Then, priority weights are calculated and priority sampling is performed on the sample data obtained by the agent from environmental interactions using the policy network. Next, importance weights are calculated on the sampled data, effectively correcting biases generated during priority sampling. Finally, the parameters of the policy network and the weight scoring network are updated using the sampled data. During the agent's decision-making process, the agent selects and executes appropriate actions based on the current state of the environment using the policy network. The environment provides reward feedback and updates its state after each new action, thus iteratively executing the process.

[0049] The present invention will be further described below with reference to the accompanying drawings:

[0050] Example:

[0051] In this embodiment, the reinforcement learning intelligent decision-making method based on priority experience replay includes two parts: a training process and an intelligent decision-making process. Taking image navigation task as the application scenario, the following is a detailed description with reference to the accompanying drawings:

[0052] See Figure 1 The training process includes:

[0053] S1. Model Initialization. Initialization primarily involves initializing the target policy network and the weight scoring network. The target policy network consists of 3 convolutional layers and 2 fully connected layers. A visual representation of the connections is as follows: Figure 5 As shown, the first three layers are convolutional layers, and the last three layers are fully connected layers. The network layer structure of the weight scoring network is the same as that of the target policy network. The target policy network and the weight scoring network are initialized by randomly setting network parameters. The network input is the state information observed by the current agent. The difference is that the output of the target policy network is the state-action value vector composed of all executable actions, and the output of the policy scoring network is the priority weight value of the current sample.

[0054] S2. Environmental Interaction Data Acquisition. The agent's target policy network guides the agent's interaction with the environment; the specific interaction process can be found in [link to relevant documentation]. Figure 2 It mainly consists of the following steps:

[0055] S21. Obtain environmental information; after encoding the image information of the environment, obtain it and perform corresponding preprocessing, such as grayscale processing.

[0056] S22. Input the image information into the target policy network. The processed image information is a multi-dimensional vector, which is input into the target policy network. Through the forward propagation of the neural network, the probability of executing each action is obtained. Here, the sum of the probabilities of all actions is 1. Therefore, the action taken by the agent under the current image information is a sample of the discrete random variable of action. In the discrete random variable, the probability distribution is the probability corresponding to each possible action.

[0057] S23. Execute the action selected by the target policy network; the previous step obtained the probability distribution of the target policy network for each action selection, and the distribution is sampled to obtain the specific action to be implemented. This action is the action that the agent needs to execute next.

[0058] S24. After the agent completes the current action, the environment provides a reward for that action and transitions to the next state. The transitioned state represents the environmental information inputted by the agent for its next decision.

[0059] S3. Set the priority weights of the interaction data (s, a, r, s') and put them into the replay buffer, where a represents the action, s represents the agent's state before taking the action, s' represents the agent's state after taking the action, and r represents the reward value given by the environment after the action is executed; the specific interaction process of the priority weight scoring network can be found in [link to relevant documentation]. Figure 3 It mainly consists of the following steps:

[0060] S31, Sample data; Sample data is a quadruple (s, a, r, s') formed after an agent interacts with the environment.

[0061] S32, Data Calculation: Input the environmental state information s from the sample data into the weight scoring network, and obtain the priority weight of the sample through forward propagation.

[0062] S33. Calculate the new weight values ​​for the samples, using the formula shown below, where w is the score output by the scoring network and r is the reward value:

[0063] w label =0.9*w + 0.1*r

[0064] S34. Update the parameters of the weight scoring network. The loss function is as follows:

[0065] L(θ)=(w label -w) 2

[0066] S4. After the replay buffer stores a certain amount of sample data, data is sampled from the replay buffer according to the priority weight of the data to conduct one round of training of the target policy network. The priority weight of the data includes the priority weight of the interaction data generated in this iteration and the importance weight of the replay buffer after bias correction.

[0067] S5. Calculate Importance Weights; Due to the priority sampling method, there will be some deviation in the final result. Here, importance weights are used to correct for this deviation. The calculation method for importance weights is as follows:

[0068]

[0069] Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of bias correction. When it is 1, the importance sampling weights will be completely corrected, where wj is the original weight calculated by the weight scoring network, and wi is the corrected weight.

[0070] S6. Calculate the loss function and update the target policy network parameters. The calculation formula is shown below:

[0071]

[0072] Where D represents the replay buffer, Q represents the target policy network, and θ represents the set of network parameters. Let represent the parameters of the Q-network, and E represent the expected value of the sample data (s,a,s',r) following a distribution D. γ represents the discount factor, a hyperparameter, typically set to 0.9. Based on the calculated loss value, gradient descent is used to update the network parameters of the target policy network, causing the loss function to gradually decrease along the gradient descent direction until convergence.

[0073] S7. When the selected training data is exhausted, if the set policy objective is not achieved, return to step S2 and continue the above steps. If the policy performance meets the requirements, training ends.

[0074] See Figure 4 The intelligent decision-making process for navigation includes:

[0075] S1. The agent observes its current environment and obtains image information.

[0076] S2. Based on the currently observed image information and the global navigation view, preprocess the image data.

[0077] S3. Input the state s into the trained policy model to obtain the probabilities of all executable actions.

[0078] S4. Sample the action to obtain the action performed by the agent this time. At the same time, the agent receives the reward value given by the environment and observes the changed environment.

[0079] S5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning intelligent decision-making method based on priority-based experience replay, characterized in that, The method includes a training process and a decision-making process, specifically including the following steps: Training process: A1: Model initialization; The model includes a target policy network and a weight scoring network. The network initialization is completed by randomly setting the network parameters. The inputs to both networks are the state information observed by the current agent. The output of the target policy network is the state-action value vector composed of all executable actions. The output of the policy scoring network is the priority weight value of the current sample. A2: Environmental interaction data collection; the agent's target policy network guides the agent's interaction with the environment, specifically including the following steps: A21: Acquire environmental information; Encode the image information of the environment, acquire it, and perform preprocessing. A22: Input the image information into the target policy network, and obtain the probability distribution of each action through the forward propagation of the neural network. Here, the sum of the probabilities of all actions is 1. A23: The obtained probability distribution is sampled to obtain the specific implementation action, which is the next action that the agent needs to perform; A24: After the agent completes the current action a, the environment provides a reward feedback r for the action and transitions from state s to the next state s'. The transitioned state is the environmental information that the agent will input for the next decision. A3: Use a weighted scoring network to set the priority weights of the interactive data (s,a,r,s') and put them into the replay buffer; A4: After the replay buffer stores a certain amount of sample data, data is sampled from the replay buffer according to the priority weight of the data, so as to carry out one round of training of the target policy network. A5: Calculate the importance weights; use the importance weights to correct for deviations in the priority weights, calculated as follows: Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of bias correction; where wj is the original weight calculated by the weighted scoring network, and wi is the corrected weight. A6: Calculate the loss function and update the target policy network parameters. The formula for calculating the loss function is shown below: Where D represents the replay buffer, Q represents the target policy network, and θ represents the set of network parameters. Let E represent the parameters of the target policy network, E represent the expected value of the sample data (s,a,s',r) following the distribution of D, and γ represent the discount coefficient. For the calculated loss value, the network parameters of the target policy network are updated using the gradient descent method, so that the loss function gradually decreases along the gradient descent direction until it converges. A7: When the selected training data is exhausted, if the set policy objective is not achieved, return to step A2 and continue the above steps; if the policy performance meets the requirements, end the training. Decision-making process: B1: The agent observes its current environment and acquires image information; B2: Preprocess the image data based on the currently observed image information and the global navigation view; B3: Input the state s into the trained model to obtain the probabilities of all executable actions; B4: Sample the action to obtain the action performed by the agent this time. At the same time, the agent receives the reward value given by the environment and observes the changed environment. B5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

2. The reinforcement learning intelligent decision-making method based on priority experience replay as described in claim 1, characterized in that, The specific steps of step A3 are as follows: S31, Sample data; Sample data is a quadruple (s, a, r, s') formed after an agent interacts with the environment; S32, Data Calculation: Input the environmental state information s from the sample data into the weight scoring network, and obtain the priority weight of the sample through forward propagation; S33. Calculate the new weight values ​​for the samples, using the formula shown below, where w is the score output by the weight scoring network: w label =0.9*w+0.1*r S34. Update the parameters of the weight scoring network. The loss function is as follows: L(θ)=(w label -w) 2 。 3. The reinforcement learning intelligent decision-making method based on priority experience replay as described in claim 1, characterized in that, The target policy network consists of 3 convolutional layers and 3 fully connected layers, with the first 3 layers being convolutional layers and the last 3 layers being fully connected layers; the network layer structure of the weight scoring network is the same as that of the target policy network.

4. The reinforcement learning intelligent decision-making method based on priority experience replay as described in claim 1, characterized in that, In step A4, the priority weight of the data includes the priority weight of the interactive data generated in this iteration and the importance weight of the replay buffer after deviation correction.

Images