Reinforced learning intelligent decision-making method based on priority experience replay

Abstract

The invention relates to the technical field of machine learning and reinforcement learning, discloses a reinforcement learning intelligent decision-making method based on priority experience replay, and provides a priority experience replay method based on a weight scoring network in order to solve the problem that an intelligent agent is difficult to learn in a complex environment in existing reinforcement learning. And an intelligent agent is guided to carry out priority quantification on the data in the replay buffer area, so that historical experience data is more efficiently utilized to carry out learning. In the training process, the comprehensive loss function is calculated according to the training data, and the target strategy network is updated by using the gradient descent method, so that the decision performance of the target strategy network is improved. By means of the reinforcement learning intelligent decision-making method based on priority experience replay, the problem that in existing reinforcement learning, an intelligent agent is difficult to learn in a complex environment can be effectively solved, and the decision-making efficiency and accuracy of the intelligent agent are improved.

Description

Technical Field

[0001] The invention belongs to the field of machine learning and reinforcement learning, and in particular relates to a reinforcement learning intelligent decision-making method based on priority experience replay. Background Art

[0002] One of the main goals of artificial intelligence is to give intelligent agents the ability to perform diverse and complex tasks. In the current field of machine learning, mainstream research directions often focus on pre-collected or carefully constructed data sets with clear labels. These static data provide the basis for model training, allowing machine learning algorithms to learn and identify specific patterns or laws from them. However, as an outstanding representative of the closed-loop learning paradigm, reinforcement learning is unique in its methodology in that it can use dynamically generated data and real-time feedback labels to guide the entire learning process. By constantly interacting with the environment, reinforcement learning algorithms can gradually optimize their own behavioral strategies through trial and error, achieving self-improvement and perfection.

[0003] Today, reinforcement learning has achieved remarkable results in many sequential decision-making problems. Whether it is board games, video games or robot control tasks, reinforcement learning has demonstrated strong problem-solving capabilities. However, it is worth noting that when traditional reinforcement learning methods use historical trajectory data, they often simply use these data directly in the algorithm training process, ignoring the potential application value behind these data.

[0004] In fact, these historical trajectory data not only record the behavior of the agent in the process of interacting with the environment, but also contain rich environmental information and the learning status of the agent. In-depth mining and analysis of these data can not only provide more abundant training signals for reinforcement learning algorithms, but also reveal the decision-making mechanism and learning rules of the agent in the process of solving tasks. Therefore, how to use historical trajectory data more effectively will become an important research direction in the field of reinforcement learning in the future.

[0005] Reinforcement learning has achieved great success in sequential decision-making problems. Although on-policy methods are effective, using historical experience can produce better sample utilization, which is very important in environments where samples are difficult to obtain, and can also further save resources.

[0006] Experience replay enables data reuse and is widely used in off-policy reinforcement learning. The Prioritized Memory Replay Mechanism (PER) believes that samples with larger temporal differential errors in the replay buffer are more important because the agent has a lower degree of mastery of this part of knowledge. Based on PER, researchers have proposed various sampling strategies for priority sampling, such as Prioritized Sequential Replay (PSER), Recent Memory Enhancement (ERE), etc. On the other hand, Distribution Correction (DisCor) believes that the data distribution of the approximate dynamic programming (ADP) update directly affects the learned solution, and the replay buffer needs to be reweighted to a distribution with corrective feedback, because the "bootstrapping" of reinforcement learning leads to inaccurate estimates of the Q value, which leads to a lack of corrective feedback in reinforcement learning. However, both algorithms only consider their own factors and do not combine them. Summary of the invention

[0007] The present invention is based on the Deep Q-Network algorithm and proposes a reinforcement learning intelligent decision-making method based on priority experience replay to solve the problems of low data utilization in the replay buffer and slow improvement of algorithm performance in existing reinforcement learning.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A reinforcement learning intelligent decision-making method based on priority experience replay includes a training process and a decision-making process, specifically including the following steps:

[0010] Training process:

[0011] A1: Model initialization; The model includes a target policy network and a weight scoring network. The network initialization is completed by randomly setting network parameters. The network input is the state information observed by the current agent. The output of the target policy network is the state-action value vector composed of all executable actions. The output of the policy scoring network is the priority weight value of the current sample. Specifically, the target policy network consists of 3 convolutional layers and 3 fully connected layers, the first 3 layers are convolutional layers, and the last 3 layers are fully connected layers. The network layer structure of the weight scoring network is the same as that of the target policy network.

[0012] A2: Collect data through environmental interaction. The target policy network of the agent will guide the agent to interact with the environment. Specifically, it includes the following steps:

[0013] A21: Obtain environmental information; encode the image information of the environment and obtain it, and perform preprocessing;

[0014] A22: Input the image information into the target policy network, and obtain the probability distribution of each action through the forward propagation of the neural network. Here, the sum of the probabilities of all actions is 1.

[0015] A23: Sample the obtained probability distribution to obtain a specific implementation action, which is the action that the agent needs to perform next;

[0016] A24: After the agent completes the current action a, the environment gives the reward feedback r for the action, and the state s is transferred to the next state s'. The state after the transfer is the environmental information input for the next decision of the agent;

[0017] A3: Use the weighted scoring network to set the priority weight of the interaction data (s, a, r, s') and put it into the replay buffer. The specific steps are as follows:

[0018] S31, sample data; sample data is a four-tuple (s, a, r, s') formed after an intelligent agent interacts with the environment.

[0019] S32, data calculation; input the environmental state information s in the sample data into the weight scoring network respectively, and forward propagate to obtain the priority weight of the sample.

[0020] S33. Calculate the new weight value of the sample. The calculation formula is as follows, where w is the score output by the weight scoring network:

[0021] w label =0.9*w+0.1*r

[0022] S34. Update the parameters of the weighted scoring network. The loss function is as follows:

[0023] L(θ)=(w label -w) 2

[0024] A4: After the replay buffer stores a certain amount of sample data, data is sampled from the replay buffer according to the priority weight of the data to perform a round of training for the target policy network; the priority weight of the data includes the priority weight of the interaction data generated during this iteration and the importance weight of the replay buffer after deviation correction;

[0025] A5: Calculate the importance weight; use the importance weight to correct the deviation. The calculation method is as follows:

[0026]

[0027] Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of deviation correction; where wj is the original weight calculated by the weight scoring network, and wi is the corrected weight;

[0028] A6: Calculate the loss function and update the target strategy network parameters. The loss function calculation formula is as follows:

[0029]

[0030] Where D represents the replay buffer, Q represents the target policy network, and θ represents the parameter set of the network. represents the parameters of the target policy network, E represents the expected value of the sample data (s, a, s', r) under the distribution of D; γ represents the discount coefficient; for the calculated loss value, the network parameters of the target policy network are updated using the gradient descent method, so that the loss function gradually decreases along the gradient descent direction until convergence;

[0031] A7: When the selected training data is used up, if the set strategy goal is not achieved, return to step A2 and continue the above steps; if the performance of the strategy meets the requirements, the training ends;

[0032] Decision-making process:

[0033] B1: The agent observes the current environment and obtains image information;

[0034] B2: Preprocess the image data according to the currently observed image information and the global navigation view;

[0035] B3: Input the state s into the trained model to obtain the probabilities corresponding to all executable actions.

[0036] B4: Sample the actions to obtain the actions performed by the agent this time. At the same time, the agent accepts the reward value given by the environment and observes the changed environment.

[0037] B5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

[0038] Compared with the existing technology, the beneficial effects of the present invention are mainly reflected in the following aspects:

[0039] First, the present invention can more accurately select samples with higher quality through the guidance of priority weights, thereby achieving rapid convergence of the reinforcement learning decision model; this improvement not only significantly improves the training efficiency of the model, but also greatly improves the computer processing speed, reduces the time cost required for training, and saves users valuable resources and time.

[0040] Secondly, the method is not only applicable to specific off-policy methods, but also has wide universality and can be applied to various off-policy scenarios. This feature makes the present invention more flexible and applicable in practical applications and can meet the needs of different scenarios.

[0041] Finally, the present invention effectively solves the problem of convergence difficulty of reinforcement learning strategy by improving the utilization rate of samples and the performance of the algorithm. In terms of algorithm performance, the present invention has significant advantages and can bring better user experience and higher efficiency.

[0042] In summary, the present invention has significant beneficial effects in the field of reinforcement learning. It not only improves the training efficiency and sample utilization of the model, but also solves the problem of difficult convergence of reinforcement learning strategies, bringing new breakthroughs to the development of related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the model training process in an embodiment of the present invention;

[0044] Figure 2 is a decision interaction flow chart in the model training process in an embodiment of the present invention;

[0045] Figure 3 A flowchart of priority weight calculation in the model training process in an embodiment of the present invention;

[0046] Figure 4 is a model intelligent decision-making flow chart in an embodiment of the present invention;

[0047] Figure 5 A diagram showing a visualized connection mode of a target policy network in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] This method aims to propose a method of using a weight scoring network in reinforcement learning to weight the data in the replay buffer and perform priority sampling, so as to solve the problems of low sample utilization, long training time, and unsatisfactory training effect in complex tasks faced by existing reinforcement learning. During the training process, the weight scoring network required for calculating the priority weight is first initialized, and the policy network of reinforcement learning is initialized at the same time. The policy network is responsible for guiding the agent to make decisions; then, the priority weight is calculated for the sample data obtained by the agent from the environment interaction using the policy network, and priority sampling is performed; then, the importance weight is calculated for the sampled sample data, and the importance weight can effectively correct the deviation generated in the priority sampling process; finally, the sampled data is used to update the parameters of the policy network and the weight scoring network. In the decision-making process of the agent, the agent needs to be more aware of the current state of the environment and use the policy network to select the corresponding action to execute. The environment will give reward feedback after the new action of the agent and update to the new state at the same time, so as to iteratively execute.

[0049] The present invention will be further described below in conjunction with the accompanying drawings:

[0050] Example:

[0051] In this embodiment, the priority experience replay-based reinforcement learning intelligent decision-making method includes two parts: a training process and an intelligent decision-making process. The image navigation task is used as an application scenario, and is specifically described below in conjunction with the accompanying drawings:

[0052] See also Figure 1 , the training process includes:

[0053] S1. Model initialization. Initialization mainly initializes the target policy network and the weight scoring network. The target policy network consists of 3 convolutional layers and 2 fully connected layers. A visual connection method is as follows Figure 5 As shown in the figure, the first three layers are convolutional layers, and the last three layers are fully connected layers; the network layer structure of the weight scoring network is the same as the target policy network; the target policy network and the weight scoring network complete the network initialization through randomly set network parameters, and the network inputs are all the state information observed by the current agent. The difference is that the output of the target policy network is the state-action value vector composed of all executable actions, and the output of the strategy scoring network is the priority weight value of the current sample.

[0054] S2, environmental interaction data collection. The target strategy network of the agent will guide the agent to interact with the environment. The specific interaction process can be seen in Figure 2 , mainly divided into the following steps:

[0055] S21, obtaining environmental information; encoding the image information of the environment and obtaining it, and performing corresponding preprocessing, such as grayscale processing, etc.

[0056] S22, input the image information into the target policy network; the processed image information is a multi-dimensional vector, which is input into the target policy network, and the probability of executing each action is obtained through the forward propagation of the neural network, where the sum of the probabilities of all actions is 1. Therefore, what action the agent takes under the current image information is a sampling of the discrete random variable action. In the discrete random variable, the probability distribution is the probability corresponding to each possible action.

[0057] S23, execute the action selected by the target policy network; in the previous step, the probability distribution of each action selected by the target policy network is obtained, and the distribution is sampled to obtain a specific implementation action, which is the action that the agent needs to perform next.

[0058] S24: After the agent completes the current action, the environment gives reward feedback for the action and the state is transferred to the next state. The state after the transfer is the environmental information input for the next decision of the agent.

[0059] S3. Set the priority weights of the interaction data (s, a, r, s') and put them into the replay buffer, where a represents the action, s represents the state of the agent before taking the action, s' represents the state of the agent after taking the action, and r represents the reward value given by the environment after the action is executed; the specific interaction process of the weighted scoring network of the priority weight can be found in Figure 3 , mainly divided into the following steps:

[0060] S31, sample data; sample data is a four-tuple (s, a, r, s') formed after an intelligent agent interacts with the environment.

[0061] S32, data calculation; input the environmental state information s in the sample data into the weight scoring network respectively, and forward propagate to obtain the priority weight of the sample.

[0062] S33. Calculate the new weight value of the sample. The calculation formula is as follows, where w is the score output by the scoring network and r is the reward value:

[0063] w label =0.9*w+0.1*r

[0064] S34. Update the parameters of the weighted scoring network. The loss function is as follows:

[0065] L(θ)=(w label -w) 2

[0066] S4. After the replay buffer stores a certain amount of sample data, data is sampled from the replay buffer according to the priority weight of the data to perform a round of training for the target policy network; the priority weight of the data includes the priority weight of the interaction data generated during this iteration and the importance weight in the replay buffer after deviation correction.

[0067] S5. Calculate the importance weight. Since the priority sampling method is used, there will be a certain deviation in the final result. Here, the importance weight is used to correct the deviation. The calculation method of the importance weight is as follows:

[0068]

[0069] Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of deviation correction. When it is 1, the importance sampling weight will be completely corrected, where wj is the original weight calculated by the weight scoring network, and wi is the corrected weight.

[0070] S6. Calculate the loss function and update the target strategy network parameters. The calculation formula is as follows:

[0071]

[0072] Where D represents the replay buffer, Q represents the target policy network, and θ represents the parameter set of the network. represents the parameters of the Q network, and E represents the expected value of the sample data (s, a, s', r) under the distribution of D. γ represents the discount coefficient, which is a hyperparameter and is generally taken as 0.9. Based on the calculated loss value, the network parameters of the target policy network are updated using the gradient descent method, so that the loss function gradually decreases along the gradient descent direction until convergence.

[0073] S7: After the selected training data is used up, if the set strategy goal is not achieved, return to step S2 and continue the above steps. If the performance of the strategy meets the requirements, the training is terminated.

[0074] See also Figure 4 , the intelligent decision-making process of navigation includes:

[0075] S1. The agent observes the current environment and obtains image information.

[0076] S2. Preprocess the image data according to the currently observed image information and the global navigation view.

[0077] S3. Input the state s into the trained policy model to obtain the probabilities corresponding to all executable actions.

[0078] S4. Sample the actions to obtain the actions performed by the agent this time. At the same time, the agent accepts the reward value given by the environment and observes the changed environment.

[0079] S5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision is completed.

[0080] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by technicians in this technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A reinforcement learning intelligent decision-making method based on priority experience replay, characterized in that: The method includes a training process and a decision-making process, and specifically includes the following steps: Training process: A1: Model initialization; the model includes a target policy network and a weight scoring network. The network initialization is completed by randomly setting network parameters. The inputs of both networks are the state information observed by the current agent. The output of the target policy network is the state-action value vector composed of all executable actions, and the output of the policy scoring network is the priority weight value of the current sample. A2: Collect data through environmental interaction. The target strategy network of the agent guides the agent to interact with the environment. Specifically, it includes the following steps: A21: Obtain environmental information; encode the image information of the environment and obtain it, and perform preprocessing; A22: Input the image information into the target policy network, and obtain the probability distribution of each action through the forward propagation of the neural network. Here, the sum of the probabilities of all actions is 1. A23: Sample the obtained probability distribution to obtain a specific implementation action, which is the action that the agent needs to perform next; A24: After the agent completes the current action a, the environment gives the reward feedback r for the action, and the state s is transferred to the next state s'. The state after the transfer is the environmental information input for the next decision of the agent; A3: Use the weight scoring network to set the priority weight of the interaction data (s, a, r, s') and put it into the replay buffer; A4: After a certain amount of sample data is stored in the replay buffer, data is sampled from the replay buffer according to the priority weight of the data to perform a round of training for the target policy network; A5: Calculate the importance weight; use the importance weight to correct the deviation of the priority weight. The calculation method is as follows: Where N represents the sample capacity of the replay buffer, β is a hyperparameter used to control the degree of deviation correction; where wj is the original weight calculated by the weight scoring network, and wi is the corrected weight; A6: Calculate the loss function and update the target strategy network parameters. The loss function calculation formula is as follows: Where D represents the replay buffer, Q represents the target policy network, and θ represents the parameter set of the network. represents the parameters of the target policy network, E represents the expected value of the sample data (s, a, s', r) under the distribution of D; γ represents the discount coefficient; for the calculated loss value, the network parameters of the target policy network are updated using the gradient descent method, so that the loss function gradually decreases along the gradient descent direction until convergence; A7: When the selected training data is used up, if the set strategy goal is not achieved, return to step A2 and continue the above steps; if the performance of the strategy meets the requirements, the training ends; Decision-making process: B1: The agent observes the current environment and obtains image information; B2: Preprocess the image data according to the currently observed image information and the global navigation view; B3: Input the state s into the trained model to obtain the probabilities of all executable actions; B4: Sample the actions to obtain the actions performed by the agent this time. At the same time, the agent accepts the reward value given by the environment and observes the changed environment. B5. Repeat steps S1 to S4 until the navigation task is completed and the intelligent decision-making ends.

2. According to claim 1, a priority experience replay-based reinforcement learning intelligent decision-making method is characterized in that: The specific steps of step A3 are as follows: S31, sample data; sample data is a four-tuple (s, a, r, s') formed after the agent interacts with the environment; S32, data calculation; input the environmental state information s in the sample data into the weight scoring network respectively, and forward propagate to obtain the priority weight of the sample; S33. Calculate the new weight value of the sample. The calculation formula is as follows, where w is the score output by the weight scoring network: w label =0.9*w+0.1*r S34. Update the parameters of the weighted scoring network. The loss function is as follows: L(θ)=(w label -w) 2 。 3. The method for intelligent decision-making based on priority experience replay based on reinforcement learning according to claim 1, characterized in that: The target strategy network consists of 3 convolutional layers and 3 fully connected layers, the first 3 layers are convolutional layers, and the last 3 layers are fully connected layers; the network layer structure of the weighted scoring network is the same as that of the target strategy network.

4. The method for intelligent decision-making based on priority experience replay based on reinforcement learning according to claim 1, characterized in that: In step A4, the priority weight of the data includes the priority weight of the interactive data generated in this iteration and the importance weight in the replay buffer after deviation correction.

Images