Strategy entropy-based reinforcement learning experience-first playback method, system, equipment and medium
By introducing network loss and methods that reduce policy entropy, the importance of experience in reinforcement learning is reasonably assessed, which solves the problems of noise interference and biased evaluation in existing technologies, and achieves more efficient and accurate policy optimization.
Patent Information
- Application Number
- CN202511532576.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-01-27
AI Technical Summary
Existing reinforcement learning experience replay methods are too one-sided in evaluating the importance of experience, lack theoretical support, and result in noise interference and low learning efficiency.
We adopt a policy entropy-based reinforcement learning experience-priority replay method, which updates the priority of sample experiences by calculating network loss and reducing policy entropy, dynamically adjusts the experience sampling probability, and reasonably allocates the importance assessment of experiences.
It improves the learning efficiency and accuracy of reinforcement learning, reduces noise interference, and ensures that agents can optimize strategies more quickly in complex environments and adapt to different tasks and environments.
Smart Images
Figure CN121413701A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of reinforcement learning technology and relates to a reinforcement learning experience-first replay method, system, device, and medium based on policy entropy. Background Technology
[0002] Reinforcement learning (RL) is an important learning paradigm in machine learning that achieves learning objectives through the interaction between an agent and its environment. In reinforcement learning, the agent selects actions based on its current state, and the environment provides rewards and new states based on the agent's actions. The agent's goal is to learn a policy that maximizes the cumulative rewards gained over long-term interactions.
[0003] Experience replay is a key technique in reinforcement learning training. The basic idea behind experience replay is to store the experiences generated by the agent's interactions with the environment, and then randomly sample these stored experiences during training to update the agent's policy. This technique dates back to the 1990s, when researchers discovered that directly using experiences generated during agent-environment interactions for learning leads to instability in the learning process. This is because experiences generated by the agent in continuous interactions are often highly correlated, and this correlation can cause the learning process to be disturbed by local optima, resulting in poor learning outcomes.
[0004] Experience replay breaks down the correlation between experiences by storing them in a replay buffer and randomly sampling them during training. This random sampling allows the agent to access more diverse experiences during learning, thereby improving the stability and efficiency of learning. Furthermore, experience replay effectively utilizes limited interactive experiences, avoiding the repeated use of the same experience during training and further improving learning efficiency.
[0005] However, traditional experience replay methods also have some limitations. First, the capacity of the experience replay buffer is usually limited; as new experiences are continuously generated, older experiences may be discarded. This could result in some useful experiences being discarded before they are even used for training. Second, not all experiences are equally important to the learning process; some experiences may contribute more to policy updates, while others may be relatively less important. Traditional experience replay methods treat all experiences as equally important, leading to low learning efficiency for the agent.
[0006] To address these issues, researchers have proposed several improved experience replay methods. One such method is loss-based Prioritized Experience Replay (PER). The core idea of loss-based PER is to prioritize experiences based on their importance and then sample them according to this priority during the sampling process. The importance of an experience is typically measured by calculating its Temporal Difference (TD) error; experiences with larger TD errors are considered more important. By prioritizing the sampling of important experiences, PER can improve learning efficiency, enabling the agent to converge to the optimal policy more quickly.
[0007] Besides loss-based experience replay, some researchers have proposed other experience replay methods. These include Attentive Experience Replay (AER), which uses the similarity of experience to other experiences; Hindsight Experience Replay (HER), which relies on the coherence of experience trajectories; and Experience Replay based on Reducible Loss (ReLo). These methods comprehensively consider multiple factors such as the TD error of experience, state similarity, and loss differences. By designing complex sampling strategies, they enable the agent to utilize experience more effectively during training. These methods have achieved good results in various reinforcement learning tasks, demonstrating the importance and potential of experience replay in reinforcement learning. However, these methods are too one-sided in their evaluation of experience and lack sufficient theoretical support. During the training process, the experience selected by these methods can introduce noise interference and uncertainty. Summary of the Invention
[0008] The purpose of this invention is to provide a method, system, device, and medium for priority replay of reinforcement learning experience based on policy entropy, in order to solve the technical problems of overly one-sided evaluation of experience and lack of theoretical support, as well as noise interference in the training process of the agent.
[0009] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a reinforcement learning experience-first replay method based on policy entropy, comprising the following steps: Based on sample experience, network loss and policy entropy can be reduced; Update sample empirical priorities based on network loss and policy entropy reduction. The empirical sampling probability is obtained based on the priority of sample experience.
[0010] Furthermore, the method for obtaining network loss based on sample experience and reducing policy entropy is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. The sample experience is input into the corresponding agent, and the network loss is obtained according to the reinforcement learning algorithm.
[0011] Furthermore, the method for obtaining network loss and reducing policy entropy based on sample experience is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. By inputting sample experience into the corresponding agent, policy entropy can be reduced by obtaining the policy entropy of the online network and the policy entropy of the target network.
[0012] Furthermore, the formula for obtaining the reduced strategy entropy is as follows:
[0013] in, It can reduce strategy entropy; It is the policy entropy of the online network; It is the policy entropy of the target network.
[0014] Furthermore, the step of updating the sample empirical priority based on network loss and the ability to reduce policy entropy includes: Based on network loss and the ability to reduce policy entropy loss, one can obtain information that can reduce policy entropy loss. Prioritize sample experience based on reducing policy entropy loss.
[0015] Furthermore, the formula for obtaining the reduced policy entropy loss is as follows:
[0016] in, To reduce policy entropy loss; It is a network loss; It involves adjusting hyperparameters that can reduce the impact of policy entropy; It can reduce strategy entropy; The formula for obtaining the sample experience priority is as follows:
[0017] in, It is a constant, taking positive values; Prioritize based on sample experience.
[0018] Furthermore, the step of obtaining the empirical sampling probability based on sample experience priority includes: The experience is sampled according to the priority of the sample experience, and the sampling probability of each sample experience is obtained; The formula for obtaining the sampling probability of the sample experience is as follows:
[0019] in, Indicating sample experience Prioritize experience; For sample experience Priority; This represents the empirical sampling probability for each sample. It is a hyperparameter that controls the impact of priority.
[0020] Secondly, the present invention provides a reinforcement learning experience-first replay system based on policy entropy, comprising: The data acquisition module is used to obtain network loss and reduce policy entropy based on sample experience; The sample experience priority update module is used to update the sample experience priority based on network loss and the policy entropy that can be reduced. The empirical sampling probability acquisition module is used to obtain the empirical sampling probability based on the empirical priority of the samples.
[0021] Thirdly, the present invention provides an electronic device, comprising: a processor; a memory for storing computer program instructions; and steps for implementing a policy entropy-based reinforcement learning experience-first replay method when executing the computer program.
[0022] Fourthly, the present invention provides a storage medium storing computer program instructions, which are loaded and executed by a processor, wherein the processor performs a reinforcement learning experience-first replay method based on policy entropy.
[0023] Compared with the prior art, the present invention has the following beneficial effects: This invention obtains network loss and reduced policy entropy based on sample experience. Network loss accurately measures the difference between the action produced by the agent's current policy and the optimal action, providing a quantitative basis for evaluating the quality of the policy. By introducing reduced policy entropy, it helps reduce the interference of noise on the learning process, improves learning efficiency, and enables the agent to more accurately capture environmental information, thereby making better decisions. The two indicators work together to provide a comprehensive basis for rationally determining the priority of sample experience. Updating the sample experience priority based on network loss and reduced policy entropy allows for a more scientific determination of the value of each sample experience for policy optimization. By combining the two, the contribution of each experience to policy optimization can be accurately assessed, thereby rationally allocating priorities. The update method of this invention can automatically identify and prioritize sampling those experiences that are more valuable to policy optimization. By rationally updating the sample experience priority, it ensures that the agent can access the most valuable experiences during the learning process, thereby improving the accuracy of policy optimization. Obtaining the experience sampling probability based on the sample experience priority allows for dynamic adjustment of the experience sampling probability. This invention can effectively improve the efficiency and accuracy of policy optimization in reinforcement learning, providing a new solution for policy optimization in complex environments.
[0024] The system of this invention includes a data acquisition module, a sample experience priority update module, and an experience sampling probability acquisition module. The data acquisition module is used to acquire the network loss and the policy entropy reduction factor based on sample experience; the sample experience priority update module is used to update the sample experience priority based on the network loss and the policy entropy reduction factor; and the experience sampling probability acquisition module is used to acquire the experience sampling probability based on the sample experience priority. These modules work together to effectively improve the efficiency and accuracy of policy optimization in reinforcement learning.
[0025] The electronic device and storage medium of this invention can also effectively improve the efficiency and accuracy of policy optimization in reinforcement learning. Attached Figure Description
[0026] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a system block diagram of an embodiment of the present invention; Figure 3 This is a flowchart of a method according to another embodiment of the present invention; Figure 4 This is a schematic diagram of an embodiment of the present invention; Figure 5 This is a flowchart of another embodiment of the present invention. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0029] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1: See Figure 1 This embodiment discloses a reinforcement learning experience-first replay method based on policy entropy, including the following steps: S1, based on sample experience, yields network loss and reduced policy entropy. Network loss accurately measures the difference between the action produced by the agent's current policy and the optimal action, providing a quantitative basis for evaluating policy quality. Introducing reduced policy entropy helps reduce noise interference in the learning process, improves learning efficiency, and enables the agent to more accurately capture environmental information, thus making better decisions. In the experience replay stage, it effectively suppresses noise interference by automatically identifying and prioritizing samples of experiences more valuable for policy optimization, thereby reducing invalid sampling, improving overall sampling efficiency, and ensuring the accuracy of policy optimization. The combined effect of these two metrics provides a comprehensive basis for rationally prioritizing sample experience.
[0030] In this embodiment of the invention, the method for obtaining network loss is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. The sample experience is input into the corresponding agent, and the network loss is obtained according to the reinforcement learning algorithm.
[0031] In this embodiment of the invention, the method for reducing policy entropy acquisition is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. By inputting sample experience into the corresponding agent, policy entropy can be reduced by obtaining the policy entropy of the online network and the policy entropy of the target network.
[0032] In this embodiment of the invention, the formula for obtaining the reduced strategy entropy is as follows:
[0033] in, It can reduce strategy entropy; It is the policy entropy of the online network; It is the policy entropy of the target network.
[0034] S2, by updating the priority of sample experiences based on network loss and the reduction of policy entropy, the value of each sample experience for policy optimization can be determined more scientifically. Experiences with high network loss mean that the agent performs poorly in the state corresponding to that experience and need to be focused on and learned; while experiences with low reduction of policy entropy indicate that they are less affected by noise and are more reliable. By combining the two, the contribution of each experience to policy optimization can be accurately assessed, thereby allocating priorities reasonably. The update method of this invention can automatically identify and prioritize sampling those experiences that are more valuable to policy optimization. During experience replay, high-priority experiences are sampled and learned more frequently, thereby reducing the number of invalid samples. The agent can focus on learning key experiences, quickly adjust the policy, improve learning efficiency, and accelerate convergence to the optimal policy. By reasonably updating the priority of sample experiences, it is ensured that the agent can access the most valuable experiences during the learning process, thereby improving the accuracy of policy optimization.
[0035] In this embodiment of the invention, updating the sample empirical priority based on network loss and reduced policy entropy includes: Based on network loss and the ability to reduce policy entropy loss, one can obtain information that can reduce policy entropy loss. Prioritize sample experience based on reducing policy entropy loss.
[0036] In this embodiment of the invention, the formula for obtaining the reduced policy entropy loss is as follows:
[0037] in, To reduce policy entropy loss; It is a network loss; It involves adjusting hyperparameters that can reduce the impact of policy entropy; It can reduce strategy entropy; The formula for obtaining the sample experience priority is as follows:
[0038] in, It is a constant, taking positive values; Prioritize based on sample experience.
[0039] S3 obtains the empirical sampling probability based on the sample experience priority and can dynamically adjust the empirical sampling probability.
[0040] In this embodiment of the invention, obtaining the empirical sampling probability based on sample experience priority includes: The experience is sampled according to the priority of the sample experience, and the sampling probability of each sample experience is obtained; The formula for obtaining the sampling probability of the sample experience is as follows:
[0041] in, Indicating sample experience Prioritize experience; For sample experience Priority; This represents the empirical sampling probability for each sample. It is a hyperparameter that controls the impact of priority.
[0042] This invention effectively improves the efficiency and accuracy of policy optimization in reinforcement learning, providing a new solution for policy optimization in complex environments. Compared with existing technologies, this invention has the following advantages: This invention introduces a method to reduce policy entropy, which helps to reduce noise interference in the learning process, improves learning efficiency, and enables the agent to more accurately capture environmental information, thereby making better decisions. It effectively suppresses noise interference in the experience replay stage, automatically identifying and prioritizing the sampling of experiences more valuable for policy optimization, thus reducing the number of invalid samples, improving overall sampling efficiency, and ensuring the accuracy of policy optimization.
[0043] This invention can adapt to complex and ever-changing environments. By dynamically adjusting the empirical sampling probability, it achieves adaptive optimization for different tasks and environments, achieving excellent performance in both continuous and discrete control tasks. It can handle uncertainties in the environment, such as nonlinear dynamics and unknown noise distributions, thus having broad applicability in practical applications and providing a more reliable and accurate policy optimization scheme for reinforcement learning.
[0044] This invention enhances the agent's ability to extract information features by maximizing the mutual information between the hidden representation of the input and the distribution of policy actions. This helps the agent better understand the structure and dynamics of the environment, thereby making more accurate decisions.
[0045] See Figure 2Based on the above method, this invention also discloses a reinforcement learning experience-first replay system based on policy entropy, comprising: The data acquisition module is used to obtain network loss and reduce policy entropy based on sample experience; The sample experience priority update module is used to update the sample experience priority based on network loss and the policy entropy that can be reduced. The empirical sampling probability acquisition module is used to obtain the empirical sampling probability based on the empirical priority of the samples. The various modules work together to effectively improve the efficiency and accuracy of policy optimization in reinforcement learning, providing a new solution for policy optimization in complex environments.
[0046] Example 2: See Figure 3 This invention discloses a reinforcement learning experience-first replay method based on policy entropy. By introducing Reducible Policy Entropy (RPE) to calculate the sampling probability of experience and reducing uncertainty from TD error, it improves the efficiency and accuracy of policy optimization in reinforcement learning, achieving high-precision estimation of the policy optimization process. The method includes the following steps: S1. Calculate network loss based on sample experience; Based on the selected reinforcement learning algorithm and the sample experience obtained from constructing the state space. The sample experience is fed into the corresponding agent, and the network loss for this sample experience is calculated according to the selected reinforcement learning algorithm. The calculation method of network loss is related to the use of reinforcement learning algorithms.
[0047] S2. Calculations based on sample experience can reduce policy entropy; Sample experience The data is fed into the corresponding agent to calculate the reduced policy entropy based on the experience of this sample. In reinforcement learning, policy entropy is a metric for measuring policy uncertainty. This invention defines a method to reduce policy entropy by calculating the difference in policy entropy between the online network and the target network, as shown in the following expression:
[0048] in It is the policy entropy of the online network. It is the policy entropy of the target network.
[0049] In this way, policy entropy reduction can quantify the policy entropy that can be reduced in a specific experience, thus providing a new perspective for assessing the importance of experience.
[0050] S3. Combine network loss and the updated sample experience priority that can reduce policy entropy to construct an experience replay method; The empirical priority of samples is calculated by combining the reduction of policy entropy with network loss, and the specific formula is as follows:
[0051] in, To reduce policy entropy loss, It's a network loss. This refers to adjusting hyperparameters that can reduce the impact of policy entropy. The sample empirical priority is expressed as:
[0052] in It is a positive constant approaching zero, ensuring that each experience has a probability of being sampled.
[0053] The method of this invention not only considers the loss of experience, but also the change in policy entropy, making the sampling process pay more attention to experience that has a significant impact on policy optimization.
[0054] S4. Calculate the empirical sampling probability based on the priority; The experience is sampled according to the priority of the sample experience, and the specific expression for calculating the sampling probability of each experience is as follows:
[0055] in, This indicates the priority of sample experience i. Indicates the priority of sample experience j; This represents the sampling probability for each experience; It is a hyperparameter that controls the impact of priority.
[0056] This invention demonstrates through theoretical analysis that the reinforcement learning experience-first replay method based on policy entropy can provide a tighter lower bound on the mutual information between the input hidden representation and the policy action distribution, thereby implicitly maximizing mutual information during the sampling process.
[0057] In information theory, mutual information is a measure of the degree of interdependence between two random variables. In reinforcement learning, the mutual information between the input hidden representation and the policy action distribution is... Its variational lower bound is expressed as:
[0058]
[0059]
[0060] in It can be interpreted as Network loss , It is the policy entropy of the online network, while the policy entropy of the target network is... It can be treated as a constant during training and does not affect the mutual information between the input hidden representation and the policy action distribution.
[0061] This method, which combines reducing policy entropy with network loss to calculate experience-based priority maximization mutual information, helps reduce noise interference and extract more informative features, thereby improving the efficiency and accuracy of policy optimization.
[0062] See Figure 4 This invention is based on a robust adaptive experience replay method that reduces policy entropy and incorporates information theory.
[0063] In practical applications, reinforcement learning environments are often complex and dynamic, and the policy optimization process may be affected by noise and outliers. The experience-first replay method and system based on policy entropy in reinforcement learning, by introducing concepts that reduce policy entropy and information theory, not only improves the robustness of policy optimization but also adaptively adjusts the sampling policy to cope with different environments and tasks. This method performs excellently in both continuous and discrete control tasks, demonstrating its applicability and effectiveness in complex environments.
[0064] Example 3: like Figure 5 As shown, this embodiment provides a reinforcement learning experience-first replay method based on policy entropy, including the following steps: S1, Constructing a state-space model and selecting a reinforcement learning algorithm based on the reinforcement learning environment; In reinforcement learning, the state-space model is crucial for describing the dynamics of the environment. This invention first constructs a state-space model based on the characteristics of the reinforcement learning environment, including descriptions of the environment states and probability distributions of state transitions, and then selects an appropriate reinforcement learning algorithm based on the task type and the state-space model.
[0065] S2, builds experience playback based on the interaction results between the agent and the environment; In reinforcement learning, the interaction between the agent and the environment generates a series of states, actions, rewards, and sample experiences of new states. This invention first constructs an experience replay buffer based on these interaction results. Specifically, after each interaction between the agent and the environment, the generated sample experience tuples are stored in the replay buffer.
[0066] S3, training agents based on reinforcement learning algorithms; Experiences are sampled according to priority, and the specific expression for calculating the sampling probability of each experience is as follows:
[0067] in, Indicating sample experience priority, Indicating sample experience Priority; This represents the sampling probability for each experience; It is a hyperparameter that controls the impact of priority.
[0068] The network loss of the selected batch of samples is calculated based on the chosen basic algorithm, the gradient is calculated, and the agent is optimized.
[0069] S4, update sample empirical priority based on network loss and policy entropy reduction; The difference in policy entropy between the online network and the target network is used to define a policy entropy that can be reduced, as shown in the following expression:
[0070] in, It is the policy entropy of the online network. It is the policy entropy of the target network.
[0071] The priority of experience is calculated by combining the reduction of policy entropy with the reduction of policy entropy loss, as shown in the following formula:
[0072] in, It is the loss of a given offline strategy algorithm. This refers to adjusting hyperparameters that can reduce the impact of policy entropy. The sample empirical priority is expressed as: , in It is a very small positive number, ensuring that each experience has a probability of being sampled.
[0073] Based on the above method, this embodiment provides a testing method for the reinforcement learning experience-first replay method based on policy entropy, as detailed below: S1, Test Environment: The test software environment for the reinforcement learning experience-first replay method based on policy entropy proposed in this invention is Python 3.9, CPU is Intel Core i5-13600KF 3.50 GHz, GPU is NVIDIA GeForce RTX 3060, memory is 32GB, and hard disk storage is 1TB.
[0074] S2, Experimental Environment: The experimental environment of this invention consists of two general reinforcement learning frameworks: DeepMind Control Suite (DMC) and Arcade Learning Environment (ALE). DMC is an environment for continuous action reinforcement learning, while ALE is an environment for discrete action reinforcement learning.
[0075] S3, Baseline Method: To evaluate the effectiveness of the present invention, three benchmark methods were selected and tested in two environments: uniformly distributed random sampling (RS), priority experience replay (PER), experience replay based on reduced loss (ReLo), and the method of the present invention (ReEnt).
[0076] S4, Experiment Setup: To verify the effectiveness of the policy entropy-based reinforcement learning experience-first replay method and system, two reinforcement learning environments were set up, and different deep reinforcement learning algorithms were used in different environments.
[0077] In the DMC environment, we used the Soft Actor-Critic (SAC) deep learning algorithm, with environmental tasks including cheetah run, finger spin, hopper hop, quadruped walk, reacher easy, and walker run. In the ALE environment, we used the Rainbow Deep Q-value Network (Rainbow-DQN) deep reinforcement learning algorithm, with environmental tasks including Alien, Amidar, Assault, Frostbite, and James Bond. The average score of the algorithm across multiple test environments was used as the experimental metric.
[0078] S5, Experimental Results: The experimental results comparing existing mainstream solutions with those of this invention are shown in Tables 1 and 2. The results demonstrate that the reinforcement learning experience-first replay method and system based on policy entropy proposed in this invention outperform other methods.
[0079] Table 1. Average scores of agents using different sampling methods in various tasks within the DMC environment:
[0080] Table 2. Average scores of agents using different sampling methods in various tasks within the ALE environment:
[0081] This invention proposes a novel experience replay method, namely, a reinforcement learning experience-first replay method and system based on policy entropy. This method has the ability to implicitly maximize the mutual information between the input hidden representation and the policy action distribution. By reducing policy entropy, this invention effectively reduces noise, thereby improving the accuracy of feature extraction from experience. Simultaneously, this method reduces the uncertainty from TD errors by calculating the sampling probability of experience, thus enhancing the stability of the policy.
[0082] This invention introduces a more efficient sampling mechanism during experience replay by reducing policy entropy. The constructed sampling mechanism aims to reduce the impact of noise in experience replay, thereby improving the accuracy of extracting experience features during policy learning.
[0083] This robust experience replay method improves policy learning capabilities in complex environments, such as continuous and discrete control tasks. Because it is both robust and efficient, it is well-suited for reinforcement learning tasks subjected to complex disturbances, enabling high-quality experience sampling and policy optimization.
[0084] An electronic device includes: a processor; a memory for storing computer program instructions; and an implementation of a policy entropy-based reinforcement learning experience-first replay method when executing the computer program.
[0085] A storage medium storing computer program instructions, which are loaded and executed by a processor, wherein the processor performs a reinforcement learning experience-first replay method based on policy entropy.
[0086] A computer program product comprising computer instructions that instruct a computer to execute a policy entropy-based reinforcement learning experience-first replay method.
[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0091] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.
Claims
1. A reinforcement learning experience-first replay method based on policy entropy, characterized in that, Includes the following steps: Based on sample experience, network loss and policy entropy can be reduced; Update sample empirical priorities based on network loss and policy entropy reduction. The empirical sampling probability is obtained based on the priority of sample experience.
2. The reinforcement learning experience-first replay method based on policy entropy according to claim 1, characterized in that, The method for obtaining network loss and reducing policy entropy based on sample experience is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. The sample experience is input into the corresponding agent, and the network loss is obtained according to the reinforcement learning algorithm.
3. The reinforcement learning experience-first replay method based on policy entropy according to claim 1, characterized in that, The method for obtaining network loss and reducing policy entropy based on sample experience is as follows: A state space is constructed based on a reinforcement learning algorithm, and sample experience is acquired through agent interaction. By inputting sample experience into the corresponding agent, policy entropy can be reduced by obtaining the policy entropy of the online network and the policy entropy of the target network.
4. The reinforcement learning experience-first replay method based on policy entropy according to claim 3, characterized in that, The formula for obtaining the reduced strategy entropy is as follows: in, It can reduce strategy entropy; It is the policy entropy of the online network; It is the policy entropy of the target network.
5. The reinforcement learning experience-first replay method based on policy entropy according to claim 1, characterized in that, The step of updating sample empirical priorities based on network loss and policy entropy reduction includes: Based on network loss and the ability to reduce policy entropy loss, one can obtain information that can reduce policy entropy loss. Prioritize sample experience based on reducing policy entropy loss.
6. The reinforcement learning experience-first replay method based on policy entropy according to claim 5, characterized in that, The formula for obtaining the reduced policy entropy loss is as follows: in, To reduce policy entropy loss; It is a network loss; It involves adjusting hyperparameters that can reduce the impact of policy entropy; It can reduce strategy entropy; The formula for obtaining the sample experience priority is as follows: in, It is a constant, taking positive values; Prioritize based on sample experience.
7. The reinforcement learning experience-first replay method based on policy entropy according to claim 1, characterized in that, The step of obtaining the empirical sampling probability based on sample experience priority includes: The experience is sampled according to the priority of the sample experience, and the sampling probability of each sample experience is obtained; The formula for obtaining the sampling probability of the sample experience is as follows: in, Indicating sample experience Prioritize experience; For sample experience Priority; This represents the empirical sampling probability for each sample. It is a hyperparameter that controls the impact of priority.
8. A reinforcement learning experience-first replay system based on policy entropy, characterized in that, include: The data acquisition module is used to obtain network loss and reduce policy entropy based on sample experience; The sample experience priority update module is used to update the sample experience priority based on network loss and the policy entropy that can be reduced. The empirical sampling probability acquisition module is used to obtain the empirical sampling probability based on the empirical priority of the samples.
9. An electronic device, comprising: A processor; a memory, an electronic device for storing computer program instructions; characterized in that, when executing the computer program, it implements the steps of the policy entropy-based reinforcement learning experience-first replay method as described in any one of claims 1-7.
10. A storage medium storing computer program instructions, characterized in that, When the computer program instructions are loaded and run by the processor, the processor executes the reinforcement learning experience-first replay method based on policy entropy as described in any one of claims 1-7.