Model optimization method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a model optimization method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of reinforcement learning. According to the model optimization method provided by the invention, high-quality historical strategy experience is screened out from historical strategy experience generated by previous interaction between an intelligent agent and an environment; then, a strategy adopted by a reward function optimization agent is constructed by using high-quality historical strategy experience and a total variation distance, so that the utilization rate of a historical experience sample can be improved in reinforcement learning, and the cost and time required by interaction between the agent and the environment in reinforcement learning can be reduced; moreover, the introduced reward function can take the strategy difference as a negative reward, thereby limiting the amplitude of strategy updating, and facilitating the guarantee of the stability of the strategy.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and more specifically, to a model optimization method, device, electronic device and computer-readable storage medium in the field of reinforcement learning technology. Background Art

[0002] Reinforcement Learning (RL) has achieved remarkable success in many fields in recent years, such as video games, robot control, and multi-agent systems. However, in practical applications, the utilization of historical experience samples is still an important challenge for reinforcement learning. In particular, in continuous control tasks, how to use historical experience samples to accelerate policy optimization has become a research hotspot. Therefore, how to improve the utilization of historical experience samples in reinforcement learning has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a model optimization method, device, electronic device and computer-readable storage medium, which can improve the utilization rate of historical experience samples in reinforcement learning, and can also reduce the cost and time required for the interaction between the intelligent agent and the environment in reinforcement learning.

[0004] In a first aspect, a model optimization method is provided, which includes: when the i-th interaction condition between an agent and an environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience; wherein the strategy experience attribute is used to measure the quality of the strategy experience, and the experience buffer stores the historical strategy experience adopted by the agent for the i-th interaction with the environment before, i being a positive integer greater than or equal to 1; using a reward function and the target strategy experience to optimize the strategy adopted by the agent so that the agent interacts with the environment for the i-th time; wherein the reward function is constructed by the total variation distance; when the i-th interaction between the agent and the environment is completed, obtaining the environment reward obtained by the reward function; when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, stopping the optimization of the strategy adopted by the agent; when the reward difference is not within the preset difference interval, continuing the i+1-th interaction between the agent and the environment, and repeating the above steps until the reward difference is within the preset difference interval.

[0005] In the above technical scheme, the present application adopts the method of screening the historical strategy experience in the experience buffer according to the strategy experience attribute used to measure the quality of the strategy experience when the i-th interaction condition between the intelligent agent and the environment is met, and obtains the target experience strategy, and adopts the reward function constructed by the total variation distance and the target strategy experience to optimize the strategy adopted by the intelligent agent, so that the intelligent agent interacts with the environment for the i-th time, and when the i-th interaction between the intelligent agent and the environment is completed, the environment reward obtained by the reward function is obtained, and when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, the strategy adopted by the intelligent agent is stopped from being optimized, and when the reward difference is not within the preset difference interval, the optimization is continued. The agent interacts with the environment for the i+1th time, and repeats the above steps until the reward difference is within the preset difference range. This technical solution realizes the use of current policy data, high-quality historical policy experience and the reward function constructed by the TV distance to optimize the strategy adopted by the agent. It can not only improve the utilization rate of historical experience samples in reinforcement learning, but also reduce the cost and time required for the interaction between the agent and the environment in reinforcement learning, as well as reduce the dependence on actual environmental interaction data during the strategy learning process, which is conducive to improving the efficiency and speed of the agent's strategy learning. In addition, the introduced reward function can use the strategy difference as a negative reward, thereby limiting the amplitude of the strategy update, which is conducive to ensuring the stability of the strategy.

[0006] In combination with the first aspect, in some possible implementations, the strategy experience attribute includes a strategy experience reward, and the strategy experience reward is the sum of the environment rewards obtained when the intelligent agent adopts the strategy to interact with the environment; screening the historical strategy experience in the experience buffer according to the strategy experience attribute to obtain the target strategy experience includes: taking the historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

[0007] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the strategy experience attribute includes an advantage estimate, and the advantage estimate is obtained through a generalized advantage estimate; the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: taking the historical strategy experience in the experience buffer whose advantage estimate is greater than or equal to a second threshold as the target strategy experience.

[0008] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the strategy experience attribute includes state similarity, and the state similarity is the similarity between a first state and a second state, the first state is the state before the intelligent agent interacts with the environment for the i-th time, and the second state is the state when the intelligent agent completes the interaction with the environment using the historical strategy; the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: taking the historical strategy experience in the experience buffer whose state similarity is greater than or equal to a third threshold as the target strategy experience.

[0009] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, when the i-th interaction condition between the intelligent agent and the environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: when the i-th interaction condition between the intelligent agent and the environment is met, the number of historical strategy experiences in the experience buffer is obtained; when the number is greater than or equal to a fourth threshold, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience.

[0010] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the model optimization method also includes: when the reward difference is not within the preset difference range, storing the strategy experience generated when the i-th interaction between the agent and the environment is completed in the experience buffer to update the experience buffer.

[0011] In combination with the first aspect and the above implementation, in some possible implementations, the reward function is:

[0012]

[0013] in, represents the environmental reward, R(s,a) represents the original reward, ξ represents the weight, TV(π,π k )(s) represents the total variation distance, s represents the state, a represents the action, (s, a) represents the state-action pair, π represents the new strategy obtained after optimizing the strategy adopted by the agent, π k Indicates the current policy.

[0014] In a second aspect, a model optimization device is provided, the model optimization device comprising:

[0015] An experience screening module is used to screen the historical strategy experience in the experience buffer according to the strategy experience attribute to obtain the target strategy experience when the i-th interaction condition between the intelligent agent and the environment is met; wherein the strategy experience attribute is used to measure the quality of the strategy experience, and the experience buffer stores the historical strategy experience adopted by the intelligent agent in the i-th interaction with the environment before, i is a positive integer greater than or equal to 1;

[0016] A model optimization module is used to optimize the strategy adopted by the agent using a reward function and the target strategy experience so that the agent interacts with the environment for the i-th time; wherein the reward function is constructed by the total variation distance; when the i-th interaction between the agent and the environment is completed, the environment reward obtained by the reward function is obtained; when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, the optimization of the strategy adopted by the agent is stopped; when the reward difference is not within the preset difference interval, the agent continues to interact with the environment for the i+1th time, and the above steps are repeated until the reward difference is within the preset difference interval.

[0017] In conjunction with the second aspect, in some possible implementations, the strategy experience attribute includes a strategy experience reward, and the strategy experience reward is the sum of the environment rewards obtained when the agent adopts the strategy to interact with the environment, and the experience screening module includes:

[0018] The first screening unit is configured to use the historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

[0019] In combination with the second aspect and the above implementation, in some possible implementations, the strategy experience attribute includes an advantage estimate, the advantage estimate is obtained by generalized advantage estimation, and the experience screening module includes:

[0020] The second screening unit is used to use the historical strategy experience in the experience buffer whose advantage estimate is greater than or equal to a second threshold as the target strategy experience.

[0021] In combination with the second aspect and the above implementation manner, in some possible implementation manners, the strategy experience attribute includes state similarity, the state similarity is the similarity between a first state and a second state, the first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using the historical strategy, and the experience screening module includes:

[0022] The third screening unit is used to use the historical strategy experience in the experience buffer whose state similarity is greater than or equal to a third threshold as the target strategy experience.

[0023] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the experience screening module is specifically used to obtain the number of historical strategy experiences in the experience buffer when the i-th interaction condition between the intelligent agent and the environment is met; when the number is greater than or equal to a fourth threshold, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience.

[0024] In combination with the second aspect and the above implementations, in some possible implementations, the model optimization device further includes:

[0025] The experience updating unit is used to store the strategy experience generated when the agent completes the i-th interaction with the environment into the experience buffer when the reward difference is not within the preset difference interval, so as to update the experience buffer.

[0026] In combination with the second aspect and the above implementation, in some possible implementations, the reward function is:

[0027]

[0028] in, represents the environmental reward, R(s,a) represents the original reward, ξ represents the weight, TV(π,π k )(s) represents the total variation distance, s represents the state, a represents the action, (s, a) represents the state-action pair, π represents the new strategy obtained after optimizing the strategy adopted by the agent, π k Indicates the current policy.

[0029] In a third aspect, an electronic device is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the model optimization method in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, a computer program product is provided, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the model optimization method in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the model optimization method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic flow chart of a model optimization method provided in an embodiment of the present application is shown;

[0033] Figure 2 A comparison diagram of the environmental rewards obtained by the HiPPO algorithm provided by the present application and the existing algorithm is shown;

[0034] Figure 3 A schematic diagram of the structure of a model optimization device provided in an embodiment of the present application is shown;

[0035] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0038] Reinforcement learning has achieved remarkable success in many fields in recent years, such as video games, robot control, and multi-agent systems. However, in practical applications, the utilization of historical experience samples is still an important challenge facing reinforcement learning. In particular, in continuous control tasks, how to use historical experience samples to accelerate policy optimization has become a hot topic of research. For example, PPO (Proximal Policy Optimization) is one of the currently popular approximate policy optimization algorithms. Although it performs well in a variety of continuous tasks, its utilization of historical experience samples is still limited. Existing algorithms such as GePPO (Generalized Proximal Policy Optimization) and P3O (Proximal Policy Optimization with Parameter Sharing) have utilized historical experience data to some extent, but due to the strict conditions of policy approximation, they have failed to fully exploit global historical experience data, resulting in low utilization of historical experience samples.

[0039] Based on the above problems, the present application provides a model optimization method, device, electronic device and computer-readable storage medium. The present application introduces a highlight replay mechanism to optimize proximal policy learning (PPO). By screening out high-quality historical policy experience from the historical policy experience generated by the previous interaction between the intelligent agent and the environment, and then using the high-quality historical policy experience and the reward function constructed by the total variation distance (TV) to optimize the strategy adopted by the intelligent agent, it can not only improve the utilization rate of historical experience samples in reinforcement learning, but also reduce the cost and time required for the interaction between the intelligent agent and the environment in reinforcement learning.

[0040] The following is an embodiment of a model optimization method provided in an embodiment of the present application.

[0041] Figure 1 A schematic flow chart of a model optimization method provided in an embodiment of the present application is shown, such as Figure 1As shown, the model optimization method provided in the embodiment of the present application is used for the optimization of the agent in reinforcement learning, and reinforcement learning is mainly composed of an agent, an environment, a state, an action, and a reward. The agent refers to the ontology of reinforcement learning, which acts as a learner or a decision maker; the environment refers to everything outside the reinforcement learning agent, which is mainly composed of a state set. The state is a data representing the environment, and the state set is all possible states in the environment. The action refers to the action that the agent can make, and the action set is all the actions that the agent can make. The reward refers to the positive / negative feedback signal obtained by the agent after performing an action, and the reward set is all the feedback information that the agent can obtain. The thinking process of how the agent chooses an action is called a strategy. After the agent performs an action, the environment will be converted to a new state, and a reward signal (positive reward or negative reward) will be given to the environment in the new state; then, the agent performs a new action according to a certain strategy based on the reward of the new state and the feedback of the environment. The above process is a way for the agent and the environment to interact through state, action, and reward. Through reinforcement learning, the agent can know what state it is in and what action it should take to get the maximum reward.

[0042] The above model optimization method includes the following schemes:

[0043] S110: When the i-th interaction condition between the agent and the environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, where i is a positive integer greater than or equal to 1.

[0044] In an exemplary embodiment, satisfying the i-th interaction condition between the agent and the environment indicates that the agent has the ability to interact with the environment for the i-th time, which means that the agent has completed the first i-1 interactions with the environment and is ready to start the i-th interaction. After the first i-1 interactions are completed, the agent has accumulated experience strategies for the first i-1 interactions with the environment. Among them, the experience strategies accumulated by the agent for the first i-1 interactions with the environment are called historical strategy experiences, and each historical experience strategy has a strategy experience attribute, which is used to measure the quality of strategy experience. For example, the strategy experience attribute can be used to measure which historical strategy experiences are high-quality historical strategy experiences (also called highlighted historical strategy experiences) and which historical strategy experiences are low-quality historical strategy experiences.

[0045] The agent stores the accumulated historical strategy experience of the previous i-1 interactions with the environment in the experience buffer, that is, the experience buffer stores the historical strategy experience adopted by the agent in the i-th interaction with the environment. Under the premise that the historical strategy experience is stored in the experience buffer, before the agent reinforcement learning model and the environment are ready to start the i-th interaction, based on the highlight replay mechanism, the target strategy experience (high-quality historical strategy experience or highlighted historical strategy experience) is screened out from the experience buffer according to the strategy experience attributes and pre-set rules. The screened target strategy experience has a greater contribution to the optimization of the strategy, and the target strategy experience constitutes the highlighted experience replay set, that is, the highlighted experience replay set includes high-quality historical strategy experience.

[0046] S120: Use the reward function and the target strategy experience to optimize the strategy adopted by the agent so that the agent interacts with the environment for the i-th time.

[0047] This application introduces a Highlight Replay mechanism to optimize proximal policy learning (PPO), that is, the algorithm used in the agent is improved. The improved algorithm is called HiPPO. The reward function is constructed in advance through the total variation distance. This application constrains the optimization of the strategy used by the agent through the reward function. The design of the reward function is as follows:

[0048] In actual optimization, directly dealing with constraints usually brings challenges. To overcome this problem, this application adopts an alternative penalty mechanism, that is, implicitly optimizing the algorithm through discounted penalties. In reinforcement learning, the reward function quantifies the immediate feedback obtained by the agent for performing a specific action in a given state. The reward function plays a vital role in guiding the agent to learn the optimal strategy and maximize the expected reward. This application integrates the constraint term into the reward function, specifically by using the TV distance as a negative reward. The TV distance is used to measure the difference between two state-action distributions, especially in the process of policy iteration or update, to quantify the degree of policy difference. Highlight historical policy experience is introduced in the calculation to provide a more comprehensive TV distance design. Incorporating TV distance as a penalty into the reward function can effectively consider the constraints of policy changes in the optimization target, while considering the changes in state distribution during the policy update process. Such constraints can promote smooth updates of policies and avoid instability caused by excessive updates, thereby helping the intelligent agent converge to a stable and efficient strategy faster.

[0049] Since the reward is affected by the TV distance, it is necessary to balance the impact of the penalty term on the environmental reward. The penalty term should not be too large, otherwise it may cause the strategy to continuously maximize the TV distance instead of learning how to maximize the environmental reward. Therefore, this application introduces the parameter ξ as a weight or penalty term for the TV distance to better adjust the reward function. Without this weight coefficient, the model may degenerate. By introducing a reward function, it is equivalent to introducing constraints when updating the strategy, which can prevent the strategy from jumping too much, and can ensure the smoothness and stability of the strategy change. The reward function is applied to the optimization process of the strategy adopted by the agent. The reward function (also called the penalty reward function) constructed in this application is defined as follows:

[0050]

[0051] in, represents the environmental reward or the modified original reward, R(s,a) represents the original reward, ξ represents the weight, TV(π,π k )(s) represents the total variation distance, s represents the state, a represents the action, (s, a) represents the state-action pair, π represents the new strategy (also called future strategy) obtained after optimizing the agent, and k represents the current strategy π k , which is the strategy currently being optimized, (s, a) is the strategy from π k is sampled from the following distribution.

[0052] Regarding the interaction between the agent and the environment; when the agent performs action a in state s, the environment will give an immediate feedback (reward) based on the result of the action, which depends on the goal of the current task. For example, in a robot control task, the reward function may be related to the distance, speed or posture of the agent; in a game environment, the reward may be related to the number of enemies defeated or resources collected.

[0053] About multi-step cumulative rewards (discounted rewards): Sometimes, the rewards in reinforcement learning are not limited to the immediate rewards of the current step, but the accumulated rewards of multiple time steps. This cumulative reward can weigh the impact of future rewards on current decisions through the discount factor γ. Therefore, the formula for R(s,a) is as follows:

[0054]

[0055] Among them, γ represents the discount factor, which is used to control the importance of future rewards in current decisions, r(s t ,a t ) represents immediate reward, and t represents time.

[0056] About TV(π,π k The formula for )(s) is as follows:

[0057]

[0058] Among them, π(s|a) represents the probability of strategy π taking action a in a given state s, π k (s|a) represents the strategy π k The probability of taking action a in a given state s, j represents the strategy πj sampled from the highlighted experience replay set H, π j (s|a) represents the probability that the policy πj sampled from the highlighted experience replay set H takes action a in state s.

[0059] By analyzing the effect of changes in the penalty term ξ on policy stability and performance, we find that if ξ is too large, it may over-constrain policy updates, making them too conservative, which may slow down the learning process. On the contrary, if ξ is too small, it may not be able to stabilize updates, resulting in faster learning but more unstable performance. Therefore, in a specific environment, a reasonable choice of ξ can enhance the effect of the TV distance penalty, thereby suppressing large changes in policy updates. This will lead to more stable policy updates because the penalty term constrains drastic changes in the policy and smooths the learning process.

[0060] After obtaining the highlighted experience replay set, the target strategy experience is obtained from the highlighted experience replay set, and the strategy data of the i-th interaction between the agent and the environment (which can be understood as the current strategy data), the reward function and the target strategy experience are used to optimize the strategy adopted by the agent. The agent uses the optimized strategy to perform corresponding actions, thereby interacting with the environment for the i-th time.

[0061] Before optimization, the objective function of the HiPPO algorithm is defined. Since the objective function of the PPO algorithm is to update the strategy by minimizing a clipped proxy objective function, the HiPPO algorithm is based on the PPO algorithm and is enhanced. Therefore, constrained optimization is introduced based on the playback of high-quality historical strategy experience. The objective function of HiPPO can be expressed as:

[0062]

[0063] Among them, π j represents the strategy sampled from the highlighted experience replay set H, π represents the current strategy, s represents the state, a represents the action, A πk (s,a) represents the advantage function, ∈ represents the scope of pruning, and dπj represents a strategy π j The state-action distribution under .

[0064] The core of the optimization is the objective function of HiPPO mentioned above. At the same time, the optimization process is constrained by the policy difference penalty term, so that the updated policy does not deviate too far from the original policy, thereby maintaining the stability of the policy.

[0065] The steps for optimizing the strategy in the optimization agent include:

[0066] 1. Introducing highlighted historical strategy experience: One of the innovations of the HiPPO algorithm is to select highlighted historical strategy experience from historical strategy experience and introduce these highlighted historical strategy experience in the optimization process, so as to make better use of previous highlighted historical strategy experience.

[0067] 2. Calculate the advantage function: By interacting the current strategy and the historical strategy, calculate the advantage function A of each action πk (s, a). In the HiPPO algorithm, this step is calculated based on the generalized advantage estimation (GAE).

[0068] 3. Objective function clipping: During the policy optimization process, it is necessary to ensure that the new policy (optimized policy) does not deviate too much from the old policy. The HiPPO algorithm limits the update range through clipping, which ensures that the policy changes are within a certain range when the update is executed.

[0069] S130: When the agent completes the i-th interaction with the environment, obtain the environment reward obtained through the reward function;

[0070] S140: Determine whether the reward difference between the environment reward and the preset environment reward is within the preset difference interval; if yes, execute S150; if no, continue the i+1th interaction between the agent and the environment, and repeat S110 to S140 until the reward difference is within the preset difference interval, which can also be understood as taking the i+1th interaction between the agent and the environment as the ith interaction between the agent and the environment, and returning to execute S110 to S140 until the reward difference is within the preset difference interval;

[0071] S150: Stop optimizing the strategy adopted by the agent.

[0072] When the agent interacts with the environment for the i-th time, determine whether the i-th interaction between the agent and the environment is completed. If the i-th interaction between the agent and the environment is completed, obtain the environment reward obtained by the reward function, and then calculate the reward difference between the environment reward and the preset environment reward, and then determine whether the reward difference is within the preset difference interval. Among them, the preset environment reward is set according to the environment reward obtained by the existing algorithm, for example, the preset environment reward is the maximum value of the environment reward obtained by the existing algorithm. Existing algorithms include PPO, GePPO, TRPO (Trust Region Policy Optimization), Dual-clip PPO (Dual Clip Proximal Policy Optimization), etc.

[0073] Determine whether the reward difference is within a preset difference interval, that is, determine whether the HiPPO algorithm provided in this application is better than the existing algorithm. If the reward difference is within the preset difference interval, it means that the strategy adopted by the agent is optimized by using the reward function and the target strategy experience, so that after the agent completes the i-th interaction with the environment, the HiPPO algorithm provided in this application is already better than the existing algorithm, and then stop optimizing the strategy adopted by the agent; if the reward difference is not within the preset difference interval, it means that the strategy adopted by the agent is optimized by using the reward function and the target strategy experience, so that after the agent completes the i-th interaction with the environment, whether the HiPPO algorithm provided in this application is still better than the existing algorithm, then it is necessary to continue to optimize the strategy adopted by the agent, and then take the i+1th interaction between the agent and the environment as the i-th interaction between the agent and the environment, and return to execute S110 to continue optimizing the strategy adopted by the agent, that is, continue to execute S110 to S140 until it is determined that the reward difference is within the preset difference interval, and then stop optimizing the strategy adopted by the agent.

[0074] like Figure 2 As shown, Figure 2 The figure shows the comparison results of the environmental rewards obtained by the HiPPO algorithm provided by this application and the existing algorithm. Figure 2The data in it are the environmental reward data obtained by the HiPPO algorithm provided in this application and the existing algorithms (PPO, GePPO, TRPO, Dual-clip PPO) in multiple MuJoCo (Multi-Joint dynamics with Contact) environments. Humanoid-v3 represents the environment simulating a humanoid robot, Ant-v3 represents the environment simulating a four-legged insect robot, HalfCheetah-v3 represents the environment simulating a two-group robot, Hopper-v3 represents the environment simulating a three-legged jumping robot, Walker2d-v3 represents the environment simulating a two-legged walking robot, and Swimmer-v3 represents the environment simulating a swimming robot. "Mean±Standard Deviation Rewards at Every 0.2MSteps" means that the average reward μ (environmental reward) and its standard deviation σ are reported every 0.2 million steps (i.e., every 200,000 steps). By comparing the environmental rewards obtained by the HiPPO algorithm with those obtained by the existing algorithms, for any of the above six environments, the performance of the HiPPO algorithm provided by this application is better than that of PPO, GePPO, TRPO, and Dual-clip PPO, and it has shown faster learning speed and better final performance in multiple simulated continuous control tasks, that is, the HiPPO algorithm provided by this application has better performance. The HiPPO algorithm provided by this application can be applied to fields such as autonomous driving and robot control.

[0075] The present application adopts the method of screening the historical strategy experience in the experience buffer according to the strategy experience attribute used to measure the quality of the strategy experience when the i-th interaction condition between the intelligent agent and the environment is met, obtaining the target experience strategy, and optimizing the strategy adopted by the intelligent agent by using the reward function and the target strategy experience constructed by the total variation distance, so that the intelligent agent interacts with the environment for the i-th time, and when the i-th interaction between the intelligent agent and the environment is completed, obtaining the environment reward obtained by the reward function, and stopping optimizing the strategy adopted by the intelligent agent when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, and continuing to optimize the strategy adopted by the intelligent agent when the reward difference is not within the preset difference interval. The invention adopts a technical solution that optimizes the strategy adopted by the agent by utilizing the current strategy data, high-quality historical strategy experience and the reward function constructed by the TV distance, which can not only improve the utilization rate of historical experience samples in reinforcement learning, but also reduce the cost and time required for the interaction between the agent and the environment in reinforcement learning, and reduce the dependence on the actual environment interaction data in the strategy learning process, which is conducive to improving the efficiency and speed of the agent for strategy learning. In addition, the introduced reward function can use the strategy difference as a negative reward, thereby limiting the amplitude of the strategy update, which is conducive to ensuring the stability of the strategy.

[0076] The following Figure 1 The specific implementation methods of each step in the illustrated embodiment are described as follows:

[0077] In a possible implementation, the strategy experience attribute includes a strategy experience reward, which is the sum of the environment rewards obtained when the agent adopts a strategy to interact with the environment. For example, historical strategy experience A is the strategy experience generated when the agent adopts historical strategy A to interact with the environment. Then, the strategy experience reward corresponding to historical strategy experience A is the sum of the environment rewards obtained when the agent adopts historical strategy A to interact with the environment.

[0078] Understanding of environmental rewards and: The environmental rewards and mentioned in HiPPO can be understood as cumulative rewards, which reflect the performance of a strategy in a specific period of time. Environmental rewards and are used as weights for selecting target strategy experience from historical strategy experience, that is, environmental rewards and are assigned to each historical strategy experience. The higher the environmental rewards and historical strategy experience, the more likely it is to be used as the target strategy experience. π represents the environmental reward and, ρ π The calculation formula is as follows:

[0079]

[0080] The meaning of the above formula is: when the agent uses strategy π to interact with the environment, the immediate reward r(s) obtained in the time range from time t = 0 to Tt ,a t ), that is, the cumulative sum of environmental rewards and .

[0081] Since the formula used to calculate the “environmental reward sum” is ρ π The calculation formula for calculating the "environmental reward" is the calculation formula of R(s,a), so the "environmental reward and" mentioned in this application is different from the "environmental reward".

[0082] The above method filters the historical strategy experience in the experience buffer according to the strategy experience attribute, and obtains the target strategy experience including the following schemes:

[0083] The historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to the first threshold is used as the target strategy experience, so as to obtain high-quality historical strategy experience.

[0084] For example, the experience buffer includes historical strategy experience 1-historical strategy experience 50, wherein historical strategy experience with strategy experience reward greater than or equal to the first threshold includes historical strategy experience 15-historical strategy experience 30, then historical strategy experience 15-historical strategy experience 30 is used as the target strategy experience.

[0085] In one possible implementation, the policy experience attribute includes an advantage estimate, which is obtained by generalized advantage estimation. Generalized advantage estimation is a method for estimating the advantage function in the policy gradient algorithm, which can more accurately capture the change in benefits brought about by policy changes. In reinforcement learning, the advantage function measures the superiority of an action relative to other actions. The advantage estimate is an estimate of the advantage function, which tells the agent how much additional reward the current action can bring compared to other possible actions, that is, the additional reward that the current action of the agent can bring compared to other possible actions is the above advantage estimate.

[0086] The above method filters the historical strategy experience in the experience buffer according to the strategy experience attribute, and obtains the target strategy experience including the following schemes:

[0087] The historical strategy experience in the experience buffer whose advantage estimate is greater than or equal to the second threshold is used as the target strategy experience, so as to obtain high-quality historical strategy experience.

[0088] For example, the experience buffer includes historical strategy experiences 1 to 50, wherein historical strategy experiences in the experience buffer whose advantage estimates are greater than or equal to the second threshold include historical strategy experiences 1 to 3 and historical strategy experiences 32 to 36. Then, historical strategy experiences 1 to 3 and historical strategy experiences 32 to 36 are used as target strategy experiences.

[0089] In one possible implementation, the strategy experience attribute includes state similarity, where the state similarity is the similarity between a first state and a second state, where the first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using a historical strategy.

[0090] The above method filters the historical strategy experience in the experience buffer according to the strategy experience attribute, and obtains the target strategy experience including the following schemes:

[0091] The historical strategy experience in the experience buffer whose state similarity is greater than or equal to the third threshold is used as the target strategy experience, so as to obtain high-quality historical strategy experience.

[0092] For example, the experience buffer includes historical strategy experiences 1-50, wherein the historical strategy experiences in the experience buffer whose state similarity is greater than or equal to the third threshold include historical strategy experiences 45-50. Then, historical strategy experiences 45-50 are used as target strategy experiences.

[0093] In a possible implementation, the strategy experience attribute includes at least one of a strategy experience reward, an advantage estimate, and a state similarity. The above-mentioned screening of the historical strategy experience in the experience buffer according to the strategy experience attribute to obtain the target strategy experience includes at least one of the following schemes:

[0094] The historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to the first threshold is used as the target strategy experience;

[0095] The historical strategy experience in the experience buffer whose advantage estimate is greater than or equal to the second threshold is used as the target strategy experience;

[0096] The historical strategy experience in the experience buffer whose state similarity is greater than or equal to the third threshold is used as the target strategy experience.

[0097] In a possible implementation, when the i-th interaction condition between the agent and the environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including the following schemes:

[0098] When the i-th interaction condition between the agent and the environment is met, the number of historical strategy experiences in the experience buffer is obtained;

[0099] When the number is greater than or equal to the fourth threshold, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience.

[0100] The fourth threshold includes two meanings. The first meaning is that the fourth threshold is used to measure whether the experience buffer is filled with historical policy experience. The second meaning is that the fourth threshold is used to indicate that the experience buffer stores historical policy experience, but the historical policy experience does not fill the experience buffer. For the first meaning, when the i-th interaction condition between the intelligent agent and the environment is met, if the number of historical policy experiences in the experience buffer is greater than or equal to the fourth threshold, it means that the experience buffer has been filled with historical policy experience, then the target policy experience can be screened, that is, when the experience buffer has been filled with historical policy experience, the target policy experience is screened out, which is conducive to obtaining more high-quality historical policy experience. For the second meaning, when the i-th interaction condition between the intelligent agent and the environment is met, if the number of historical policy experiences in the experience buffer is greater than or equal to the fourth threshold, it means that the experience buffer stores historical policy experience, but the experience buffer is not filled, then the target policy experience can be screened, that is, when the experience buffer stores historical policy experience, but the experience buffer is not filled, then the target policy experience can be screened out, which is conducive to improving the optimization efficiency of the strategy.

[0101] In a possible implementation, the above model optimization method also includes the following scheme:

[0102] When the reward difference is not within the preset difference interval, the strategy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer to update the experience buffer.

[0103] After the agent's strategy is optimized by using the reward function and the target strategy experience, so that the agent completes the i-th interaction with the environment, if the reward difference between the environment reward obtained by the reward function and the preset environment reward is not within the preset difference interval, the agent's strategy needs to be optimized. In order to ensure that the historical strategy experience stored in the experience buffer is the latest, when the reward difference is not within the preset difference interval, the strategy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer, thereby updating the historical strategy experience already stored in the experience buffer. Among them, the strategy experience generated when the agent completes the i-th interaction with the environment stored in the experience buffer carries the strategy experience attribute. If the experience buffer is full, the earliest stored historical strategy experience is first deleted, and then the strategy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer together with the environment reward. If the experience buffer is not full, the strategy experience generated when the agent completes the i-th interaction with the environment is directly stored in the experience buffer together with the environment reward, without the need to delete the historical strategy experience. By updating the historical strategy experience in the experience buffer, it can be ensured that the historical strategy experience in the experience buffer is always the latest generated, which can enable the intelligent agent to better cope with environmental challenges and help improve learning efficiency and final performance.

[0104] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0105] Figure 3 A schematic diagram of the structure of a model optimization device provided in an embodiment of the present application is shown. Figure 3 As shown, the model optimization device 300 includes:

[0106] The experience screening module 310 is used to screen the historical strategy experience in the experience buffer according to the strategy experience attribute to obtain the target strategy experience when the i-th interaction condition between the agent and the environment is met; wherein the strategy experience attribute is used to measure the quality of the strategy experience, and the experience buffer stores the historical strategy experience adopted by the agent in the i-th interaction with the environment before, i being a positive integer greater than or equal to 1;

[0107] The model optimization module 320 is used to optimize the strategy adopted by the agent using the reward function and the target strategy experience so that the agent interacts with the environment for the i-th time; wherein the reward function is constructed by the total variation distance; when the i-th interaction between the agent and the environment is completed, the environment reward obtained by the reward function is obtained; when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, the optimization of the strategy adopted by the agent is stopped; when the reward difference is not within the preset difference interval, the agent continues to interact with the environment for the i+1th time, and the above steps are repeated until the reward difference is within the preset difference interval.

[0108] In a possible implementation, the strategy experience attribute includes a strategy experience reward, and the strategy experience reward is the sum of the environment rewards obtained when the agent adopts the strategy to interact with the environment. The experience screening module 310 includes:

[0109] The first screening unit is configured to use the historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

[0110] In a possible implementation, the strategy experience attribute includes an advantage estimate, and the advantage estimate is obtained by generalized advantage estimation. The experience screening module 310 includes:

[0111] The second screening unit is used to use the historical strategy experience in the experience buffer whose advantage estimate is greater than or equal to a second threshold as the target strategy experience.

[0112] In a possible implementation, the strategy experience attribute includes state similarity, and the state similarity is the similarity between a first state and a second state, the first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using the historical strategy, and the experience screening module 310 includes:

[0113] The third screening unit is used to use the historical strategy experience in the experience buffer whose state similarity is greater than or equal to a third threshold as the target strategy experience.

[0114] In one possible implementation, the experience screening module 310 is specifically used to obtain the number of historical strategy experiences in the experience buffer when the i-th interaction condition between the intelligent agent and the environment is met; when the number is greater than or equal to a fourth threshold, the historical strategy experiences in the experience buffer are screened according to the strategy experience attributes to obtain the target strategy experience.

[0115] In a possible implementation, the model optimization device 300 further includes:

[0116] The experience updating unit is used to store the strategy experience generated when the agent completes the i-th interaction with the environment into the experience buffer when the reward difference is not within the preset difference interval, so as to update the experience buffer.

[0117] In a possible implementation, the reward function is:

[0118]

[0119] in, represents the environmental reward, R(s,a) represents the original reward, ξ represents the weight, TV(π,π k )(s) represents the total variation distance, s represents the state, a represents the action, (s, a) represents the state-action pair, π represents the new strategy obtained after optimizing the strategy adopted by the agent, π k Indicates the current policy.

[0120] It should be noted that the model optimization device provided in the above embodiment only uses the division of the above functional modules as an example when executing the model optimization method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model optimization device provided in the above embodiment belongs to the same concept as the model optimization method embodiment. Therefore, for details not disclosed in the device embodiment of the present application, please refer to the above-mentioned model optimization method embodiment of the present application, which will not be repeated here.

[0121] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0122] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. Figure 4 As shown, the electronic device 400 includes: a memory 401 and a processor 402, wherein the memory 401 stores an executable program code 4011, and the processor 402 is used to call and execute the executable program code 4011 to perform a model optimization method.

[0123] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be corresponded to, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0124] When each functional module is divided according to each function, the electronic device may include: an experience screening module, a model optimization module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0125] The electronic device provided in this embodiment is used to execute the above-mentioned model optimization method, and thus can achieve the same effect as the above-mentioned implementation method.

[0126] When an integrated unit is used, the electronic device may include a processing module and a storage module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device in executing related program codes and data.

[0127] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0128] This embodiment also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a model optimization method in the above-mentioned embodiment.

[0129] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a model optimization method in the above-mentioned embodiment.

[0130] In addition, the electronic device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the electronic device may include a connected processor and a memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor may call and execute instructions so that the chip executes a model optimization method in the above-mentioned embodiments.

[0131] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding model optimization method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding model optimization method provided above, and will not be repeated here.

[0132] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0133] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model optimization method, characterized in that: The model optimization method comprises: When the i-th interaction condition between the agent and the environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience; wherein the strategy experience attribute is used to measure the quality of the strategy experience, and the experience buffer stores the historical strategy experience adopted by the agent in the i-th interaction with the environment before, i is a positive integer greater than or equal to 1; Using a reward function and the target strategy experience to optimize the strategy adopted by the agent so that the agent interacts with the environment for the i-th time; wherein the reward function is constructed by the total variation distance; When the agent completes the i-th interaction with the environment, obtaining an environmental reward obtained by the reward function; When the reward difference between the environment reward and the preset environment reward is within a preset difference interval, stop optimizing the strategy adopted by the agent; When the reward difference is not within the preset difference interval, the agent continues to interact with the environment for the (i+1)th time, and the above steps are repeated until the reward difference is within the preset difference interval.

2. The model optimization method according to claim 1, characterized in that: The strategy experience attribute includes a strategy experience reward, which is the environment reward and obtained when the agent adopts the strategy to interact with the environment; The historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: The historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to the first threshold is used as the target strategy experience.

3. The model optimization method according to claim 1, characterized in that: The strategy empirical attribute includes an advantage estimate, and the advantage estimate is obtained by generalized advantage estimation; The historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: The historical strategy experience in the experience buffer in which the advantage estimate is greater than or equal to a second threshold is used as the target strategy experience.

4. The model optimization method according to claim 1, characterized in that: The strategy experience attribute includes state similarity, which is the similarity between a first state and a second state, wherein the first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using the historical strategy; The historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience, including: The historical strategy experience in the experience buffer whose state similarity is greater than or equal to a third threshold is used as the target strategy experience.

5. The model optimization method according to claim 1, characterized in that: When the i-th interaction condition between the agent and the environment is met, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute, and the target strategy experience is obtained, including: When the i-th interaction condition between the agent and the environment is met, the amount of historical strategy experience in the experience buffer is obtained; When the number is greater than or equal to a fourth threshold, the historical strategy experience in the experience buffer is screened according to the strategy experience attribute to obtain the target strategy experience.

6. The model optimization method according to any one of claims 1 to 6, characterized in that: The model optimization method further comprises: When the reward difference is not within the preset difference interval, the strategy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer to update the experience buffer.

7. The model optimization method according to any one of claims 1 to 6, characterized in that: The reward function is: in, represents the environmental reward, R(s,a) represents the original reward, ξ represents the weight, TV(π,π k )(s) represents the total variation distance, s represents the state, a represents the action, (s, a) represents the state-action pair, π represents the new strategy obtained after optimizing the strategy adopted by the agent, π k Indicates the current policy.

8. A model optimization device, characterized in that: The model optimization device comprises: An experience screening module is used to screen the historical strategy experience in the experience buffer according to the strategy experience attribute to obtain the target strategy experience when the i-th interaction condition between the intelligent agent and the environment is met; wherein the strategy experience attribute is used to measure the quality of the strategy experience, and the experience buffer stores the historical strategy experience adopted by the intelligent agent in the i-th interaction with the environment before, i is a positive integer greater than or equal to 1; A model optimization module is used to optimize the strategy adopted by the agent using a reward function and the target strategy experience so that the agent interacts with the environment for the i-th time; wherein the reward function is constructed by the total variation distance; when the i-th interaction between the agent and the environment is completed, the environment reward obtained by the reward function is obtained; when the reward difference between the environment reward and the preset environment reward is within the preset difference interval, the optimization of the strategy adopted by the agent is stopped; when the reward difference is not within the preset difference interval, the agent continues to interact with the environment for the i+1th time, and the above steps are repeated until the reward difference is within the preset difference interval.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the model optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the model optimization method according to any one of claims 1 to 7 is implemented.

Images