Optimization method, device and equipment applied to robot control model and medium

Abstract

The application provides an optimization method and device applied to a robot control model, electronic equipment and a computer readable storage medium, relates to the technical field of reinforcement learning, and the optimization method provided by the application can filter high-quality historical policy experience from historical policy experience generated by past interaction between an agent and an environment, and then optimize the policy adopted by the agent by using the high-quality historical policy experience and a reward function constructed by a total variation distance, so that the utilization rate of historical experience samples in reinforcement learning can be improved, the cost and time required for interaction between the agent and the environment in reinforcement learning can be reduced, and the introduced reward function can take the policy difference as a negative reward, so that the amplitude of policy updating is limited, and the stability of the policy is ensured.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, and more specifically, to an optimization method, apparatus, electronic device, and computer-readable storage medium applied to robot control models in the field of reinforcement learning technology. Background Technology

[0002] Reinforcement learning (RL) has achieved significant success in several fields in recent years, such as video games, robot control, and multi-agent systems. However, in practical applications, the utilization rate of historical experience samples remains a significant challenge for reinforcement learning, especially in continuous control tasks. How to leverage historical experience samples to accelerate policy optimization has become a hot research topic. Therefore, improving the utilization rate of historical experience samples in reinforcement learning is an urgent problem to be solved. Summary of the Invention

[0003] This application provides an optimization method, apparatus, electronic device, and computer-readable storage medium for robot control models. This application can improve the utilization rate of historical experience samples in reinforcement learning and reduce the cost and time required for agent-environment interaction in reinforcement learning.

[0004] Firstly, an optimization method for robot control models is provided. This method includes: when the i-th interaction condition between the agent and the environment is met, filtering historical policy experiences in an experience buffer based on policy experience attributes to obtain target policy experience; wherein the policy experience attributes are used to measure the quality of policy experience, and the experience buffer stores historical policy experiences used by the agent in its previous i-th interaction with the environment, where i is a positive integer greater than or equal to 1; optimizing the policy adopted by the agent using a reward function and the target policy experience to enable the agent to perform the i-th interaction with the environment; wherein the reward function is constructed using total variation distance; when the i-th interaction between the agent and the environment is completed, obtaining the environmental reward obtained through the reward function; stopping the optimization of the policy adopted by the agent when the reward difference between the environmental reward and a preset environmental reward is within a preset difference interval; and continuing the (i+1)-th interaction between the agent and the environment when the reward difference is not within the preset difference interval, repeating the above steps until the reward difference is within the preset difference interval.

[0005] In the above technical solution, this application employs a method where, when the conditions for the i-th interaction between the agent and the environment are met, historical policy experiences in the experience buffer are filtered according to policy experience attributes used to measure the quality of policy experience to obtain a target policy experience. The policy adopted by the agent is then optimized using a reward function constructed through the total variation distance and the target policy experience, enabling the agent to perform the i-th interaction with the environment. Upon completion of the i-th interaction, the environmental reward obtained through the reward function is acquired. If the reward difference between the environmental reward and the preset environmental reward is within a preset difference range, the optimization of the agent's policy is stopped; otherwise, the optimization continues. The technical solution involves the agent interacting with the environment for the (i+1)th time, and repeating the above steps until the reward difference falls within a preset difference range. This approach optimizes the agent's policy using current policy data, high-quality historical policy experience, and a reward function constructed from the TV distance. This not only improves the utilization rate of historical experience samples in reinforcement learning but also reduces the cost and time required for the agent's interaction with the environment, as well as the dependence on actual environmental interaction data during policy learning. This improves the efficiency and speed of policy learning. Furthermore, the introduced reward function treats policy differences as negative rewards, thereby limiting the magnitude of policy updates and ensuring policy stability.

[0006] In conjunction with the first aspect, in some possible implementations, the policy experience attribute includes a policy experience reward, which is the sum of the environmental reward obtained by the agent when it interacts with the environment using a policy; the step of filtering historical policy experiences in the experience buffer according to the policy experience attribute to obtain the target policy experience includes: taking historical policy experiences in the experience buffer whose policy experience reward is greater than or equal to a first threshold as the target policy experience.

[0007] Combining the first aspect and the above implementation methods, in some possible implementation methods, the policy experience attribute includes an advantage estimate, which is obtained through generalized advantage estimation; the step of filtering historical policy experiences in the experience buffer according to the policy experience attribute to obtain the target policy experience includes: taking historical policy experiences in the experience buffer whose advantage estimate is greater than or equal to a second threshold as the target policy experience.

[0008] Combining the first aspect and the above implementation methods, in some possible implementation methods, the policy experience attribute includes state similarity, which is the similarity between a first state and a second state. The first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using a historical policy. The step of filtering the historical policy experience in the experience buffer according to the policy experience attribute to obtain the target policy experience includes: taking the historical policy experience in the experience buffer whose state similarity is greater than or equal to a third threshold as the target policy experience.

[0009] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the step of filtering historical policy experiences in the experience buffer according to the policy experience attribute to obtain the target policy experience when the i-th interaction condition between the agent and the environment is met includes: obtaining the number of historical policy experiences in the experience buffer when the i-th interaction condition between the agent and the environment is met; and filtering the historical policy experiences in the experience buffer according to the policy experience attribute to obtain the target policy experience when the number is greater than or equal to a fourth threshold.

[0010] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the optimization method further includes: when the reward difference is not within the preset difference range, storing the policy experience generated when the agent completes the i-th interaction with the environment into the experience buffer to update the experience buffer.

[0011] Combining the first aspect and the above implementation methods, in some possible implementations, the reward function is:

[0012] ;

[0013] in, Indicates environmental rewards, Let ξ represent the original reward and ξ represent the weight. Let s represent the total variation distance, s represent the state, a represent the action, (s, a) represent the state-action pair, and π represent the new policy obtained after optimizing the agent's policy. k Indicates the current strategy.

[0014] Secondly, an optimization device for robot control models is provided, the optimization device comprising:

[0015] An experience filtering module is used to filter historical policy experiences in the experience buffer according to policy experience attributes when the i-th interaction condition between the agent and the environment is met, so as to obtain target policy experience; wherein, the policy experience attributes are used to measure the quality of policy experience, and the experience buffer stores the historical policy experience used by the agent in the i-th previous interaction with the environment, where i is a positive integer greater than or equal to 1.

[0016] The model optimization module is used to optimize the policy adopted by the agent using a reward function and the experience of the target policy, so that the agent can interact with the environment for the i-th time. The reward function is constructed using the total variation distance. When the i-th interaction between the agent and the environment is completed, the environmental reward obtained through the reward function is acquired. When the reward difference between the environmental reward and a preset environmental reward is within a preset difference interval, the optimization of the agent's policy is stopped. When the reward difference is not within the preset difference interval, the agent continues to interact with the environment for the (i+1)-th time, and the above steps are repeated until the reward difference is within the preset difference interval.

[0017] In conjunction with the second aspect, in some possible implementations, the policy experience attribute includes a policy experience reward, which is the sum of environmental rewards obtained by the agent when it interacts with the environment using a policy. The experience screening module includes:

[0018] The first filtering unit is used to select historical strategy experiences in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

[0019] Combining the second aspect and the above implementation methods, in some possible implementation methods, the policy experience attribute includes advantage estimation, which is obtained through generalized advantage estimation, and the experience screening module includes:

[0020] The second filtering unit is used to select historical policy experiences in the experience buffer whose advantage estimate is greater than or equal to the second threshold as the target policy experience.

[0021] Combining the second aspect and the above implementation methods, in some possible implementation methods, the policy experience attribute includes state similarity, where the state similarity is the similarity between a first state and a second state, the first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes its interaction with the environment using historical strategies. The experience screening module includes:

[0022] The third filtering unit is used to select historical policy experiences in the experience buffer whose state similarity is greater than or equal to a third threshold as the target policy experience.

[0023] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the experience filtering module is specifically used to obtain the number of historical policy experiences in the experience buffer when the i-th interaction condition between the agent and the environment is met; when the number is greater than or equal to a fourth threshold, the historical policy experiences in the experience buffer are filtered according to the policy experience attributes to obtain the target policy experience.

[0024] In conjunction with the second aspect and the above-described implementation, in some possible implementations, the optimization device further includes:

[0025] An experience update unit is used to store the policy experience generated when the agent completes the i-th interaction with the environment into the experience buffer when the reward difference is not within the preset difference range, so as to update the experience buffer.

[0026] Combining the second aspect and the above implementation methods, in some possible implementations, the reward function is:

[0027] ;

[0028] in, Indicates environmental rewards, Let ξ represent the original reward and ξ represent the weight. Let s represent the total variation distance, s represent the state, a represent the action, (s, a) represent the state-action pair, and π represent the new policy obtained after optimizing the agent's policy. k Indicates the current strategy.

[0029] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to execute the optimization method applied to the robot control model as described in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to execute the optimization method applied to the robot control model as described in the first aspect or any possible implementation thereof.

[0031] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the optimization method applied to a robot control model as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0032] Figure 1 A schematic flowchart illustrating an optimization method for a robot control model provided in an embodiment of this application is shown.

[0033] Figure 2 The diagram shows a comparison of the environmental rewards obtained by the HiPPO algorithm provided in this application with those obtained by existing algorithms.

[0034] Figure 3 A schematic diagram of the structure of an optimization device for a robot control model provided in an embodiment of this application is shown;

[0035] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0036] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0037] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0038] Reinforcement learning has achieved remarkable success in various fields in recent years, such as video games, robot control, and multi-agent systems. However, in practical applications, the utilization rate of historical experience samples remains a significant challenge for reinforcement learning, especially in continuous control tasks. How to leverage historical experience samples to accelerate policy optimization has become a research hotspot. For example, Proximal Policy Optimization (PPO) is one of the currently popular approximate policy optimization algorithms. Although it performs well in various continuous tasks, its utilization of historical experience samples is still limited. Existing algorithms such as Generalized Proximal Policy Optimization (GePPO) and Proximal Policy Optimization with Parameter Sharing (P3O) utilize historical experience data to some extent, but due to the strict conditions of policy approximation, they fail to fully mine global historical experience data, resulting in low utilization of historical experience samples.

[0039] To address the aforementioned issues, this application provides an optimization method, apparatus, electronic device, and computer-readable storage medium for robot control models. This application introduces a highlight replay mechanism to optimize proximal policy learning (PPO). By selecting high-quality historical policy experiences from past interactions between the agent and the environment, and then utilizing these high-quality historical policy experiences and a reward function constructed from the total variation distance (TV), the policy adopted by the agent is optimized. This not only improves the utilization rate of historical experience samples in reinforcement learning but also reduces the cost and time required for agent-environment interaction during reinforcement learning.

[0040] The following is an embodiment of an optimization method for robot control models provided in this application.

[0041] Figure 1 This application provides an embodiment of an optimization method for a robot control model, as illustrated in the schematic flowchart. Figure 1As shown in the embodiments of this application, the optimization method provided is used for optimizing an agent in reinforcement learning. Reinforcement learning mainly consists of an agent, environment, state, action, and reward. The agent refers to the entity in reinforcement learning, acting as a learner or decision-maker; the environment refers to everything outside the agent, mainly composed of a set of states. A state is data representing the environment, and the state set is all possible states in the environment. An action refers to the action the agent can perform, and the action set is all the actions the agent can perform. A reward refers to the positive / negative feedback signal obtained by the agent after performing an action, and the reward set is all the feedback information the agent can obtain. The thought process by which the agent chooses an action is called a strategy. After the agent performs an action, the environment will transition to a new state, and the environment in this new state will provide a reward signal (positive or negative reward). Subsequently, the agent, based on the new state and the reward feedback from the environment, executes a new action according to a certain strategy. The above process is the way the agent and the environment interact through state, action, and reward. Through reinforcement learning, the agent can know what action it should take in what state to obtain the maximum reward.

[0042] The optimization methods applied to robot control models mentioned above include the following schemes:

[0043] S110: When the i-th interaction condition between the agent and the environment is met, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience, where i is a positive integer greater than or equal to 1.

[0044] In an exemplary embodiment, satisfying the condition for the i-th interaction between the agent and the environment indicates that the agent possesses the ability to perform the i-th interaction with the environment. This means that the agent has completed the first i-1 interactions with the environment and is ready to begin the i-th interaction. After the completion of the first i-1 interactions, the agent has accumulated experience strategies from the first i-1 interactions with the environment. These accumulated experience strategies from the first i-1 interactions with the environment are called historical strategy experiences, and each historical strategy has a strategy experience attribute. The strategy experience attribute is used to measure the quality of the strategy experience; for example, it can be used to determine which historical strategy experiences are high-quality (also known as highlighted historical strategy experiences) and which are low-quality historical strategy experiences.

[0045] The agent stores the accumulated historical policy experience from the previous i-1 interactions with the environment in an experience buffer. This buffer contains the historical policy experience used by the agent in its interactions with the environment up to the i-th time. Given this historical policy experience in the experience buffer, before the agent's reinforcement learning model begins its i-th interaction with the environment, a highlighting replay mechanism is used to filter target policy experiences (high-quality historical policy experiences or highlighted historical policy experiences) from the experience buffer based on policy experience attributes and pre-defined rules. The selected target policy experiences contribute more significantly to policy optimization, forming a highlighted experience replay set, which includes high-quality historical policy experiences.

[0046] S120: Optimize the policy adopted by the agent using the reward function and the experience of the target policy, so that the agent can interact with the environment for the i-th time.

[0047] This application optimizes proximal policy learning (PPO) by introducing a highlight replay mechanism, which improves the algorithm used in the agent. The improved algorithm is called HiPPO. The reward function is pre-constructed using the total variation distance. This application uses the reward function to constrain the optimization of the policy adopted by the agent. The design of the reward function is as follows:

[0048] In practical optimization, directly handling constraints often presents challenges. To overcome this issue, this application employs an alternative penalty mechanism: implicitly optimizing the algorithm through discount penalties. In reinforcement learning, the reward function quantifies the immediate feedback obtained by an agent performing a specific action in a given state. The reward function plays a crucial role in guiding the agent to learn the optimal policy and maximize the expected reward. This application integrates constraints into the reward function by using TV distance as a negative reward. TV distance measures the difference between two state-action distributions, particularly during policy iteration or updates, to quantify the degree of policy difference. Highlighting historical policy experience is incorporated into the calculation to provide a more comprehensive TV distance design. Incorporating TV distance as a penalty into the reward function effectively considers the constraints of policy changes on the optimization objective, while also taking into account changes in state distribution during policy updates. Such constraints promote smooth policy updates, avoiding instability caused by over-updates, thereby helping the agent converge to a stable and efficient policy more quickly.

[0049] Since the reward is influenced by the TV distance, it is necessary to balance the impact of the penalty term on the environmental reward. The penalty term should not be too large, otherwise the policy may continuously maximize the TV distance instead of learning how to maximize the environmental reward. Therefore, this application introduces a parameter ξ as a weight or penalty term for the TV distance to better adjust the reward function. Without this weight coefficient, the model may degenerate. By introducing the reward function, a constraint is introduced during policy updates, which can prevent the policy from jumping too much and ensure the smoothness and stability of policy changes. This reward function is applied to the optimization process of the policy adopted by the agent. The reward function (also known as the penalty reward function) constructed in this application is defined as follows:

[0050] ;

[0051] in, This refers to environmental rewards or the original rewards after modification. Let ξ represent the original reward and ξ represent the weight. Let π represent the total variation distance, s represent the state, a represent the action, (s, a) represent the state-action pair, π represent the new policy obtained after optimizing the agent (also known as the future policy), and k represent the current policy. k That is, the policy currently being optimized, (s, a) is the policy π. k The samples were obtained from the following distribution.

[0052] Regarding the interaction between the agent and the environment: When the agent performs action a in state s, the environment provides immediate feedback (reward) based on the result of the action. This feedback depends on the goal of the current task. For example, in robot control tasks, the reward function may be related to the distance, speed, or posture of the agent; in a game environment, the reward may be related to the number of enemies defeated or resources collected.

[0053] Regarding multi-step cumulative rewards (discounted rewards): Sometimes, rewards in reinforcement learning are not limited to the immediate reward of the current step, but rather accumulate over multiple time steps. This cumulative reward can be used to weigh the impact of future rewards on the current decision using a discount factor γ. Therefore, regarding The formula is as follows:

[0054] ;

[0055] Where γ represents the discount factor, used to control the importance of future rewards in the current decision, r(s) t ,a t ) represents an immediate reward, and t represents time.

[0056] about The formula is as follows:

[0057] ;

[0058] Where π(s|a) represents the probability that policy π will take action a given state s. k (s|a) represents strategy π k Let j represent the probability of taking action a given state s, and let j denote the policy π sampled from the highlighted experience replay set H. j , π j (s|a) represents the strategy π for sampling from the highlighted experience replay set H. j The probability of taking action a in state s.

[0059] Analyzing the impact of changes in the penalty term ξ on policy stability and performance reveals that if ξ is too large, it may over-constrain policy updates, making them too conservative and potentially slowing down the learning process. Conversely, if ξ is too small, it may lead to unstable updates, resulting in faster learning but more unstable performance. Therefore, in specific environments, appropriately choosing ξ can enhance the effect of the TV distance penalty, thereby suppressing large changes in policy updates. This will result in more stable policy updates because the penalty term constrains drastic policy changes, smoothing out the learning process.

[0060] After obtaining the highlighted experience replay set, the target policy experience is obtained from the highlighted experience replay set. The policy data of the agent's i-th interaction with the environment (which can be understood as the current policy data), the reward function, and the target policy experience are used to optimize the policy adopted by the agent. The agent uses the optimized policy to execute the corresponding action, thereby interacting with the environment for the i-th time.

[0061] Before optimization, the objective function of the HiPPO algorithm was defined. Since the PPO algorithm updates the policy by minimizing the objective function of a pruned agent, and the HiPPO algorithm is an enhancement based on the PPO algorithm, constrained optimization is introduced based on replaying high-quality historical policy experience. The objective function of HiPPO can be expressed as:

[0062]

[0063] Where, π j This represents the policy sampled from the highlighted experience replay set H, where π represents the current policy, s represents the state, and a represents the action. Represents the dominance function. Indicates the area to be cut. Represents a certain strategy π j State-action distribution.

[0064] The core of the optimization is the objective function of HiPPO mentioned above. At the same time, the optimization process is constrained by the policy difference penalty term, so that the updated policy will not deviate too far from the original policy, thereby maintaining the stability of the policy.

[0065] The steps involved in optimizing a policy within an agent include:

[0066] 1. Introducing historical highlighting strategy experience: One of the innovations of the HiPPO algorithm is to select historical highlighting strategy experience from historical strategy experience and introduce these historical highlighting strategy experience into the optimization process, so as to make better use of the previous historical highlighting strategy experience.

[0067] 2. Calculate the advantage function: By interacting with the current policy and historical policies, calculate the advantage function for each action. In the HiPPO algorithm, this step is calculated based on Generalized Advantage Estimation (GAE).

[0068] 3. Objective function pruning: During policy optimization, it is necessary to ensure that the new policy (optimized policy) does not deviate too much from the old policy. The HiPPO algorithm limits the update range by pruning, which ensures that the policy change is within a certain range when the update is performed.

[0069] S130: When the agent completes the i-th interaction with the environment, obtain the environmental reward obtained through the reward function;

[0070] S140: Determine whether the reward difference between the environmental reward and the preset environmental reward is within the preset difference range; if yes, execute S150; if no, continue the (i+1)th interaction between the agent and the environment, and repeat S110 to S140 until the reward difference is within the preset difference range. This can also be understood as treating the (i+1)th interaction between the agent and the environment as the ith interaction between the agent and the environment, and returning to execute S110 to S140 until the reward difference is within the preset difference range.

[0071] S150: Stop optimizing the strategy adopted by the agent.

[0072] In the case of the agent's i-th interaction with the environment, it is determined whether the i-th interaction is complete. If the i-th interaction is complete, the environmental reward obtained through the reward function is acquired. Then, the reward difference between the environmental reward and the preset environmental reward is calculated, and it is determined whether the reward difference falls within the preset difference range. The preset environmental reward is set based on environmental rewards obtained by existing algorithms; for example, the preset environmental reward is the maximum value among environmental rewards obtained by existing algorithms. Existing algorithms include PPO, GePPO, TRPO (Trust Region Policy Optimization), and Dual-clip PPO, etc.

[0073] The algorithm determines whether the reward difference is within a preset difference range, i.e., whether the HiPPO algorithm provided in this application is superior to existing algorithms. If the reward difference is within the preset difference range, it means that the strategy adopted by the agent has been optimized using the reward function and target policy experience, so that after the agent completes the i-th interaction with the environment, the HiPPO algorithm provided in this application is superior to existing algorithms, and the optimization of the agent's strategy is stopped. If the reward difference is not within the preset difference range, it means that after the agent completes the i-th interaction with the environment, the HiPPO algorithm provided in this application is still superior to existing algorithms, and the optimization of the agent's strategy needs to continue. The (i+1)-th interaction between the agent and the environment is taken as the i-th interaction between the agent and the environment, and the algorithm returns to execute S110 to continue optimizing the agent's strategy. That is, the algorithm continues to execute S110 to S140 until it is determined that the reward difference is within the preset difference range, at which point the optimization of the agent's strategy is stopped.

[0074] like Figure 2 As shown, Figure 2 The image shows a comparison of the environmental rewards obtained by the HiPPO algorithm provided in this application and existing algorithms. Figure 2The data in this document represents the environmental rewards obtained by the HiPPO algorithm provided in this application and existing algorithms (PPO, GePPO, TRPO, Dual-clip PPO) in multiple MuJoCo (Multi-Joint dynamics with Contact) environments. Humanoid-v3 represents an environment simulating a humanoid robot, Ant-v3 represents an environment simulating a quadrupedal insect robot, HalfCheetah-v3 represents an environment simulating a bipedal robot, Hopper-v3 represents an environment simulating a tripedal hopping robot, Walker2d-v3 represents an environment simulating a bipedal walking robot, and Swimmer-v3 represents an environment simulating a swimming robot. "Mean ± Standard Deviation Rewards at Every 0.2M Steps" refers to the reported average reward μ (environmental reward) and its standard deviation σ at every 0.2 million steps (i.e., every 200,000 steps). By comparing the environmental rewards obtained by the HiPPO algorithm with those of existing algorithms, for any one of the six environments mentioned above, the HiPPO algorithm provided in this application outperforms PPO, GePPO, TRPO, and Dual-clip PPO. It demonstrates faster learning speed and better final performance in multiple simulated continuous control tasks, indicating that the HiPPO algorithm provided in this application has superior performance. The HiPPO algorithm provided in this application can be applied to fields such as autonomous driving and robot control.

[0075] This application employs a method where, upon satisfying the i-th interaction condition between the agent and the environment, historical policy experiences in the experience buffer are filtered based on policy experience attributes used to measure the quality of policy experience to obtain a target policy experience. The agent's policy is then optimized using a reward function constructed through the total variation distance and the target policy experience, enabling the agent to perform the i-th interaction with the environment. Upon completion of the i-th interaction, the environmental reward obtained through the reward function is acquired. If the reward difference between the environmental reward and a preset environmental reward falls within a preset difference range, the optimization of the agent's policy is stopped. If the reward difference does not fall within the preset difference range, the interaction between the agent and the environment continues. The technical solution involves performing the (i+1)th interaction with the environment and repeating the above steps until the reward difference falls within a preset difference range. This approach optimizes the agent's policy using current policy data, high-quality historical policy experience, and a reward function constructed from the TV distance. This not only improves the utilization rate of historical experience samples in reinforcement learning but also reduces the cost and time required for the agent to interact with the environment and decreases the dependence on actual environmental interaction data during policy learning. This is beneficial for improving the agent's efficiency and speed in policy learning. Furthermore, the introduced reward function can treat policy differences as negative rewards, thereby limiting the magnitude of policy updates and ensuring policy stability.

[0076] The following are Figure 1 The specific implementation methods of each step in the illustrated embodiment will be explained below:

[0077] In one possible implementation, the policy experience attribute includes a policy experience reward, which is the sum of environmental rewards obtained by the agent when it interacts with the environment using the policy. For example, historical policy experience A is the policy experience generated by the agent after it interacts with the environment using historical policy A. Therefore, the policy experience reward corresponding to historical policy experience A is the sum of environmental rewards obtained by the agent when it interacts with the environment using historical policy A.

[0078] Understanding the Environmental Reward Sum: The environmental reward sum mentioned in HiPPO can be understood as a cumulative reward, reflecting the performance of a strategy over a specific period. The environmental reward sum serves as the weight for selecting target strategy experience from historical strategy experience; that is, an environmental reward sum is assigned to each historical strategy experience. Historical strategy experiences with higher environmental reward sums have a greater chance of being selected as target strategy experiences. (Through ρ...) π Represents environmental rewards and ρ π The calculation formula is as follows:

[0079] ;

[0080] The above formula means that when an agent interacts with the environment using policy π, the immediate reward r(s) obtained within the time range from t=0 to T is:t ,a t The cumulative sum of environmental rewards and incentives.

[0081] Since the formula used to calculate "environmental reward sum" is ρ π The calculation formula for "environmental reward" is as follows: The calculation formula is different from the "environmental reward" mentioned in this application.

[0082] The above-mentioned filtering of historical policy experiences in the experience buffer based on policy experience attributes yields target policy experiences including the following schemes:

[0083] Historical strategy experiences with strategy experience rewards greater than or equal to the first threshold in the experience buffer are used as target strategy experiences, thus obtaining high-quality historical strategy experiences.

[0084] For example, if the experience buffer includes historical strategy experience 1 to historical strategy experience 50, and the historical strategy experience with a strategy experience reward greater than or equal to the first threshold includes historical strategy experience 15 to historical strategy experience 30, then historical strategy experience 15 to historical strategy experience 30 will be used as the target strategy experience.

[0085] In one possible implementation, the policy experience attribute includes advantage estimation, which is obtained through generalized advantage estimation. Generalized advantage estimation is a method used to estimate the advantage function in policy gradient algorithms, and it can more accurately capture the changes in reward resulting from policy changes. In reinforcement learning, the advantage function measures the superiority of an action relative to other actions. Advantage estimation is an estimate of the advantage function; it tells the agent how much additional reward the current action will bring compared to other possible actions. In other words, the additional reward that the agent's current action can bring compared to other possible actions is the aforementioned advantage estimation.

[0086] The above-mentioned filtering of historical policy experiences in the experience buffer based on policy experience attributes yields target policy experiences including the following schemes:

[0087] Historical policy experiences in the experience buffer with an advantage estimate greater than or equal to the second threshold are used as target policy experiences, thus obtaining high-quality historical policy experiences.

[0088] For example, the experience buffer includes historical strategy experience 1 to historical strategy experience 50. Among them, the historical strategy experiences in the experience buffer whose advantage estimate is greater than or equal to the second threshold include historical strategy experience 1 to historical strategy experience 3 and historical strategy experience 32 to historical strategy experience 36. Then, historical strategy experience 1 to historical strategy experience 3 and historical strategy experience 32 to historical strategy experience 36 are taken as target strategy experiences.

[0089] In one possible implementation, the policy experience attribute includes state similarity, which is the similarity between the first state and the second state. The first state is the state of the agent before the i-th interaction with the environment, and the second state is the state of the agent when the interaction with the environment is completed using the historical policy.

[0090] The above-mentioned filtering of historical policy experiences in the experience buffer based on policy experience attributes yields target policy experiences including the following schemes:

[0091] Historical policy experiences with state similarity greater than or equal to the third threshold in the experience buffer are used as target policy experiences, thus obtaining high-quality historical policy experiences.

[0092] For example, if the experience buffer includes historical strategy experiences 1 to 50, and the historical strategy experiences in the experience buffer with a state similarity greater than or equal to the third threshold include historical strategy experiences 45 to 50, then historical strategy experiences 45 to 50 will be used as the target strategy experiences.

[0093] In one possible implementation, the policy experience attribute includes at least one of policy experience reward, advantage estimation, and state similarity. The above-mentioned filtering of historical policy experiences in the experience buffer based on the policy experience attribute yields the target policy experience, including at least one of the following schemes:

[0094] Historical strategy experiences in the experience buffer whose strategy experience reward is greater than or equal to the first threshold are used as target strategy experiences;

[0095] Historical policy experiences in the experience buffer with an advantage estimate greater than or equal to the second threshold are used as the target policy experience;

[0096] Historical policy experiences in the experience buffer with a state similarity greater than or equal to the third threshold are used as target policy experiences.

[0097] In one possible implementation, when the i-th interaction condition between the agent and the environment is met, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience, including the following schemes:

[0098] When the i-th interaction condition between the agent and the environment is met, obtain the number of historical policy experiences in the experience buffer.

[0099] When the number is greater than or equal to the fourth threshold, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience.

[0100] The fourth threshold has two meanings. First, it measures whether the experience buffer is full of historical policy experience. Second, it indicates that the experience buffer contains historical policy experience, but it is not full. For the first meaning, when the i-th interaction condition between the agent and the environment is met, if the number of historical policy experiences in the experience buffer is greater than or equal to the fourth threshold, it means the experience buffer is full, and target policy experience filtering can be performed. That is, when the experience buffer is full, selecting target policy experience is beneficial for obtaining more high-quality historical policy experience. For the second meaning, when the i-th interaction condition between the agent and the environment is met, if the number of historical policy experiences in the experience buffer is greater than or equal to the fourth threshold, it means the experience buffer contains historical policy experience, but it is not full, and target policy experience filtering can be performed. That is, when the experience buffer contains historical policy experience, but it is not full, selecting target policy experience is beneficial for improving the optimization efficiency of the policy.

[0101] In one possible implementation, the optimization method applied to the robot control model described above also includes the following schemes:

[0102] When the reward difference is not within the preset difference range, the policy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer to update the experience buffer.

[0103] After optimizing the agent's policy using a reward function and target policy experience, if the difference between the environment reward obtained through the reward function and the preset environment reward is not within a preset difference range after the agent completes the i-th interaction with the environment, further optimization of the agent's policy is required. To ensure that the historical policy experience stored in the experience buffer is the most recently generated, when the reward difference is not within the preset difference range, the policy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer, thus updating the historical policy experience already stored in the experience buffer. The policy experience generated when the agent completes the i-th interaction with the environment and stored in the experience buffer carries a policy experience attribute. If the experience buffer is full, the oldest stored historical policy experience is deleted first, and then the policy experience generated when the agent completes the i-th interaction with the environment and the environment reward are stored together in the experience buffer. If the experience buffer is not full, the policy experience generated when the agent completes the i-th interaction with the environment and the environment reward are directly stored in the experience buffer without deleting historical policy experience. By updating the historical policy experience in the experience buffer, it can be ensured that the historical policy experience in the experience buffer is always the latest generated, which enables the agent to better cope with environmental challenges and is conducive to improving learning efficiency and final performance.

[0104] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0105] Figure 3 This illustration shows a structural schematic diagram of an optimization device for a robot control model provided in an embodiment of this application, as shown below. Figure 3 As shown, the optimization device 300 includes:

[0106] The experience filtering module 310 is used to filter the historical policy experience in the experience buffer according to the policy experience attribute when the i-th interaction condition between the agent and the environment is met, so as to obtain the target policy experience; wherein, the policy experience attribute is used to measure the quality of the policy experience, and the experience buffer stores the historical policy experience used by the agent in the i-th previous interaction with the environment, where i is a positive integer greater than or equal to 1.

[0107] The model optimization module 320 is used to optimize the policy adopted by the agent using a reward function and the experience of the target policy, so that the agent can interact with the environment for the i-th time; wherein, the reward function is constructed by the total variation distance; when the agent completes the i-th interaction with the environment, the environmental reward obtained by the reward function is obtained; when the reward difference between the environmental reward and the preset environmental reward is within a preset difference interval, the optimization of the policy adopted by the agent is stopped; when the reward difference is not within the preset difference interval, the agent continues to perform the (i+1)-th interaction with the environment, and the above steps are repeated until the reward difference is within the preset difference interval.

[0108] In one possible implementation, the policy experience attribute includes a policy experience reward, which is the sum of the environmental rewards obtained by the agent when it interacts with the environment using a policy. The experience screening module 310 includes:

[0109] The first filtering unit is used to select historical strategy experiences in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

[0110] In one possible implementation, the policy experience attribute includes an advantage estimate, which is obtained through generalized advantage estimation. The experience screening module 310 includes:

[0111] The second filtering unit is used to select historical policy experiences in the experience buffer whose advantage estimate is greater than or equal to the second threshold as the target policy experience.

[0112] In one possible implementation, the policy experience attribute includes state similarity, which is the similarity between a first state and a second state. The first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes its interaction with the environment using a historical policy. The experience screening module 310 includes:

[0113] The third filtering unit is used to select historical policy experiences in the experience buffer whose state similarity is greater than or equal to a third threshold as the target policy experience.

[0114] In one possible implementation, the experience filtering module 310 is specifically used to obtain the number of historical policy experiences in the experience buffer when the i-th interaction condition between the agent and the environment is met; when the number is greater than or equal to a fourth threshold, the historical policy experiences in the experience buffer are filtered according to the policy experience attributes to obtain the target policy experience.

[0115] In one possible implementation, the optimization device 300 further includes:

[0116] An experience update unit is used to store the policy experience generated when the agent completes the i-th interaction with the environment into the experience buffer when the reward difference is not within the preset difference range, so as to update the experience buffer.

[0117] In one possible implementation, the reward function is:

[0118] ;

[0119] in, Indicates environmental rewards, Let ξ represent the original reward and ξ represent the weight. Let s represent the total variation distance, s represent the state, a represent the action, (s, a) represent the state-action pair, and π represent the new policy obtained after optimizing the agent's policy. k Indicates the current strategy.

[0120] It should be noted that the optimization device provided in the above embodiments, when executing the optimization method applied to the robot control model, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the optimization device provided in the above embodiments and the optimization method embodiments applied to the robot control model belong to the same concept. Therefore, for details not disclosed in the device embodiments of this application, please refer to the above embodiments of the optimization method applied to the robot control model of this application, which will not be repeated here.

[0121] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0122] Figure 4 This application provides a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes a memory 401 and a processor 402. The memory 401 stores executable program code 4011, and the processor 402 is used to call and execute the executable program code 4011 to perform an optimization method applied to a robot control model.

[0123] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0124] When dividing the electronic device into functional modules corresponding to various functions, the device may include: an experience screening module, a model optimization module, etc. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0125] The electronic device provided in this embodiment is used to execute the above-described optimization method applied to a robot control model, and thus can achieve the same effect as the above-described implementation method.

[0126] When using integrated units, the electronic device may include a processing module and a storage module. The processing module is used to control and manage the actions of the electronic device. The storage module is used to support the execution of relevant program code and data by the electronic device.

[0127] The processing module may be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0128] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement an optimization method for a robot control model as described in the above embodiment.

[0129] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement an optimization method for a robot control model as described in the above embodiment.

[0130] In addition, the electronic device provided in the embodiments of this application may specifically be a chip, component or module. The electronic device may include a connected processor and a memory. The memory is used to store instructions. When the electronic device is running, the processor may call and execute the instructions to make the chip execute an optimization method applied to a robot control model in the above embodiments.

[0131] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding optimization method for robot control model provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding optimization method for robot control model provided above, and will not be repeated here.

[0132] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0133] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An optimization method applied to robot control models, characterized in that, The optimization method includes: When the conditions for the i-th interaction between the agent and the environment are met, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience; wherein, the agent is a robot, the environment refers to everything outside the reinforcement learning agent, and is composed of a set of states, wherein a state is a piece of data representing the environment, the set of states includes all possible states in the environment, the policy experience attribute is used to measure the quality of the policy experience, and the experience buffer stores the historical policy experience used by the agent in the i-th previous interaction with the environment, where i is a positive integer greater than or equal to 1; The agent's strategy is optimized using a reward function and the experience of the target policy to enable the agent to interact with the environment for the i-th time; wherein, the reward function is constructed using the total variation distance and is related to the distance, speed, or posture of the agent's movement; When the agent completes the i-th interaction with the environment, the environmental reward obtained through the reward function is acquired. The environmental reward refers to the positive / negative feedback signal obtained by the agent after performing an action. When the difference between the environmental reward and the preset environmental reward is within the preset difference range, the optimization of the strategy adopted by the agent is stopped. When the reward difference is not within the preset difference range, the agent continues to interact with the environment for the (i+1)th time, and the above steps are repeated until the reward difference is within the preset difference range. The policy experience attribute includes a policy experience reward, which is the sum of the environmental reward obtained by the agent when it interacts with the environment using a policy. The step of filtering historical policy experiences in the experience buffer based on policy experience attributes to obtain target policy experiences includes: The historical strategy experience in the experience buffer whose strategy experience reward is greater than or equal to the first threshold is used as the target strategy experience.

2. The optimization method according to claim 1, characterized in that, The policy experience attribute includes an advantage estimate, which is obtained through generalized advantage estimation. The step of filtering historical policy experiences in the experience buffer based on policy experience attributes to obtain target policy experiences includes: The historical policy experience in the experience buffer whose advantage estimate is greater than or equal to the second threshold is used as the target policy experience.

3. The optimization method according to claim 1, characterized in that, The strategy experience attribute includes state similarity, which is the similarity between a first state and a second state. The first state is the state before the agent interacts with the environment for the i-th time, and the second state is the state when the agent completes the interaction with the environment using the historical strategy. The step of filtering historical policy experiences in the experience buffer based on policy experience attributes to obtain target policy experiences includes: The historical policy experiences in the experience buffer whose state similarity is greater than or equal to the third threshold are used as the target policy experiences.

4. The optimization method according to claim 1, characterized in that, When the i-th interaction condition between the agent and the environment is met, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience, which includes: When the i-th interaction condition between the agent and the environment is met, the number of historical policy experiences in the experience buffer is obtained. When the number is greater than or equal to the fourth threshold, the historical policy experience in the experience buffer is filtered according to the policy experience attribute to obtain the target policy experience.

5. The optimization method according to any one of claims 1 to 4, characterized in that, The optimization method further includes: When the reward difference is not within the preset difference range, the policy experience generated when the agent completes the i-th interaction with the environment is stored in the experience buffer to update the experience buffer.

6. The optimization method according to any one of claims 1 to 4, characterized in that, The reward function is: ； in, Indicates environmental rewards, Let ξ represent the original reward and ξ represent the weight. Let s represent the total variation distance, s represent the state, a represent the action, (s, a) represent the state-action pair, and π represent the new policy obtained after optimizing the agent's policy. k Indicates the current strategy.

7. An optimization device for robot control models, characterized in that, The optimization device includes: An experience filtering module is used to filter historical policy experiences in the experience buffer according to policy experience attributes when the i-th interaction condition between the agent and the environment is met, to obtain target policy experience. Here, the agent is a robot, the environment refers to everything outside the reinforcement learning agent, and is composed of a set of states. Each state is data representing the environment, and the state set includes all possible states in the environment. The policy experience attributes are used to measure the quality of the policy experience, and the experience buffer stores the historical policy experiences used by the agent in its previous i-th interaction with the environment, where i is a positive integer greater than or equal to 1. The model optimization module is used to optimize the strategy adopted by the agent using a reward function and the experience of the target strategy, so that the agent can interact with the environment for the i-th time. The reward function is constructed using the total variation distance and is related to the distance, speed, or posture of the agent's movement. When the i-th interaction between the agent and the environment is completed, the environmental reward obtained through the reward function is acquired. When the reward difference between the environmental reward and a preset environmental reward is within a preset difference range, the optimization of the agent's strategy is stopped. When the reward difference is not within the preset difference range, the agent continues to interact with the environment for the (i+1)-th time, and the above steps are repeated until the reward difference is within the preset difference range. The environmental reward refers to the positive / negative feedback signal obtained by the agent after performing an action. The strategy experience attribute includes a strategy experience reward, which is the sum of the environmental reward obtained by the agent when it interacts with the environment using a strategy. The step of filtering historical strategy experiences in the experience buffer according to the strategy experience attribute to obtain the target strategy experience includes: taking historical strategy experiences in the experience buffer whose strategy experience reward is greater than or equal to a first threshold as the target strategy experience.

8. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the optimization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the optimization method as described in any one of claims 1 to 6.

Images