A robot control method with strategy regularization to improve exploration efficiency

By introducing policy regularization methods and noise states in reinforcement learning and updating Q-values ​​and policy networks, the problems of low exploration efficiency and instability in robot control are solved, and exploration ability and robustness are improved.

CN118596159BActive Publication Date: 2024-11-08UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411071231.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The existing reinforcement learning methods have problems in robot control, such as inexploratory efficiency and instability in strategy learning, especially the distribution transfer problem caused by the differences in data distribution during training and execution, which leads to robots being too cautious when exploring the environment and missing out. Value information.

Method used

By maintaining the policy network and multiple Q-value networks, combining buffer pool data, a policy regularization method is adopted to introduce noise state and calculate the differences between the policy and the noise policy, and to update the Q-value and policy network to improve exploration capabilities and robustness .

Benefits of technology

It improves the exploration efficiency and stability of strategy learning in robot control, ensures that high-value areas are more effectively discovered during the exploration process, and enhances the robot's exploration ability and performance of the environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118596159B_ABST
    Figure CN118596159B_ABST
Patent Text Reader

Abstract

The present invention discloses a robot control method for improving exploration efficiency by using policy regularization, and relates to the technical field of machine learning. The method comprises: constructing a policy network, a plurality of Q-value networks and a buffer pool; the robot uses the policy network to generate actions to interact with the environment, and records the interaction trajectory and stores it in the buffer pool; sampling state data from the buffer pool to construct a policy regularization value; updating the Q-value network and the policy network according to the buffer pool data; repeating the interaction between the policy and the environment and updating the Q-value network and the policy network until the policy performance meets the preset requirements; and controlling the robot to perform tasks according to the final policy network. The present invention encourages exploration in more areas by constructing a policy regularization term, thereby solving the problem of low sample exploration efficiency caused by underestimated Q values ​​in reinforcement learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a robot control method for improving exploration efficiency through strategy regularization. Background Art

[0002] When reinforcement learning is used to tackle robotic tasks, Q-value overestimation bias is a major contributor to poor sample efficiency and unstable policy learning. To address these issues, TD3 proposes to adopt the minimum of two independent Q-value approximators. While this technique can improve the stability of policy learning, it can lead to pessimistic underexploration. This means that the agent may become overly cautious when exploring the environment, potentially missing out on valuable information necessary for optimal learning and decision-making. To improve the exploration ability of policies, existing off-policy actor-critic methods mainly focus on using optimistic principles and distribution correction estimates in the face of uncertainty. However, they still suffer from distribution shift during the off-policy training of the target policy, or assuming a distribution that deviates from the actual problem. This challenge arises from the difference between the data distribution encountered during training and the distribution encountered during execution, leading to suboptimal performance and learning instability. Addressing this distribution shift is critical to ensuring the effectiveness and robustness of robotic control algorithms.

[0003] In the prior art, various methods have been explored in existing reinforcement learning methods to address the problem of low exploration efficiency in actor-critic algorithms, such as upper confidence bound (UCB) criterion reference, adding noise to deterministic policies, intrinsic motivation reward reference, and distribution correction estimation (DICE). Policy entropy is introduced into the objective function of reinforcement learning and maximized during the learning process to improve exploration. However, when the robot learns, the entropy term may be overestimated when revisiting similar states, potentially hindering the agent's ability to effectively discover new areas with high value. In addition, applying the optimistic principle in the face of uncertainty (OFU) has been shown to be an effective strategy to alleviate pessimistic underexploration. In the OFU framework, the reward of UCB is added to the standard value function to form an optimistic value function. Although these methods in OFU can alleviate pessimistic value function estimation, it is still affected by the distribution change of the target policy during off-policy training. The DICE method strictly relies on the assumption of the target policy distribution and fixes this assumption, which may not always hold in the real world. Therefore, exploration efficiency is basically not addressed during policy optimization. Summary of the invention

[0004] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a robot control method that improves exploration efficiency through strategy regularization.

[0005] The technical solution of the present invention is a robot control method for improving exploration efficiency through strategy regularization, which comprises:

[0006] Step 1: Maintain a policy network, multiple Q-value networks, and a buffer pool;

[0007] Step 2: The robot uses the policy network to generate actions to interact with the environment, and records the interaction trajectory and stores it in the buffer pool;

[0008] Step 3: Sample state data from the buffer pool to construct the strategy regularization value;

[0009] Step 4: Update the Q value network and the strategy network according to the buffer pool data;

[0010] Step 5: Repeat the interaction between the strategy and the environment and update the Q-value network and the strategy network until the strategy performance meets the preset requirements;

[0011] Step 6: Control the robot to perform tasks according to the final policy network.

[0012] Furthermore, the specific method of constructing the policy network and multiple Q value networks in step 1 is:

[0013] Maintaining a strategic network , which is used to receive the environmental state data detected by the robot as input and output the parameters for controlling the movement of the robot. By sampling the mean and variance of the multidimensional Gaussian distribution output by the last fully connected layer, that is, ,in is the optimizable parameter of the policy network, represents the mean of the multidimensional Gaussian distribution, Represents the covariance of a multidimensional Gaussian distribution;

[0014] Maintain multiple Q-value networks to estimate the value of actions, that is, the expected value of the long-term cumulative reward for performing an action in a given state; use the Q-value network to evaluate the policy network the quality of the action chosen;

[0015] Furthermore, the specific method of step 2 is:

[0016] The interaction process starts from the environment being in a certain initial state Start, the initial state Defined by the environment or randomly defined by the agent during learning;

[0017] At each time step , the agent follows its learned strategy and current status Output a random policy action ;

[0018] Robot performs actions After that, the new state of the environment is observed , and give corresponding rewards ;This state transition is determined by the dynamic laws of the environment;

[0019] Repeating the interaction process generates a large amount of trajectory data of the robot interacting with the environment;

[0020] Status data , action data , the next moment status data and return value The trajectory data is stored in the buffer pool.

[0021] Furthermore, the specific method of step 3 is:

[0022] Randomly sample state data from the buffer pool , and add random noise from normal distribution to these sampled state data, expressed as follows:

[0023] ,

[0024] In the formula is the variance of the noise state distribution, Indicates that the noise state still falls within the robot's observable range of the environment, ensuring the usability of the state;

[0025] According to the noise state With status data , the strategy model generates corresponding strategies and ;

[0026] According to the noise state and noise action, the corresponding action value is obtained, that is, the following formula:

[0027]

[0028] In the formula represents the Bellman operation, represents the strategy generated under the noise state, represents the action value function under the noise state at the next moment, represents the state value function under the noise state at the next moment, represents the discount factor, Indicates immediate return, represents the noise state at the next moment, Indicates the action value corresponding to the noise state at the next moment, Indicates the current state. Represents the action value function under the noise state at the current moment;

[0029] The difference between the state value brought by the normal state and the state value brought by the noise state is used to calculate the difference in the exploration ability of the evaluation strategy, that is, the following formula

[0030]

[0031] In the formula, represents the state value function at the current moment, represents the noise state value function at the current moment, Indicates normal strategy Noise Strategy The total difference distance between is a constant that is independent of the strategy.

[0032] Furthermore, the specific method of step 4 is:

[0033] Once the variance of the noise state is determined , the Q-value function is updated by optimizing the following formula:

[0034] L( Q ϕ 1 )&= E D [( Y target ' - Q 1 (s,a; ϕ 1 ) ) 2 ] L( Q ϕ 2 )&= E D [( Y target ' - Q 2 (s,a; ϕ 2 ) ) 2 ],

[0035] In the formula is the target Q value, , They represent the action value function under the noise state at the next moment with different network parameters, , From the buffer pool The sampling is obtained, represents the parameters of the first Q-value function, represents the parameters of the second Q-value function, Indicates the return value;

[0036] The policy network is updated by minimizing the following policy loss:

[0037] ;

[0038] In the formula, Indicates that the Q function is used directly for the strategy The loss to be optimized, Represents a buffer The mean of the sampled data, , They represent the action value functions under different network parameters respectively. The output of the policy network also contains the mean vector and covariance vector of the Gaussian policy.

[0039] Use normal strategy Noise Strategy The total difference distance between constructs the strategy regularization term for optimization, which is the following formula:

[0040] ;

[0041] In the formula represents the KL divergence, Representation strategy and noise strategy The distance difference between

[0042] Regularization is introduced into the policy by minimizing the following loss, emphasizing the robustness of the policy:

[0043] ;

[0044] In the formula, Expressing the strategy and noise strategy The distance loss between It is sampled from the buffer pool;

[0045] Combining the above formula, the strategy is optimized by minimizing the following formula:

[0046] ;

[0047] in is a hyperparameter that balances natural performance and robustness.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] The robot control method of the present invention improves exploration efficiency through strategy regularization, and promotes more exploratory behaviors in the robot's strategy by incorporating an Actor regularization term into the strategy update target, thereby solving the problem of low sample exploration efficiency caused by underestimated Q value in reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the interactive update process in the reinforcement learning Actor-Critic method;

[0051] Figure 2 This is a schematic diagram of the steps of the method of the embodiment of the present application;

[0052] Figure 3 A schematic diagram of updating the Q value network and the strategy network of the embodiment method of the present application;

[0053] Figure 4The working process of the Q value function of the embodiment of the present application;

[0054] Figure 5 This is the process of the policy network of the embodiment of the present application. DETAILED DESCRIPTION

[0055] Please refer to Figure 2 , which is a flow chart of a robot control method for improving exploration efficiency through strategy regularization provided in an embodiment of the present invention. The robot control method for improving exploration efficiency through strategy regularization may specifically include the contents described in the following steps S1 to S6.

[0056] S1: Constructing strategy network, multiple Q-value networks and buffer pool;

[0057] S2: The robot uses the policy network to generate actions to interact with the environment, and records the interaction trajectory and stores it in the buffer pool;

[0058] S3: Sampling state data from the buffer pool to construct a strategy regularization value;

[0059] S4: updating the Q value network and the strategy network according to the buffer pool data;

[0060] S5: Repeat the interaction between the strategy and the environment and update the Q-value network and the strategy network until the strategy performance meets the preset requirements;

[0061] S6: Control the robot to perform tasks according to the final strategy network.

[0062] In the embodiments of this application, please refer to Figure 1 , each interactive iterative update requires the robot to be controlled to run in a real environment using the strategy in the strategy model, and the trajectory data generated is used to update the Q-value function and the strategy network; thereby ensuring that after iteration, the control strategy can become the strategy algorithm for robot control for subsequent testing. The embodiment of the present application can provide a strategy network that can be executed by a real robot, and the loss function of the strategy regularization during the learning process has a direct impact on the strategy optimization. The strategy control model of the embodiment of the present application controls the execution actions of the robot, providing the robot with interactive security and efficiency.

[0063] In a possible implementation, the robot control method for improving exploration efficiency through strategy regularization is characterized in that the construction of the strategy network, the multiple Q-value networks and the buffer pool includes:

[0064] Maintaining a strategic network , which is used to receive the environmental state data detected by the robot as input and output the parameters for controlling the movement of the robot. The mean and variance of the multidimensional Gaussian distribution output by the last fully connected layer of the control strategy network are sampled and obtained, that is, ,in is the optimizable parameter of the policy network;

[0065] Maintain multiple Q-value networks to estimate the value of an action, that is, the expected value of the long-term cumulative reward for performing an action in a given state. In the Actor-Critic algorithm, a Q-value network is usually used to act as a Critic to evaluate the Actor strategy network. The quality of the action chosen.

[0066] In a possible implementation, the robot control method for improving exploration efficiency through policy regularization is characterized in that the robot uses a policy network to generate actions to interact with the environment, and records the interaction trajectory and stores it in a buffer pool, including:

[0067] The interaction process starts from the environment being in a certain initial state Start, the initial state Usually defined by the environment, but may also be randomly chosen by the agent during learning;

[0068] At each time step , the agent learns the policy network According to the current status Output a random policy action ;

[0069] Robot performs actions After that, the new state of the environment is observed , and give corresponding rewards ; This state transition is determined by the dynamics of the environment and is usually random;

[0070] Repeating the process will generate a large amount of trajectory data of the robot interacting with the environment;

[0071] The status data , action data , the next moment status data and return value The trajectory data is stored in the buffer pool.

[0072] In a possible implementation, the robot control method for improving exploration efficiency by using strategy regularization is characterized in that sampling state data from a buffer pool to construct a strategy regularization value includes:

[0073] Randomly sample state data from the buffer pool , and add random noise from a normal distribution to these sampled data, expressed as the following formula

[0074] ,

[0075] In the formula is the variance of the noise state distribution, The represented state still belongs to the robot's observable range of the environment, ensuring the usability of the state;

[0076] According to the noise state With status data , the strategy model generates corresponding strategies and ;

[0077] According to the noise state and noise action, their corresponding action values ​​can be obtained, namely, the following formula

[0078]

[0079] In the formula represents the Bellman operation, represents the strategy generated under the noise state, represents the action value function under the noise state at the next moment, represents the state value function under the noise state at the next moment, represents the discount factor;

[0080] The noise state enables the policy function to explore the action space that the original state cannot provide. The noise state guides the policy function to perform more exploration behaviors. If the policy under the noise state is directly optimized, it can only increase the robustness of the policy, and it is difficult to affect the exploration ability of the policy and the performance of the policy. Therefore, the difference between the state value brought by the normal state and the state value brought by the noise state can be used to evaluate the difference in the exploration ability of the policy, that is, the following formula

[0081]

[0082] In the formula Indicates normal strategy Noise Strategy The total difference distance between is a constant that is independent of the strategy.

[0083] For examples, see Figure 3 In a possible implementation, the robot control method for improving exploration efficiency by strategy regularization is characterized in that updating the Q value network and the strategy network according to the buffer pool data includes:

[0084] Once the variance of the noise state is determined , the Q-value function is updated by optimizing the following formula:

[0085] L( Q ϕ 1 )&= E D [( Y target ' - Q 1 (s,a; ϕ 1 ) ) 2 ] L( Q ϕ 2 )&= E D [( Y target ' - Q 2 (s,a; ϕ 2 ) ) 2 ],

[0086] In the formula is the target Q value, , From the buffer pool The sampling is obtained, represents the parameters of the first Q-value function, Represents the parameters of the second Q value function;

[0087] In the Actor-Critic framework, the Actor policy network is updated in the direction that increases the Critic value the most. In other words, the robot is encouraged to choose actions with higher Q values, and the policy network is updated by minimizing the following policy loss:

[0088]

[0089] In the formula , the output of the Actor policy network contains both the mean vector and the covariance vector of the Gaussian policy;

[0090] Direct Optimization It does not improve the exploration efficiency of the strategy. The normal strategy Noise Strategy The total difference distance between can construct the policy regularization term for optimization, namely the following formula

[0091] ;

[0092] In the formula represents KL divergence;

[0093] By introducing regularization into the policy, we emphasize its focus on policy robustness, which is achieved by minimizing the following loss:

[0094] ;

[0095] In the formula It is sampled from the buffer pool;

[0096] Combining the above formula, the following formula can be minimized to optimize the strategy:

[0097] ;

[0098] in is a hyperparameter that balances natural performance and robustness. Noise state It is generated by introducing perturbations in the historical transfer samples. Therefore, this method does not require additional interactions to generate noisy state data.

[0099] In a possible implementation, the robot control method for improving exploration efficiency by using strategy regularization is characterized by repeating the interaction between the strategy and the environment and updating the Q-value network and the strategy network until the strategy performance meets preset requirements.

[0100] For examples, see Figure 4 , shows the specific scheme of the Q value network operation of the embodiment of the present application, wherein the Q value network includes 2 neural network models, each neural network model is sequentially set with 4 layers of full connection and 3 activation functions, each layer of full connection is set with 128 hidden layers, and the activation function uses ReLU.

[0101] For examples, see Figure 5 , shows a specific scheme for the operation of the policy network of an embodiment of the present application, wherein the policy network sequentially sets 4 layers of full connection and 3 activation functions, each layer of full connection is set with 128 hidden layers, and the activation function uses ReLU.

[0102] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0103] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0104] The units described as separate components may or may not be physically separated. As units, it is obvious that a person of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0105] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a grid device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0107] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A robot control method for improving exploration efficiency by strategy regularization, characterized in that: The method includes: Step 1: Maintain a policy network, multiple Q-value networks, and a buffer pool; Step 2: The robot uses the policy network to generate actions to interact with the environment, and records the interaction trajectory and stores it in the buffer pool; Step 3: Sample state data from the buffer pool to construct the strategy regularization value; Randomly sample state data from the buffer pool , and add random noise from normal distribution to these sampled state data, expressed as follows: In the formula is the variance of the noise state distribution, Indicates that the noise state still falls within the robot's observable range of the environment, ensuring the usability of the state; According to the noise state With status data , the strategy model generates corresponding strategies and ; According to the noise state and noise action, the corresponding action value is obtained, that is, the following formula: In the formula represents the Bellman operation, represents the strategy generated under the noise state, represents the action value function under the noise state at the next moment, represents the state value function under the noise state at the next moment, represents the discount factor, Indicates immediate return, represents the noise state at the next moment, Indicates the action value corresponding to the noise state at the next moment, Indicates the current state. Represents the action value function under the noise state at the current moment; The difference between the state value brought by the normal state and the state value brought by the noise state is used to calculate the difference in the exploration ability of the evaluation strategy, that is, the following formula In the formula, represents the state value function at the current moment, represents the noise state value function at the current moment, Indicates normal strategy Noise Strategy The total difference distance between is a constant that is independent of the strategy; Step 4: Update the Q value network and the strategy network according to the buffer pool data; Step 5: Repeat the interaction between the strategy and the environment and update the Q-value network and the strategy network until the strategy performance meets the preset requirements; Step 6: Control the robot to perform tasks according to the final policy network.

2. A robot control method for improving exploration efficiency by strategy regularization as claimed in claim 1, characterized in that: The specific method for maintaining the strategy network and multiple Q value networks in step 1 is: Maintaining a strategic network , which is used to receive the environmental state data detected by the robot as input and output the parameters for controlling the movement of the robot. By sampling the mean and variance of the multidimensional Gaussian distribution output by the last fully connected layer, that is, ,in is the optimizable parameter of the policy network, represents the mean of the multidimensional Gaussian distribution, Represents the covariance of a multidimensional Gaussian distribution; Maintain multiple Q-value networks to estimate the value of actions, that is, the expected value of the long-term cumulative reward for performing an action in a given state; use the Q-value network to evaluate the policy network The quality of the action chosen.

3. The robot control method for improving exploration efficiency by strategy regularization as claimed in claim 1, characterized in that: The specific method of step 2 is: The interaction process starts from the environment being in a certain initial state Start with the initial state Defined by the environment or randomly defined by the agent during learning; At each time step , the agent follows its learned strategy and current status Output a random policy action ; Robot performs actions After that, the new state of the environment is observed , and give corresponding rewards ;This state transition is determined by the dynamic laws of the environment; Repeating the interaction process generates a large amount of trajectory data of the robot interacting with the environment; Status data , action data , next moment status data and return value The trajectory data is stored in the buffer pool.

4. The robot control method for improving exploration efficiency by strategy regularization as claimed in claim 1, characterized in that: The specific method of step 4 is: Once the variance of the noise state is determined , the Q-value function is updated by optimizing the following formula: In the formula is the target Q value, , They represent the action value function under the noise state at the next moment with different network parameters, , From the buffer pool The sampling is obtained, represents the parameters of the first Q-value function, represents the parameters of the second Q-value function, Indicates the return value; The policy network is updated by minimizing the following policy loss: In the formula, represents the loss of directly using the Q function to optimize the strategy, Represents a buffer The mean of the sampled data, , They represent the action value functions under different network parameters respectively. The output of the policy network also contains the mean vector and covariance vector of the Gaussian policy. Use normal strategy Noise Strategy The total difference distance between constructs the strategy regularization term for optimization, which is the following formula: In the formula represents the KL divergence, Representation strategy and noise strategy The distance difference between Regularization is introduced into the policy by minimizing the following loss, emphasizing the robustness of the policy: In the formula, Expressing the strategy and noise strategy The distance loss between It is sampled from the buffer pool; Combining the above formula, the strategy is optimized by minimizing the following formula: in is a hyperparameter that balances natural performance and robustness.