A reinforcement learning robot control method based on dynamic reward value pruning

Abstract

The application discloses a reinforcement learning robot control method based on dynamic reward value clipping. The method comprises the following steps: obtaining an observation signal of a robot in an environment, wherein the observation signal reflects sensor data information of a target joint of the robot; inputting the observation signal into a trained actor-critic model to obtain a control strategy for the target joint of the robot; and adjusting parameters of the target joint of the robot based on the control strategy. The obtained robot control strategy is smoother, and the stability, efficiency and safety of the robot when performing a complex task are improved.

Description

Technical Field

[0001] This invention relates to intelligent control methods, and more specifically, to a reinforcement learning robot control method based on dynamic reward value pruning. Background Technology

[0002] One of the goals of artificial intelligence is to enable systems to solve complex problems from high-dimensional, unprocessed sensory information. In recent years, the combination of deep learning and reinforcement learning has made significant progress in achieving this goal. The development of these technologies has also provided new ideas for the intelligent control of robots. Intelligent control technology for robots with autonomous learning and planning capabilities is of great significance to the technological development of robots (such as quadruped robots). It not only improves the robot's control effect, increases its stability and learning efficiency, enabling it to perform more complex tasks, but also effectively reduces various safety accidents caused by operational errors, thereby improving safety and efficiency. More importantly, this technology enables robots to perform high-risk tasks in various extreme and harsh environments, such as disaster relief and search and rescue.

[0003] Traditional autonomous planning and control methods for robots rely on modeling prior knowledge of the robot and designing controllers based on these models. Rule-based controllers require manual parameter tuning for specific tasks, resulting in control performance that is highly dependent on parameter selection and lacks generalization ability in complex environments. While neural network controllers can handle diverse tasks, they require large amounts of pre-trained data, and due to the high cost of sampling data, they are mostly used for simulations rather than real-world applications.

[0004] In existing technologies, intelligent reinforcement learning control schemes for robots suffer from problems such as overestimation of reward values ​​and low learning efficiency. Overestimation of reward values ​​may cause the optimization objective during the learning process to deviate from the actual objective, reducing generalization ability and actual performance. Especially when facing unknown or changing environments, overconfident decisions may lead to performance degradation or unstable behavior, which is still far from being applicable in real industrial environments. This overestimation problem can cause an excessive preference for certain behaviors during the learning process, thereby affecting the decision-making quality and generalization ability of the agent.

[0005] In summary, although the demand for intelligent robot control technology is growing rapidly, the introduction of artificial intelligence into the intelligent control of robotic arms is still in its early stages of exploration. Because robot control is subject to numerous environmental factors with inherent uncertainties, there is an urgent need to explore new control methods to address the problems of overestimating reward functions and low sample efficiency inherent in existing intelligent robot control methods. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a reinforcement learning robot control method based on dynamic reward value pruning. This method includes the following steps:

[0007] Acquire observation signals of the robot in the environment, which reflect sensor data information of the robot's target joints;

[0008] The observed signals are input into a trained actor critic model to obtain a control strategy for the robot's target joints;

[0009] The parameters of the robot's target joint are adjusted based on the control strategy described above;

[0010] The actor-critic model includes an actor network and a critic network. The actor network learns a policy function to select actions based on the current state, while the critic network uses a value function to evaluate the performance of the current policy and provide corresponding reward signals.

[0011] Compared with existing technologies, the advantages of this invention are that it proposes a continuous action reinforcement learning control method based on relative entropy regularization and reward value distribution. By introducing relative entropy into the actor-critic framework of continuous action reinforcement learning based on reward value distribution, it realizes an efficient robot control method that can effectively alleviate the overestimation of reward value function without human prior knowledge.

[0012] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0014] Figure 1 This is a flowchart of a reinforcement learning robot control method based on dynamic reward value pruning according to an embodiment of the present invention;

[0015] Figure 2 This is a diagram of a reinforcement learning-based robotic arm control system architecture according to an embodiment of the present invention. Detailed Implementation

[0016] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0017] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0018] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0019] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0021] This invention addresses the problem of low learning efficiency and inability to adapt to complex scenarios in robot control systems. It employs a relative entropy regularized reinforcement learning framework to solve this problem. Furthermore, by transforming the reward value function into a distribution and then performing fine-tuning, the algorithm's overestimation is reduced. This pruned reward value distribution is then combined with the relative entropy regularized reinforcement learning framework to achieve a highly efficient, stable, and robot-suitable control scheme. In this paper, "robot" includes any mechanical or robotic device that simulates human behavior or thought, or simulates other living beings (such as robotic dogs, robotic cats, robotic cars, etc.), and can be used to replace or assist humans in work. For clarity, a quadruped robot will be used as an example below.

[0022] Combination Figure 1 and Figure 2 As shown, the reinforcement learning-based quadruped robot control method based on dynamic reward value pruning includes the following steps:

[0023] Step S110: Establish the robot control system.

[0024] Taking a quadruped robot hardware system as an example, a quadruped robot working in complex environments is mainly controlled by drive motors and joints that control its walking direction. The control signals generated by each joint are learned through reinforcement learning algorithms, enabling autonomous walking. To ensure the algorithm's warm-up phase, the quadruped robot must collect data to characterize its current state and send this state data to the control system to generate corresponding control signals. By integrating this hardware-collected information, the control system can generate control signals specifically for the quadruped robot, enabling it to perform complex tasks in complex environments such as chemical plants and nuclear power plants.

[0025] In this paper, the environment refers to the physical environment in which the quadruped robot operates, including the ground and obstacles. Actions refer to the torques of each joint of the quadruped robot in a complex environment to control walking better. State refers to the sensor data of each joint, policy refers to the control of each joint of the quadruped robot, and reward is determined by the stability of the quadruped robot's walking in the complex environment.

[0026] Step S120: Construct a reinforcement learning model and train it.

[0027] In one embodiment, the reinforcement learning model employs an actor-critic framework. The actor represents the policy function, which learns a policy to maximize the reward. The critic represents the value function, which estimates the value of the current policy, i.e., evaluating the actor's performance. The actor-critic framework is a reinforcement learning algorithm based on value and policy functions. The actor network is responsible for learning the policy function and selecting actions based on the current state; the critic network is responsible for evaluating the policy's performance and providing corresponding reward signals. The goal of the actor-critic model is to continuously optimize the policy function to maximize the agent's cumulative reward.

[0028] In one embodiment, the smooth update based on relative entropy in reinforcement learning is extended to the actor-critic model framework of reward value distribution, and distribution sampling is introduced to solve the problem of calculating local relative entropy values ​​in continuous action space. A control algorithm for continuous action space that can effectively alleviate reward value overestimation and optimize cumulative error is proposed.

[0029] For example, adding relative entropy as a penalty term to the reward function can regularize the value function:

[0030]

[0031] in:

[0032]

[0033] in, It is a value function, γ t It is a reward discount factor. It is state s t The reward value is given in state s, where s0 is the initial state and t is the time index. π(a|s) is the current policy in state s. The strategy before state s is η, which controls the strength of relative entropy regularization. It is the expectation of the control strategy. It is a set of actions, where 'a' represents an action, and 'D' represents an action. KL(s) represents the relative entropy, or KL divergence. Formula (1) is a value function based on relative entropy regularization.

[0034] The addition of relative entropy regularization changes the update formula for the commentator network. With relative entropy, the update formula for the commentator network is expressed as:

[0035]

[0036] in:

[0037]

[0038] Where Ψ is the update formula for the action preference function. It is a state transition function, referring to the next state s after state s and action a. ′ The possibility, This refers to the reward value under state s and action a. Let Ψ represent the set of states, γ represent the reward discount factor, and Ψ represent the reward set. t (s,a) represents the action preference function at time t, under state s and action a, Ψ t (s) represents the action preference function at time t in state s, Ψ t (s ′ ) indicates the next state s ′ The action preference function below, This represents an approximation of the Boltzmann operator. This indicates that the Boltzmann operator operation is performed on the action preference function.

[0039] The aforementioned reinforcement learning framework with relative entropy regularization can optimize the accumulated error, but it still does not adequately address the overestimation problem of reward values. Therefore, in one embodiment, a reward value distribution framework (or a reward value distribution framework based on relative entropy regularization) is introduced, which transforms the reward value into a quantile-based distribution:

[0040]

[0041] Where Z(s, a; ψ) is the distribution of reward values ​​associated with state s, action a, and action preference function Ψ, ψ is a learnable parameter, and N represents N quantiles. Let θ represent the Dirac function at the i-th output of the parameterized model (which refers to a model that uses a neural network to learn quantiles to approximate the reward distribution). i The quantiles of the reward value model are approximated by the learned quantiles, i.e., θ. i There are i quantiles.

[0042] Based on formula (5), high quantiles can be precisely truncated to control the distribution of reward values, thereby reducing overestimation of reward values ​​and achieving better results. For example, for the N θ values ​​in formula (5) i , θ i Sort the data in ascending order, then remove the K highest quantiles, leaving NK quantiles. The reward value distribution Z after this pruning is as follows. clip (s, a; ψ) is represented as:

[0043]

[0044] Where K is a dynamically changing number:

[0045]

[0046] in, This indicates that each parameter is sorted from smallest to largest, and K represents the number of quantiles for cropping. start K represents the initial number of cuts. end This represents the number of cuts at the end of the learning process. t is the learning step size, T is the total number of steps, and d... decay It is the rate of decline.

[0047] After adjusting the distribution of reward values, it is integrated into the relative entropy regularization reinforcement learning framework, as shown in formula (8):

[0048]

[0049] Here, Ψ(s, a) represents the preference function, which incorporates quantiles and uses clipped quantiles to simulate reward values.

[0050] In the discrete action space, the distributed Bellman objective is represented as:

[0051]

[0052] in, Represents the approximate Boltzmann operator, ψ - Let ψ represent the parameters of the target network, and ψ represent the parameters of the training network. The training network updates every step, while the target network updates every C steps. Then, the two are trained by comparison. r(s, a) represents the reward value in state s and action a. This represents the estimated quantile of the ordination under state s. This represents the quantile of the target network ranking under state s and action a.

[0053] To enable use in the continuous motion state of a quadruped robot, Monte Carlo sampling is introduced for a local region in the continuous motion space, represented as:

[0054]

[0055] Where 'a' represents the action, a j It represents the j-th action ∈ of the sampling. j The noise from the j-th sample follows a Gaussian distribution. μ is the mean of the Gaussian distribution, σ 2 is the variance of the Gaussian distribution, and n is the number of samples.

[0056] Finally, within the reward value distribution framework based on relative entropy regularization, the update formula for the critic network is expressed as:

[0057]

[0058] in, It is the Huber-loss function.

[0059] For actor networks, deterministic policy gradient methods from policy gradients can be used to train and update the network, while target network techniques can be used to reduce the variance during actor network training.

[0060] For example, the update formula for the actor network is expressed as:

[0061]

[0062] in, This represents the gradient of 'a' with respect to the overall formula. φ represents the learnable parameters of the Actor network. This indicates that the gradient of the parameters φ of the actor network with respect to policy π is calculated, and the parameters are updated by backpropagation.

[0063] In summary, by incorporating relative entropy regularization and pruning high-quantile numerical distributions into reinforcement learning algorithms, efficient and stable learning can be achieved, enabling the robot to reach the ideal state in a relatively short time.

[0064] Step S130: Using the robot observation signal as input, the robot control signal is obtained by using a trained reinforcement learning model, and then the control signal is fed back to the robot control system to regulate the robot's movement.

[0065] Once the reinforcement learning model is trained, it can be used for actual robot control. For example, real-time collected robot observation signals can be input into the trained reinforcement learning model to obtain robot control signals, which can then be fed back to the robot control system to regulate the robot's subsequent movements or motions, including overall robot movement or movement of some joints.

[0066] In summary, compared with existing technologies, the continuous action reinforcement learning quadruped robot control method based on dynamic pruning reward value distribution and relative entropy regularization provided by this invention has the following advantages:

[0067] 1) By extending the idea of ​​relative entropy regularization to the actor-critic model framework in continuous action space, the policy update in the learning process becomes smoother.

[0068] 2) The reward value is transformed into a quantile-based reward value distribution and dynamically pruned, reducing the overestimation of the predicted reward value. This method of pruning the reward value distribution solves the problem of overestimation of reward values ​​in traditional continuous reinforcement learning.

[0069] 3) Integrating the dynamic pruning reward value distribution into the reinforcement learning framework with relative entropy regularization makes the update of the control policy smoother and more efficient, while helping the robot to be more stable, efficient and safe when performing complex tasks.

[0070] 4) Simulation verification shows that this invention improves the robot's control learning efficiency in complex environments and enhances sample performance through more thorough exploration, thereby enabling it to effectively learn and adapt to changing task requirements. Furthermore, this invention can be extended to other unmanned systems such as autonomous vehicles and drones, demonstrating broad application prospects.

[0071] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0072] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0073] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0074] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0075] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0079] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A reinforcement learning-based control method for a quadruped robot based on dynamic reward value pruning, comprising the following steps: Acquire observation signals of the robot in the environment, which reflect sensor data information of the robot's target joints; The observed signals are input into a trained actor critic model to obtain a control strategy for the robot's target joints; The parameters of the robot's target joint are adjusted based on the control strategy described above; The actor-critic model includes an actor network and a critic network. The actor network is used to learn a policy function to select an action based on the current state. The critic network is used to evaluate the performance of the current policy using a value function and provide a corresponding reward signal. In the process of training the actor critic model, the value function is used. Set to: in: in, It is a reward discount factor. It is a state The reward value below, It is the initial state. It is a state The current strategy is as follows: It is a state The previous strategy It's about setting parameters. Control strategy Expected value Represents relative entropy; In the process of training the actor critic model, the update formula for the critic network is set as follows: in, This is the update formula for the action preference function at time t+1. It is a state transition function, representing the state. and actions The next state The possibility, Representing state and actions The reward value below, Represents a set of states. This represents the reward discount factor. Let represent the action preference function at time t, under state s and action a. Let represent the action preference function at time t in state s. Indicates the next state The action preference function below, This represents an approximation of the Boltzmann operator. This indicates that the Boltzmann operator operation is performed on the action preference function. Represents a set of actions; In the process of training the actor critic model, the update formula of the critic network is further expressed as: in, It is the Huber-Loss function. Represents a distributed Bellman objective in a discrete action space. Let represent the Dirac function at the i-th output of the parameterized model, whereby the parameterized model learns quantiles to approximate the reward value distribution model. Let represent the i-th quantile, where i and j are indices, N represents the number of quantiles, and K represents the number of higher quantiles removed from the N quantiles. This represents the learnable parameters of the critic network; In the process of training the actor critic model, the value of K is determined according to the following formula: in, This represents the initial number of pieces cut. This represents the number of cuts at the end of the learning process, t is the learning step size, and T is the total number of steps. It is the rate of decline.

2. The method according to claim 1, characterized in that, The robots include robotic dogs, robotic cats, or robotic cars.

3. The method according to claim 1, characterized in that, The training of the actor critic model also includes Monte Carlo sampling of local regions in the continuous action space, represented as: in, It's an action. It represents the j-th action of sampling. It is the noise from the j-th sample. Follows Gaussian distribution , It is the mean of a Gaussian distribution. It is the variance of the Gaussian distribution. It refers to the number of samples.

4. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

5. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

Images