A reinforcement learning robot continuous action control method based on a stochastic discrete policy-evaluation network

Abstract

This invention relates to a reinforcement learning-based continuous action control method for robots based on a stochastic discrete policy-evaluation network, belonging to the field of reinforcement learning technology. First, the high-dimensional continuous actions in the robot control problem are discretized, with the overall continuous action space discretized according to each dimension. Each action dimension is treated as an agent and assigned an independent policy. Second, a reinforcement learning model based on an Actor-Critic structure is constructed. The policy decomposition network module decomposes and represents the local policies of each action dimension in the robot control problem in the form of discrete action probability distributions. This network can optimize the overall computational cost of the algorithm. The centralized evaluation network module provides evaluation of the global policy of robot control. Finally, during training, soft reinforcement learning ideas are integrated, and sampling efficiency is improved by maximizing stochastic entropy, achieving efficient convergence of reinforcement learning and effectively improving the robot's control performance.

Description

Technical Field

[0001] This invention relates to a continuous motion control method for a reinforcement learning robot based on a stochastic discrete policy-evaluation network, belonging to the fields of unmanned systems and reinforcement learning technology, and particularly to a policy-based reinforcement learning robot control method. Background Technology

[0002] In recent years, with the development and application of artificial intelligence technology, autonomous intelligent systems such as drones, unmanned vehicles, and intelligent robots are rapidly entering society, providing great convenience for people's work and life. Simultaneously, with the continuous increase in demand in the robotics field, robot structures are becoming increasingly complex, and biomimetic robots such as quadrupedal robots, hexapod robots, and humanoid robots are emerging rapidly. However, this structural complexity presents significant challenges to the design of robot control algorithms. For humanoid robots and similar configurations, even simple actions such as standing and walking require extremely complex and sophisticated kinematic control algorithms to ensure safety during application. For more complex tasks, such as robot dancing or robot soccer, traditional methods struggle to design efficient and reliable control algorithms, wasting considerable manpower while failing to achieve the desired robot control results.

[0003] Therefore, in recent years, a large number of researchers both domestically and internationally have devoted themselves to the field of intelligent robot control, attempting to replace traditional robot control methods with machine learning methods, thereby effectively improving robot control efficiency while saving human resources. Among them, reinforcement learning methods and technologies, as one of the alternative means to achieve general artificial intelligence, have shone brightly in the field of robot control. As early as October 2019, the artificial intelligence research institution OpenAI achieved the application of reinforcement learning algorithms to control a robotic arm to solve a Rubik's Cube under various disturbances. In 2021, Google DeepMind even realized a humanoid robot soccer match in a simulated environment. However, applying reinforcement learning to achieve this type of robot control task requires a large number of experience samples as support. To successfully train a robot control law, reinforcement learning often requires tens of millions of experience samples, and there may be unstable control situations, posing a threat to robot safety. The huge sample requirements and potential instability of reinforcement learning seriously hinder its application in the field of physical robots.

[0004] Existing successful precedents of reinforcement learning applications in robotics are often based on mature core reinforcement learning algorithms, with in-depth design for specific problems, integrating techniques such as convolutional neural networks, recurrent neural networks, and imitation learning to form a complete problem-oriented methodology, aiming to effectively solve complex problems. Therefore, core reinforcement learning algorithms are fundamental to solving robot control problems, and their performance directly determines the training efficiency and final performance of the reinforcement learning robot control methodology. Depending on the target problem to be solved, reinforcement learning algorithms can be categorized into several types, such as tabular reinforcement learning for discrete states and actions, deep Q-networks (DQN) for continuous states and actions, and policy gradient methods and policy-evaluation (Actor-Critic) methods for continuous state and action problems such as robot control. The policy-evaluation approach is currently the most efficient algorithmic structure for solving continuous motion quantities such as robot joint motor inputs. This structure involves two parallel network structures: the policy network generates the robot's actions, while the evaluation network learns a state-behavior value function through temporal difference methods, guiding the iterative updates of the policy network. Currently, the main policy-evaluation algorithms applicable to robot systems include the Deep Deterministic Policy Gradient (DDPG) proposed by Lilicrap et al., and its subsequent improved algorithms, including the Dual-Delay Deep Deterministic Policy Gradient (TD3) proposed by Fujimoto et al., and the Flexible Policy-Evaluation (SAC) proposed by Haarnoja et al. These three algorithms employ an experience replay mechanism and a single-step temporal difference method during their value function learning process. This allows for the repeated use of acquired experience samples, significantly improving the data utilization rate of reinforcement learning algorithms and reducing the difficulty of applying reinforcement learning to robot control.

[0005] While the three algorithms mentioned above have achieved good results in some tasks, they still have their own problems, leading to poor performance in some robot control tasks. The most significant issue is the high sensitivity of these algorithms to hyperparameters; only through repeated experimentation to find suitable hyperparameters can an effective robot control strategy be trained. Secondly, in the policy networks of these algorithms, the output of continuous actions requires a Tanh activation function layer for normalization, limiting the action output to the allowable range of the robot's motors. However, the Tanh function suffers from a severe gradient vanishing problem; the output actions of the policy network easily get trapped in the difficult-to-optimize Tanh gradient vanishing region, seriously affecting the overall performance of the algorithm, leading to policy exploration failure and reinforcement learning training failure.

[0006] One feasible approach to addressing these issues is to discretize the continuous action space into a discrete action space, representing discrete policies in probabilistic form using the Softmax activation function. This transforms the continuous policy network into a discrete policy network, reducing the hyperparameter sensitivity of the algorithm and mitigating the gradient vanishing problem inherent in continuous policy networks. However, the application of discrete policy networks in tasks such as robot control is primarily hindered by the dimensionality explosion problem of the discrete action space. Traditional discrete action networks require traversing and combining discrete actions across each action dimension to form the discrete action space. Consequently, as the degrees of freedom of the robot to be controlled increase, the size of the discrete state space and the number of output neurons in the discrete policy network grow exponentially, making the algorithm difficult to train. Past research has attempted to address this problem, with a representative example being BDQ proposed by Tavakoli et al., which solves the dimensionality explosion problem of the discrete action space by discretizing each dimension separately and calculating the state-behavior dominance function. However, BDQ and other related works are only at the same level as the unmodified original DDPG in terms of effectiveness, and there is a certain gap compared with more advanced TD3, SAC and other technologies, making it difficult to apply them effectively to complex robot control problems. Summary of the Invention

[0007] To address the application challenges of discrete actions in robot control, this invention provides a discrete reinforcement learning method suitable for reinforcement learning problems involving complex continuous actions, such as robot control. Based on the fundamental policy-evaluation structure, this method proposes a decomposed discrete policy network structure. This structure discretizes the overall continuous action space along each dimension, treating each action dimension as an agent and assigning it an independent policy, thus solving the dimensionality explosion problem. A single decomposed policy network is used to synchronously represent the independent policies for each dimension, optimizing the training efficiency. During policy training, this invention samples from the evaluation network for each discretized action dimension, ensuring sampling covers the entire discretized action space. This allows for synchronous training of all output units of the policy network, significantly improving algorithm efficiency. Furthermore, this invention designs a soft reinforcement learning method for the decomposed discrete policy network structure, calculating the entropy of each action dimension separately. This effectively integrates the soft reinforcement learning method into the framework of this invention, forming a highly efficient reinforcement learning robot control method based on stochastic discrete policy-evaluation. Compared to existing reinforcement learning robot control methods, the method of this invention can effectively solve the dead zone problem of continuous action output, while having low sensitivity to hyperparameters. It is more efficient for reinforcement learning training of robots and can effectively avoid the instability problem of reinforcement learning algorithms, thus obtaining an efficient and reliable reinforcement learning robot control strategy.

[0008] A reinforcement learning-based continuous motion control method for robots based on stochastic discrete policy-evaluation networks, comprising the following steps:

[0009] 1. Construct a discrete reinforcement learning model based on single-dimensional decoupling discretization;

[0010] The main reason why discrete actions are difficult to apply to reinforcement learning robot control is that centralized discretization of the robot's multi-degree-of-freedom action space leads to the problem of discrete action space. Therefore, this invention first discretizes each controllable degree of freedom of the robot and regards each discretized degree of freedom as an agent, thereby forming a multi-agent discrete action space in which the number of discrete actions is linearly related to the number of controllable degrees of freedom.

[0011] When reinforcement learning is applied to robotics, the robot control problem can be abstracted into a sequential decision problem, which consists of Markov quadruples:

[0012]

[0013] in Defined as the robot's S-dimensional continuous observation space, where each element is denoted as state information s(t), including robot sensor observation information, robot self information, etc., and t represents the time when the action occurs in the Markov decision process; Let be the M-dimensional continuous motion space of the robot, where each element is denoted as a(t), including the input quantities of the robot's motors for each axis. p is the transition probability, which determines the distribution of the state at the next moment, s(t+1)~p(s(t), a(t)); r is the reward value function, which needs to be set appropriately according to the task scenario and control requirements.

[0014] To enable the application of discrete actions to the control of reinforcement learning robots, this invention firstly... Each dimension is discretized into N parts, transforming the continuous action space into a discrete action space. Considering that traditional discretization methods result in a global discretized state space containing M... N The exponential nature of each action makes training extremely difficult. This invention employs a single-dimensional decoupling discretization method, discretizing each action dimension separately to obtain M single-dimensional discretized action spaces, resulting in a total of MN discretized actions.

[0015]

[0016] Where m is defined as the action dimension. Defined as a discretized action space of dimension m. Let be the nth discrete action in the mth action dimension. Based on the discretized action space of each dimension, each dimension is considered as an agent, and a discretized policy π for that dimension is defined. m :

[0017] a m (t)~π m (a m (t)|s(t))

[0018] Where a m (t) is defined as the action in the m-th dimension at time t. Simultaneously, the global policy π is defined:

[0019] π(a m (t)|s(t)):={π1, π2,…π M}

[0020] and exclusion strategies for each dimension That is, the strategy for all dimensions except m.

[0021]

[0022] Based on treating each action dimension as an independent agent and assigning policies accordingly, this invention designs a one-dimensional Markov decision process for multi-agent-like reinforcement learning problems composed of multi-dimensional actions. This process decomposes the overall Markov decision process of robot control into various dimensions to facilitate subsequent reinforcement learning training. The one-dimensional Markov decision process is defined as the following Markov quadruple:

[0023]

[0024] p in the quadruple m Let m be the state transition probability of a one-dimensional Markov model, which is the state transition probability based on an m-dimensional action when the policy has dimensions other than m fixed. It is defined as follows:

[0025]

[0026] r m Defined as the following reward value function:

[0027]

[0028] in Actions in dimensions other than m are determined by strategies in dimensions other than m. Sample generation:

[0029]

[0030] In the process of optimizing the overall strategy for robot reinforcement learning control It changes over time. But within each time step, it is possible to... Treating it as a fixed strategy, a deterministic one-dimensional Markov decision process is constructed.

[0031] Based on the fundamental construction of a one-dimensional Markov process, the optimization objectives of the policy for each action dimension can be given. For dimension m, the ultimate goal of reinforcement learning is to find the optimal policy. This maximizes the total reward value obtained by the agent. This reward value is defined as:

[0032]

[0033] Where p ini S→[0,1] represents the distribution of the initial state s(0). Since p m Strategies with other dimensions Therefore, the optimal strategy π*(a) is relevant. m The value of (t)|s(t) will change with the changes in the policies of other dimensions. From the perspective of the robot system as a whole, each degree of freedom has an independent Markov decision process and a discrete policy, and the optimal policy of each dimension is strongly coupled with the policies of other dimensions.

[0034] Because the transition probabilities of a one-dimensional Markov decision process change continuously with the optimization of the overall policy, the probability distribution of past empirical samples does not conform to the current transition probabilities. Therefore, a traditional Deep Q-Network (DQN) and its improved version are used to directly sample one-dimensional empirical {s(t), a}. m (t), r t Solving for the optimal policy for a single action dimension using s(t+1)} and combining it with single-step temporal difference is theoretically infeasible. Even if successful in actual training, it offers no advantage over continuous reinforcement learning algorithms. To address this issue, this invention designs a reinforcement learning algorithm architecture of centralized evaluation network-decompositional policy network and its corresponding training process, which can effectively utilize collected experience samples to train discrete random policies for each action dimension. This architecture and its corresponding training process are theoretically sound and exhibit superior performance compared to continuous reinforcement learning algorithms in practical experiments.

[0035] 2. Constructing Decomposition Strategies—Centralized Evaluation of Multi-Dimensional Discrete Reinforcement Learning Neural Network Architectures

[0036] The decomposition strategy—the neural network architecture for centralized evaluation of multi-dimensional discrete reinforcement learning mainly consists of two main modules, totaling six neural networks. The centralized evaluation network module comprises four neural networks:

[0037] ①Evaluate network 1, denoted as Q(s(t), a(t); θ Q1 The network takes state and action variables as input and outputs the expected reward value (state-action value) corresponding to the state and action. The network parameter is θ. Q1 The network has a state dimension plus an action dimension as its input dimension and a 1-dimensional output dimension. It contains two hidden layers, each with 512 neurons.

[0038] ②Evaluate network 2, denoted as Q(s(t), a(t); θ Q2 The structure of this network is the same as that of evaluation network 1. Together with evaluation network 1, it forms a dual evaluation network structure similar to Double DQN, the main purpose of which is to reduce the adverse effects of value function overestimation on reinforcement learning algorithms.

[0039] ③Target evaluation network 1, denoted as Q(s(t), a(t); θ′ Q1 To evaluate the target network of network 1, the network weights are θ′. Q1 In each iteration of the reinforcement learning algorithm, the target evaluation network 1 follows the network weights of the evaluation network 1 using a soft update method:

[0040] θ Q1 ←(1-τ)θ′ Q1 +τθ Q1

[0041] Where τ is the soft update rate, typically set to 0.005 to 0.05. The main purpose of setting up the target evaluation network is to reduce the rate of change of the temporal difference objective function and decouple the temporal difference objective function from the value function to be optimized, thereby improving the stability and efficiency of the algorithm.

[0042] ④ Target evaluation network 2, denoted as Q(s(t), a(t); θ′ Q2 ), which is the target network for evaluating network 2, and its main function is the same as that of target evaluation network 1.

[0043] The decomposition policy network module consists of two networks: a decomposition stochastic policy network and its corresponding target network. The purpose of the decomposition stochastic policy network is to fit the optimal decomposition discrete policy, which includes independent policies for all robot action dimensions, and can be denoted as:

[0044] π(a(t)|s(t); θ π )={π(a1(t)|s(t); θ π ), …, π(a) m (t)|s(t); θ π )}

[0045] The weights of the decomposed policy network are θ πThe policy for each dimension is determined by this set of parameters. The policy representation of the decomposed stochastic policy network is a probability distribution over the discrete action space of the robot's degrees of freedom:

[0046]

[0047] The network takes state s(t) as input and contains two hidden layers, each with 512 neurons. The number of neurons per layer can be increased as needed for higher action dimensions. The output layer has M×N neurons and outputs a vector of size (number of action dimensions multiplied by the number of actions per dimension). This vector is then reconstructed into an M×N output matrix O(s(t); θ π )∈R M ×N The element in the m-th row and n-th column is denoted as O. mn (s(t;θ) π Subsequently, a softmax operation is performed on the n elements of each row of the matrix to obtain an M×N policy matrix Π(s(t); θ). π )∈[0,1] M×N This matrix represents the strategy π(a(t)|s(t); θ π The element in the m-th row and n-th column represents the n-th action dimension. The probability, i.e. The calculation formula is as follows:

[0048]

[0049] In summary, decomposed policy networks can generate policies with multiple dimensions from a single set of network parameters. If we consider the action dimension as an agent in multi-agent reinforcement learning, then decomposed stochastic policy networks achieve the output of policies for multiple agents from a single network and a single set of parameters, effectively reducing the overall computational cost of the algorithm. Decomposed policy networks also have a target network, denoted by parameters θ′. π The network follows the decomposition strategy using a soft update approach:

[0050] θ Q1 ←(1-τ)θ′ Q1 +τθ Q1

[0051] 3. Train the neural network from step 2 using maximum entropy reinforcement learning.

[0052] In the process of training the neural network in step 2, this invention integrates the maximum entropy reinforcement learning method. This method maximizes the entropy of the discrete policy decomposition while optimizing the policy, thereby effectively exploring the state space and action space and increasing the overall efficiency of the algorithm. The value function of maximum entropy reinforcement learning is defined as follows:

[0053]

[0054] in, The entropy represents the action, and its value is the logarithm of the probability density of choosing action 'a' under policy π. For multi-dimensional actions, the global action entropy is equal to the sum of the entropies of each individual action:

[0055]

[0056] Let represent the entropy of the i-th dimension action. It's important to note that in the discrete action space, the probabilities of discrete actions must first be mapped back to the probability density of the continuous space before calculating the action entropy. This assumes all actions are restricted to [-1, 1]. The specific calculation method is as follows:

[0057]

[0058] α represents the adaptive heat level, which autonomously adjusts the proportion of policy entropy in the value function to maintain the overall exploration rate of the policy at a reasonable level. This invention sets a target entropy for the adaptive heat level α. This value is generally set between [0,2] and can be adjusted according to the actual usage scenario. Adaptive heat adjustment is based on the policy entropy and target entropy to keep the policy entropy near the target entropy. The adjustment method is shown below, where λ... α To adapt the heat update step size:

[0059]

[0060] The training process of the neural network in this invention is a process in which a reinforcement learning robot controller continuously interacts with the environment. Each training cycle includes the following steps:

[0061] (1) Environmental experience collection, application strategy network π(a(t)|s(t); θ π Discrete actions in each dimension are selected based on state observations:

[0062] a m (t)~π(a m (t)|s(t); θ π )

[0063] The actions are made continuous and applied to the robot. After one time step, the state observation s(t+1) and reward r are obtained for the next time step. t The experience {s(t), a(t), r} t s(t+1)} is stored in the experience pool

[0064] (2) Training sample collection, from the experience pool Randomly select an experience sample from a batch In the formula This represents the time of experience collection; then sampling is performed from the target policy network. The corresponding target action And the target entropy corresponding to this action:

[0065]

[0066] (3) Centralized evaluation network update. During this update process, both evaluation networks are updated synchronously. This update process mainly applies temporal difference TD(0) to iterate the evaluation networks. To suppress overestimation, the temporal difference objective function value is derived from the values ​​estimated by the two objective evaluation networks. We select the smaller value from the options. First, we evaluate the network's loss function in a focused manner:

[0067]

[0068]

[0069] Then, the gradient descent method is used to update the parameters of the two evaluation networks, where λ Q To centrally evaluate the learning rate of network parameters:

[0070]

[0071] (4) Decomposing the policy network for updating. This invention mainly uses the policy gradient method to train the decomposing policy network, first by updating the actions in the sampled data. As input, the policy matrix is ​​output through the policy network. And for each discrete action in the policy matrix, the probability distribution Calculate a partial policy loss function:

[0072]

[0073] in It is obtained by sampling from the policy, and does not need to be recalculated for different partial policy loss functions. Only one global action sampling is needed to generate the required loss function for different partial policy loss functions. After calculating the loss function for each discrete action, these functions are summed to obtain the global policy loss function:

[0074]

[0075] Subsequently, a gradient descent strategy is used to update the network parameters, where λ πThe learning rate for the policy network parameters:

[0076]

[0077] (5) Adaptive heat update, based on the policy matrix output in step (4). calculate And update the adaptive heat hyperparameter α;

[0078] (6) Target network update: Upgrade the parameters of the centralized evaluation target network and the decomposed strategy target network.

[0079] θ′ Qi ←τθ Qi +(1-τ)θ′ Qi For i∈{1,2}

[0080] θ′ π ←τθ π +(1-τ)θ′ π

[0081] Beneficial effects

[0082] (1) The method of the present invention firstly decouples and discretizes the multidimensional continuous action space of the robot into a single dimension, then constructs a Markov decision process model based on the single-dimensional discretized action space, and on this basis, builds a new discrete reinforcement learning network framework of decomposing discrete policy network and centralized continuous evaluation network, which solves the problem that discrete reinforcement learning cannot cope with high-level continuous action space, and provides a new paradigm for using discrete actions to solve reinforcement learning robot control problems.

[0083] (2) Compared with continuous reinforcement learning, the method of the present invention has higher stability and lower hyperparameter sensitivity, while having good final state performance and can achieve a higher reward value than continuous reinforcement learning, which is of great significance for improving the effect of reinforcement learning robot control.

[0084] (3) The method of the present invention can make efficient use of the collected experience samples, reduce the need for environmental interaction in the reinforcement learning training process, and is of great significance for the deployment of robot reinforcement learning on physical objects.

[0085] (4) The method of this invention adopts a random discrete strategy and integrates the maximum entropy reinforcement learning algorithm to achieve efficient exploration of the optimal action. The method of this invention directly calculates the entropy of each action dimension during the policy iteration process instead of sampling and fitting, which has higher policy entropy optimization efficiency. Therefore, the overall exploration effect of the algorithm is better, the convergence speed of the reinforcement learning robot controller is faster, and the time cost of the reinforcement learning training process is significantly reduced.

[0086] (5) The method of this invention successfully applies discrete policies to continuous action reinforcement learning while maintaining the policy-evaluation system architecture of continuous reinforcement learning. The evaluation network of this method is universally applicable to most continuous reinforcement learning algorithms and has the potential to further integrate distributed reinforcement learning, recurrent neural network reinforcement learning and other improved methods for different environments. It has strong adaptability and scalability and is expected to be widely used in a variety of different types of robot control tasks.

[0087] (6) This invention relates to a reinforcement learning-based continuous action control method for robots based on a stochastic discrete policy-evaluation network, belonging to the field of reinforcement learning technology, and particularly to a policy-based reinforcement learning robot control method. First, the high-dimensional continuous actions in the robot control problem are discretized, with the overall continuous action space discretized according to each dimension. Each action dimension is treated as an agent and assigned an independent policy, effectively solving the dimensionality explosion problem during discretization. Second, a reinforcement learning model based on an Actor-Critic structure is constructed, which mainly consists of two modules: a policy decomposition network module and a centralized evaluation network module. The policy decomposition network module decomposes the local policies of each action dimension in the robot control problem in the form of discrete action probability distributions, which can optimize the overall computational cost of the algorithm. The centralized evaluation network module provides evaluation of the global policy of robot control and provides optimization guidance for the local policies of each action dimension. This network can improve training efficiency through experience replay. Finally, during training, the soft reinforcement learning idea is integrated, and sampling efficiency is improved by maximizing random entropy, achieving efficient convergence of reinforcement learning and effectively improving the robot's control performance. Attached Figure Description

[0088] Figure 1 This is a network structure framework diagram of the decomposition discrete strategy provided by the present invention;

[0089] Figure 2 This is a schematic diagram illustrating the calculation of the loss function of the decomposed policy network in the reinforcement learning method based on random discrete policy-evaluation provided by the present invention. Detailed Implementation

[0090] The present invention will now be described in detail with reference to the accompanying drawings. These drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0091] This invention provides a discrete reinforcement learning robot control method based on the policy-critic (Actor-Critic) framework. It utilizes the centralized continuous evaluation-decomposition discrete policy network structure built by this invention to solve the dimensionality explosion problem of the high-dimensional continuous action space corresponding to multi-degree-of-freedom robots during discretization. It fully leverages the advantages of low sensitivity and high robustness of discrete policies to achieve efficient training and stable performance of reinforcement learning robot end-to-end control.

[0092] To solve robot control problems with continuous characteristics using discrete reinforcement learning, it is necessary to first convert the robot's control variables into continuous action variables for reinforcement learning, and then discretize these continuous action variables. For a robot with M controllable degrees of freedom, its global control variable is denoted as u, which consists of the control variables u for each degree of freedom:

[0093] u:=[u1, u2, ..., u M ] T

[0094] And each control quantity u m Each has its upper and lower limits:

[0095]

[0096] To facilitate the reinforcement learning training process, all control variables u are mapped to the range [-1, 1], forming a series of actions for the reinforcement learning robot control problem:

[0097] a: = [a1, a2, ..., a M ]

[0098]

[0099] The continuous action space corresponding to a continuous action is represented as follows: Subsequently, each dimension of the continuous action space is discretized to obtain the global discrete action space. It consists of m one-dimensional action subspaces corresponding to m degrees of freedom (action dimensions), each subspace having N discrete actions. Simultaneously with constructing the action space, an encoder is constructed to encode the robot's sensor observations into an x-dimensional vector, corresponding to an x-dimensional continuous state space. The encoder can be flexibly constructed according to actual application scenarios, such as using fully connected layers, convolutional networks, recurrent neural networks for encoding, or directly using sensor observations as the state space without encoding.

[0100] After constructing the state space, continuous action space, and discrete action space, a neural network model framework is built based on the dimensions of the state and action. For example... Figure 1The reinforcement learning network model shown consists of two modules: centralized evaluation and decomposition strategy, with a total of six networks and an adaptive heat parameter α.

[0101] Centralized Evaluation Module: This module consists of four neural networks: Evaluation Network 1 and Evaluation Network 2 form a dual evaluation network structure; Target Evaluation Network 1 and Target Evaluation Network 2 are the target networks corresponding to the aforementioned two. These four networks have the same double-hidden-layer fully connected neural network structure; generally, 512 neurons per hidden layer are sufficient for good results, but this can be adjusted according to actual usage requirements. The evaluation network is only used during the training process; its main function is to evaluate the corresponding state-behavior value information Q(s(t), a(t)) based on state feature information and continuous action information.

[0102]

[0103]

[0104] Decomposition Strategy Module: This module consists of two neural networks: a decomposition strategy network and a target decomposition strategy network. Both networks have the same double-hidden-layer fully connected neural network structure; generally, 512 neurons per hidden layer are sufficient for good results, but this can be adjusted according to actual usage requirements. The strategy network outputs the discrete motion distribution of each degree of freedom of the robot based on state feature information.

[0105]

[0106] Adaptive heat: This parameter α is autonomously adjusted based on the entropy of the policy distribution output by the decomposed policy network, so that the overall exploration rate of the policy is maintained at a pre-set target entropy. Nearby, the algorithm converges faster.

[0107] Once the model is built, it can be trained. In each training cycle, actions are sampled from the output of the policy network, applied to the robot's complete experience samples, and stored in the experience pool. Then, the network is iterated. Using a set of samples sampled from the experience pool, the centralized evaluation network module, the decomposed policy network module, and the adaptive heat index are iteratively updated according to the above method. The iterative update process of the decomposed policy network module is as follows: Figure 2 As shown in the figure, this illustrates part of the loss function. The computational process is as follows. After completing the above three iterations, the target network parameters of the two modules are soft-updated, thus completing one reinforcement learning iteration cycle. Repeat the above steps until the policy converges.

[0108] In practical use, only the decomposed policy network of the decomposed policy module needs to be used. At each decision time step, the robot's state feature information is input, and the policy matrix Π(s(t)) is output; θ π In each row of the matrix, select the single-dimensional action with the highest probability and combine them into a global action:

[0109]

[0110] a(t)={a1(t), a2(t),...M(t)}

[0111] By using the above methods to select actions, a stable robot controller can be obtained, achieving good control of complex robots.

[0112] In summary, the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning robot continuous action control method based on a random discrete policy-evaluation network, characterized by The steps of the method include: Step 1, constructing a single-dimensional discrete reinforcement learning model based on single-dimensional decoupling discretization; Step 2, according to the single-dimensional discrete reinforcement learning model constructed in step 1, constructing a decomposed policy-centralized evaluation multi-dimensional discrete reinforcement learning neural network architecture; Step 3, training the neural network architecture constructed in step 2 based on maximum entropy reinforcement learning; Step 4, using the neural network architecture trained in step 3 as the optimal policy for actual robot control, completing the reinforcement learning control of the robot; In the step 1, the single-dimension decoupling discretization of the reinforcement learning discrete action space composed of is defined as follows: in, For the action dimension, For the first The first action dimension Each discrete action has a space. Each action contains N discrete actions arranged in an arithmetic progression. In the robot's M action dimensions, each action dimension's discrete action space corresponds to a discrete policy; for the m-th action dimension's discrete action space... Its corresponding discrete strategy is That is, at any time status Down, any discrete action in The corresponding probability.

2. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 1, wherein: In the step 1, the constructed single-dimensional discrete reinforcement learning model is a single-dimensional Markov decision process four-tuple wherein is a robot state space, is a dimension of a discretized action space, is a single-dimensional Markov transition probability, i.e., the state transition probability based on a dimension other than the dimension of the policy single-dimensional reward value function : wherein, is an action in other dimensions than dimension is an action in other dimensions than dimension is a strategy in other dimensions than dimension sampling generation: 。 3. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 2, wherein: The optimal policy for a single-dimensional discrete reinforcement learning model is obtained by maximizing the following expected return: where, is the initial state distribution, is the reinforcement learning reward function discount rate.

4. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 1, wherein: In step 2, the decomposed policy-centralized evaluation multi-dimensional discrete reinforcement learning neural network architecture includes a centralized evaluation network module and a decomposed policy network module.

5. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 4, wherein: The decomposition strategy network module includes a decomposition strategy network and its corresponding target decomposition strategy network, with corresponding network parameters. The decomposition policy network is responsible for outputting discrete policies for each dimension. And optimize it to make it approach the optimal strategy. The decomposition strategy network input is state features. The output is a policy matrix. Each row represents a policy in one dimension, and the element in the m-th row and n-th column of the policy matrix... Take the nth action for the mth action dimension The probability function, i.e. .

6. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 5, wherein: The centralized evaluation network module includes evaluation network 1: , evaluation network 2: , and corresponding target evaluation network 1: , target evaluation network 2: , corresponding network parameters , evaluation network 1, 2 are used for fitting state-action value function , two networks together constitute a double evaluation network structure to prevent overestimation problem that may occur in value function learning process, target evaluation network 1, 2 are used for providing target Q function of reinforcement learning time difference update : The evaluation network is also used to evaluate the actions output from the decomposed discrete policy network, thereby optimizing the policy.

7. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 1, wherein: In step 3, the method for training the deep reinforcement learning network framework is: Step (1) reinforcement learning network and hyperparameter initialization, convert the input range of each degree of freedom of the robot into an action space, and discretize it into a single-dimensional discrete action space respectively; initialize the network parameters of the centralized evaluation network module and the decomposition policy network module; initialize the adaptive heat and policy target entropy ; initialize the experience pool; Setting the reward value function of reinforcement learning; Step (2) generates a control policy distribution of the robot controller using a decomposition strategy network based on the robot state feature observation, from which an action is sampled and applied to the robot to obtain a complete quadruple experience sample data ; Step (3) After the experience quantity in the experience sample pool reaches the standard, a batch of samples are collected from the experience sample pool , wherein represents the time of the experience collection; Based on , the target policy network generates and calculates the corresponding policy entropy: Step (4) iteratively updates the parameters of the centralized evaluation network based on a time-difference and maximum entropy reinforcement learning method , ; Step (5) iteratively updates the parameters of the decomposed policy network based on a policy gradient and maximum entropy reinforcement learning method ; Step (6) iteratively updates the adaptive hotness ; Step (7) soft updating the target network parameters; Step (8) repeating steps (2)~(7) until the robot control policy converges.

8. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 7, wherein: In the step (5), the parameters of the decomposition strategy network are updated The iterative update is mainly achieved by minimizing the following loss function through the gradient descent method: wherein is the partial policy loss function, calculated as follows: 。 9. The reinforcement learning robot continuous action control method based on random discrete policy-evaluation network according to claim 7, wherein: The step (6) is to adapt the heat Iterative updating is mainly achieved by the following methods: calculating the dimension strategy entropy : Updating the adaptive temperature: 。

Images