Robot control reinforcement learning method and device based on dynamic potential energy reward

Abstract

The application relates to the technical field of computer science, in particular to a robot control reinforcement learning method and device based on dynamic potential energy reward, wherein the method comprises the following steps: obtaining robot information and initial state information of a robot; based on the robot information and the initial state information, performing at least one round of robot control interaction on the robot; in each round of control interaction, based on a control instruction and state information of the robot, calculating a dense reward value of the robot constructed based on dynamic potential energy; and combining the dense reward value constructed based on the dynamic potential energy and training data generated in the control interaction process to train an initial control strategy neural network of the robot, so as to obtain a final control strategy neural network. Thus, the problems in the prior art that the potential energy function is fixed, the reward obtained by each state in the whole training process is determined, the strategy exploration is limited, and the robot cannot be controlled in a complex manner are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer science, and in particular, relates to a robot control reinforcement learning method and device based on dynamic potential energy reward. BACKGROUND

[0002] Reinforcement learning has achieved great success in video games, chess, financial modeling and language models due to its powerful feature extraction and decision-making capabilities. Using deep reinforcement learning-based methods to generate robot control strategies is a research hotspot in recent years. However, in the design process of the entire system, how to design a reward function that can clearly reflect the expectations of researchers so that the robot can learn various skills is a challenging task.

[0003] In related technologies, Siekmann et al. designed a reward function based on reference trajectory deviation, enabling the cassie robot to learn to walk; peng et al. used motion reorientation to generate animations and designed a reward function that combines trajectory imitation and reinforcement goals, enabling the robot to learn various actions; Kalashnikov et al. collected data in a real robotic arm as priori and used sparse rewards 1 and 0 to assign rewards to whether the robotic arm successfully grasped; the standard reinforcement learning environment BipedalWalker of Gym designed a positive reward based on the distance moved, and to avoid invalid actions, a penalty term was designed based on the size of the torque of each action; PBRS (Potential-Base Reward Shaping) proposed a normative reward function design method that can accelerate the convergence of the agent to the optimal strategy and achieve multi-objective learning.

[0004] However, in related technologies, since the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, which limits the strategy exploration and cannot control the robot in a complex manner, and thus needs to be improved. SUMMARY

[0005] The present application provides a robot control reinforcement learning method and device based on dynamic potential energy reward to solve the problems in related technologies, such as the reward obtained by each state in the entire training process being determined due to the fixed potential energy function, which limits the strategy exploration and cannot control the robot in a complex manner.

[0006] The first aspect embodiment of the application provides a robot control reinforcement learning method based on dynamic potential energy reward, applied to a model training phase, wherein the method comprises the following steps: obtaining robot information and initial state information of a robot; based on the robot information and the initial state information, performing at least one round of robot control interaction on the robot, in each round of control interaction, based on the control instruction and state information of the robot, calculating a dense reward value of the robot based on a dynamic potential energy structure; combining the dense reward value based on the dynamic potential energy structure and training data generated in the control interaction process to train an initial control policy neural network of the robot to obtain a final control policy neural network.

[0007] Optionally, in an embodiment of the application, the at least one round of robot control interaction on the robot comprises: obtaining state information at each time in the nth round of the robot control interaction process; determining a termination time T n n of termination of the nth round of robot control interaction n based on the termination time T n and the state information at each time in the nth round to obtain a state information sequence M n generated by the robot control interaction, wherein n = 1, 2, 3,..., N, M n is a sequence composed of all state information from 0 to T n in the nth round of robot control interaction, and N is the total training rounds; based on the state information sequence M n , all round state information sequences {M1, M2,..., M N ,..., M D} generated by N rounds of robot control training are obtained.

[0008] Optionally, in an embodiment of the application, before calculating the dense reward value of the robot based on the dynamic potential energy structure, it further comprises: calculating the potential energy of the robot according to the current state information of the robot; based on the potential energy, determining the weight parameter of the potential energy through uniform random scaling; or, based on the potential energy, determining the weight parameter of the potential energy through Gaussian random scaling; or, based on the potential energy, determining the weight parameter of the potential energy through quadratic random scaling; based on the potential energy and the weight parameter, constructing the dynamic potential energy of the robot; based on the dynamic potential energy, constructing a dense reward function based on the dynamic potential energy.

[0009] Optionally, in an embodiment of the application, the dense reward function based on the dynamic potential energy can be but is not limited to:

[0010] F D (s t ,a t ,s t+1 )=γΦD (s t+1 )-Φ D (s t )(0≤t<T n-1 ),

[0011] where F D (s t ,a t ,s t+1 ) is the dense reward based on dynamic potential energy, s t is the state information of the robot at time t, a t is the control instruction sent to the robot for s t , s t+1 is the new state information of the robot after responding to a t , γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r t at time t.

[0012] Optionally, in an embodiment of the present application, the calculation formula of the dynamic potential energy can be but is not limited to:

[0013]

[0014] where w i (t) (0≤i≤3) is a dynamic weight parameter, an initial weight parameter, which can be initialized according to the robot control target, P ctrl is an energy consumption penalty, P contact is a foot contact force penalty, and P Δy is a yaw penalty.

[0015] Optionally, in an embodiment of the present application, the calculation formula of the potential energy can be but is not limited to:

[0016]

[0017] where, is to encourage the robot to walk at the instruction speed v0, is to encourage the agent to walk in the instruction direction θ0, is a survival reward, is a foot sole orientation reward, which encourages the robot to put the foot sole flat and not to walk on tiptoe, σ0, σ1, σ2 are adjustment parameters, K is a normal number, which is used to reward or punish whether the robot falls down, h is the height of the center of gravity of the robot, h0 is the height of the center of gravity of the robot at time 0, v x is the moving speed of the center of gravity of the robot, v0 is the moving speed of the center of gravity of the robot at time 0, x is the walking direction, and θ zis the angle between the robot trunk and the vertical direction at time 0, fl or is the left foot orientation of the robot represented by quaternion, fl or0 is the left foot orientation of the robot at time 0, fr or is the right foot orientation of the robot represented by quaternion, fr or0 is the right foot orientation of the robot at time 0, s0is the state information of the robot at time 0, s t is the state information of the robot at time t.

[0018] Optionally, in an embodiment of the present application, the calculation formula of the uniform random scaling can be but is not limited to:

[0019]

[0020] wherein a is the lower limit of scaling, b is the upper limit of scaling, U i is the uniform distribution on the interval [a, b];

[0021] The calculation formula of the Gaussian random scaling can be but is not limited to:

[0022]

[0023] wherein a is the mean of scaling, b is the variance of scaling, N i is the Gaussian distribution with the mean a and the variance b;

[0024] The calculation formula of the quadratic random scaling can be but is not limited to:

[0025]

[0026] wherein a is the lower limit of scaling, b is the upper limit of scaling, P i is the quadratic distribution with the lower limit a and the upper limit b.

[0027] Optionally, in an embodiment of the present application, the training of the initial control policy neural network of the robot in combination with the dense reward value constructed based on the dynamic potential energy and the training data generated in the control interaction process comprises: obtaining a parameter set of the initial control policy neural network, and determining a target function of the parameter set based on the parameter set; generating an update rule for updating the parameter set based on the target function; and training the initial control policy neural network by the update rule in combination with the dense reward constructed based on the dynamic potential energy and the training data generated in the control training process.

[0028] Optionally, in an embodiment of the present application, the expression of the target function can be but is not limited to:

[0029]

[0030] wherein P(M n , θ D ) is a probability of occurrence of a sequence of state information M D when the initial control policy neural network controls the robot using a parameter set θ n , π θ (a t | s t ) is a probability of outputting a robot control instruction a D by the initial control policy neural network with a parameter set θ t when the state information is s t , and R(M n ) is a dynamic dense reward sum of M n .

[0031] The expression of the update rule can be, but is not limited to, as follows:

[0032]

[0033] wherein α is a step size, δ is an expected deviation of a dynamic weight relative to a standard weight, and β is a random value obtained after sampling.

[0034] The second aspect embodiment of the present application provides a robot control reinforcement learning method based on dynamic potential energy reward, applied to a model application stage, wherein the method comprises the following steps: obtaining actual robot information and actual initial state information of a target robot; inputting the actual robot information and the actual initial state information into a pre-trained final control policy neural network to obtain a control instruction of the target robot, wherein the pre-trained final control policy neural network is trained by a dense reward constructed based on dynamic potential energy.

[0035] The third aspect embodiment of the present application provides a robot control reinforcement learning device based on dynamic potential energy reward, applied to a model training stage, wherein the device comprises: a first obtaining module configured to obtain robot information and initial state information of a robot; a first calculating module configured to perform at least one round of robot control interaction on the robot based on the robot information and the initial state information, and calculate a dense reward value of the robot constructed based on dynamic potential energy in each round of control interaction based on a control instruction and state information of the robot; and a training module configured to train an initial control policy neural network of the robot based on the dense reward value constructed based on dynamic potential energy and training data generated in the control interaction process to obtain a final control policy neural network.

[0036] Optionally, in one embodiment of this application, the first calculation module includes: a first acquisition unit, configured to acquire state information at each moment of the nth round during the robot control interaction; and a determination unit, configured to determine the termination time T of the nth round of robot control interaction. n The first generation unit is used to generate data based on the termination time T. n The state information sequence M generated by the robot control interaction is obtained by combining the state information at each time point in the nth round. n Where n = 1, 2, 3, ..., N, M n For the nth round of robot control interaction, from 0 to R n The sequence of all state information at each time step, where N is the total number of training rounds; the second generation unit is used to generate the sequence based on the state information sequence M. n The robot control training process generates a sequence of state information for all N rounds, {M1, M2, ..., M}. n ,…,M N}

[0037] Optionally, in one embodiment of this application, it further includes: a second calculation module, configured to calculate the robot's potential energy based on the robot's current state information before calculating the dense reward value constructed based on the robot's dynamic potential energy; a first determining module, configured to determine the weight parameters of the potential energy by uniform random scaling based on the potential energy; or, a second determining module, configured to determine the weight parameters of the potential energy by Gaussian random scaling based on the potential energy; or, a third determining module, configured to determine the weight parameters of the potential energy by quadratic random scaling based on the potential energy; a first construction module, configured to construct the robot's dynamic potential energy based on the potential energy and the weight parameters; and a second construction module, configured to construct a dense reward function based on the dynamic potential energy.

[0038] Optionally, in one embodiment of this application, the dense reward function based on dynamic potential energy may be, but is not limited to:

[0039] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t (0≤t) <T n-1 ),

[0040] Among them, F D (s t ,a t ,s t+1 ) represents a dense reward based on dynamic potential energy, st is the state information of the robot at time t, a t is the state information of the robot at time t, a t is the control instruction sent to the robot, s t+1 is the new state information of the robot after responding to a t is the new state information of the robot after responding to a D is the dynamic potential, that is, the reward r at time t t .

[0041] Optionally, in an embodiment of the present application, the calculation formula of the dynamic potential can be but is not limited to:

[0042]

[0043] wherein w i (t) (0≤i≤3) is a dynamic weight parameter, is an initial weight parameter, which can be initialized according to the robot control target, P ctrl is an energy consumption penalty, P contact is a foot contact force penalty, P Δy is a yaw penalty.

[0044] Optionally, in an embodiment of the present application, the calculation formula of the potential can be but is not limited to:

[0045]

[0046] wherein, is to encourage the robot to walk at the instruction speed v0, is to encourage the agent to walk in the instruction direction θ0, is a survival reward, is a foot sole orientation reward, which encourages the robot to put the foot sole flat and not to walk on tiptoe, σ0, σ1, σ2 are adjustment parameters, K is a normal number used to reward or punish whether the robot falls down, h is the height of the center of gravity of the robot, h0 is the height of the center of gravity of the robot at time 0, v x is the moving speed of the center of gravity of the robot, v0 is the moving speed of the center of gravity of the robot at time 0, x is the walking direction, θ z is the angle between the trunk of the robot and the vertical direction, θ0 is the angle between the trunk of the robot and the vertical direction at time 0, fl or is the orientation of the left foot of the robot represented by a quaternion, fl or0 is the orientation of the left foot of the robot at time 0, fr or is the orientation of the right foot of the robot represented by a quaternion, fr or0 is the orientation of the right foot of the robot at time 0, s0 is the state information of the robot at time 0, s t is the state information of the robot at time t.

[0047] Optionally, in an embodiment of the present application, the calculation formula of the uniform random scaling can be but is not limited to:

[0048]

[0049] wherein a is the lower limit of scaling, b is the upper limit of scaling, U i is a uniform distribution on the interval [a, b];

[0050] The calculation formula of the Gaussian random scaling can be but is not limited to:

[0051]

[0052] wherein a is the mean of scaling, b is the variance of scaling, N i is a Gaussian distribution with mean a and variance b;

[0053] The calculation formula of the quadratic random scaling can be but is not limited to:

[0054]

[0055] wherein a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with lower limit a and upper limit b.

[0056] Optionally, in an embodiment of the present application, the training module comprises: a second acquisition unit configured to acquire a parameter set of an initial control policy neural network and determine a target function of the parameter set based on the parameter set; a generation unit configured to generate an update rule for updating the parameter set based on the target function; and a training unit configured to train the initial control policy neural network by the update rule in combination with the dense reward constructed based on the dynamic potential energy and training data generated in the control training process.

[0057] Optionally, in an embodiment of the present application, the expression of the target function can be but is not limited to:

[0058]

[0059] wherein P(M n , θ D ) is a probability that the state information sequence M D occurs when the initial control policy neural network with the parameter set θ n controls the robot, π θ (a t | s t ) is an initial control policy neural network with the parameter set θ D in the state information s tAt that time, output robot control command a t The probability, R(M) n ) is M n The sum of dynamic dense rewards;

[0060] The expression for the update rule can be, but is not limited to, the following:

[0061]

[0062] Where α is the step size, and δ is the expected deviation of the dynamic weights relative to the standard weights, which is a random value obtained after sampling.

[0063] A fourth aspect of this application provides a robot control reinforcement learning device based on dynamic potential energy rewards, applied in the model application stage. The device includes: a second acquisition module for acquiring actual robot information and actual initial state information of the target robot; and an input module for inputting the actual robot information and the actual initial state information into a pre-trained final control strategy neural network to obtain control commands for the target robot. The pre-trained final control strategy neural network is trained using a dense reward system constructed based on dynamic potential energy.

[0064] A fifth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot control reinforcement learning method based on dynamic potential energy reward as described in the above embodiments.

[0065] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robot control reinforcement learning method based on dynamic potential energy reward.

[0066] A seventh aspect of this application provides a computer program product, including a computer program that, when executed, implements the above-described robot control reinforcement learning method based on dynamic potential energy reward.

[0067] The embodiment of the application can perform at least one round of robot control interaction on the robot according to the robot information and the initial state information of the robot, calculate the dense reward value of the robot based on the dynamic potential energy in each round of control interaction, and then train the initial control policy neural network of the robot to obtain the final control policy neural network. The reward value obtained by dynamically adjusting the current action control of the robot based on the dense reward value constructed based on the dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the motion posture of the robot is more natural and stable. Therefore, the problems in the related art that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, the strategy exploration is limited, and the robot cannot be controlled complexly are solved.

[0068] Additional aspects and advantages of the application will be made apparent by the following description and the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0069] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0070] Figure 1 A flowchart of a robot control reinforcement learning method based on a dynamic potential energy reward according to an embodiment of the application is provided.

[0071] Figure 2 A block diagram of a Bolt robot size parameter according to an embodiment of the application is provided.

[0072] Figure 3 A block diagram of a PBRS and a DPBRS (Dynamic Potential-Based Reward Shaping) intuitive action display according to an embodiment of the application is provided.

[0073] Figure 4 A block diagram of an evaluation reward change curve when different reward functions are used according to an embodiment of the application is provided.

[0074] Figure 5 A block diagram of a robot control reinforcement learning device based on a dynamic potential energy reward according to an embodiment of the application is provided.

[0075] Figure 6 A flowchart of another robot control reinforcement learning method based on a dynamic potential energy reward according to an embodiment of the application is provided.

[0076] Figure 7A block diagram of speed tracking test results according to an embodiment of the present application;

[0077] Figure 8 A block diagram of CG yaw test results according to an embodiment of the present application;

[0078] Figure 9 A block diagram of CG height test results according to an embodiment of the present application;

[0079] Figure 10 A block diagram of robot gait with PBRS reward function according to an embodiment of the present application;

[0080] Figure 11 A block diagram of robot gait with DPBRS_U reward function according to an embodiment of the present application;

[0081] Figure 12 A block diagram of robot control reinforcement learning device based on dynamic potential energy reward according to another embodiment of the present application;

[0082] Figure 13 A structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] Embodiments of the present application are described in detail below with reference to the attached drawings, which are meant to be exemplary and not limiting.

[0084] A robot control reinforcement learning method and device based on dynamic potential energy reward are described below with reference to the accompanying drawings. In view of the fact that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, which limits policy exploration and makes it impossible to control the robot in a complex manner. The robot control reinforcement learning method based on dynamic potential energy reward provided in the present application can be used to perform robot control interaction for at least one round according to the robot information and initial state information of the robot, and the dense reward value constructed based on the dynamic potential energy of the robot can be calculated in each round of control interaction. Then, the initial control policy neural network of the robot is trained to obtain the final control policy neural network. The reward value obtained by the current action control of the robot is dynamically adjusted based on the dense reward value constructed based on the dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and making the motion posture of the robot more natural and stable. Thus, the problem that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, policy exploration is limited, and the robot cannot be controlled in a complex manner is solved.

[0085] Specifically, Figure 1 A flowchart of the robot control reinforcement learning method based on dynamic potential energy reward provided in the present application is shown in FIG. 1.

[0086] As Figure 1 shown, the robot control reinforcement learning method based on dynamic potential energy reward is applied to the model training stage, and the method includes the following steps:

[0087] In step S101, the robot information and initial state information of the robot are obtained.

[0088] It can be understood that the robot can be an ODRI (Open Dynamics Robotics Initiative) open-source point-foot biped robot Bolt, or other robots, which can be set by a person skilled in the art according to actual conditions, and the present application does not make specific limitations.

[0089] Further, in the embodiments of the present application, the state information can include but is not limited to 48-dimensional information composed of attitude information, auxiliary information and external instructions, which are not specifically limited by the present application. The attitude information can include but is not limited to Euler angles (3-dimensional) of the base body orientation, center of gravity moving speed (3-dimensional) and acceleration (3-dimensional), reflecting the stability and attitude stability of Bolt walking, angles (6-dimensional) and angular velocities (6-dimensional) of 6 joints, the agent can use the joint information to learn the relationship between the attitude and the joint control torque, past two time steps of joint angles (12-dimensional) and past one time step of joint angular velocity information (6-dimensional), to improve the agent's perception ability to speed instructions and external changes, etc., which are not specifically limited by the present application. The auxiliary information can include but is not limited to relative displacement (6-dimensional) of the two feet and the center of gravity, different gaits of the robot under different speed instructions, and the robot can perceive the current motion state using this information, etc., which are not specifically limited by the present application. The external instruction information can include but is not limited to extracting a speed instruction (1-dimensional) from the instruction speed set v0={0.3, 0.4, 0.5, 0.6, 0.7} as the target speed of the robot walking and a yaw angle instruction (1-dimensional) as the target direction of the robot walking before starting each round of iterative learning process, etc., which are not specifically limited by the present application.

[0090] As a possible implementation manner, the embodiments of the present application can obtain robot information and initial state information of the robot.

[0091] For example, as shown in Figure 2 The embodiments of the present application can use an ODRI open-source point-foot biped robot Bolt, and the Bolt control strategy learned by the reinforcement learning algorithm in the physical simulation environment can be migrated to the entity robot only by fine-tuning. The structure of Bolt makes it lightweight and flexible, but the palm of the point-foot type increases the control difficulty, making it difficult to maintain a straight line and balance when walking, so it is difficult to control it using traditional control methods. In the experiment, the Bolt model in xml file format is imported into the Mujoco physical simulation engine to realize physical simulation, the reinforcement learning algorithm is written in Python, and the state information of the robot is obtained and the control instruction is sent through the Python programming interface provided by the Mujoco physical simulation engine.

[0092] Further, in the embodiments of the present application, the Bolt robot has 6 degrees of freedom, including left and right hip joint pitch, roll and left and right knee joint pitch, and the angle ranges of the left and right hip joint pitch, roll and left and right knee joint pitch are [-10°, 10°], [-30°, 15°] and [-5°, 45°] respectively. The action space of the Bolt robot control deep reinforcement learning algorithm is 6-dimensional, corresponding to the control torque of the 6 degrees of freedom of the Bolt robot, and the range is [-1 N·m, 1 N·m]. The Actor neural network of the deep reinforcement learning algorithm directly outputs 6-dimensional control torque. The experimental goal is to control the Bolt to realize walking at 5 different speeds, and the state space is composed of 48-dimensional information including attitude information, auxiliary information and external instructions.

[0093] Optionally, in one embodiment of the present application, before calculating the dense reward value of the robot based on the dynamic potential energy, it further comprises: calculating the potential energy of the robot according to the current state information of the robot; determining the weight parameter of the potential energy by uniform random scaling based on the potential energy; or, determining the weight parameter of the potential energy by Gaussian random scaling based on the potential energy; or, determining the weight parameter of the potential energy by quadratic random scaling based on the potential energy; constructing the dynamic potential energy of the robot based on the potential energy and the weight parameter; and constructing the dense reward function based on the dynamic potential energy based on the dynamic potential energy. The dense reward function based on the dynamic potential energy can be but is not limited to:

[0094] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t<T n-1 ),

[0095] Wherein, F D (s t ,a t ,s t+1 ) is the dense reward based on the dynamic potential energy, s t is the state information of the robot at time t, a t is the control instruction issued to the robot for s t , s t+1 is the new state information of the robot after responding to a t , γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r t at time t.

[0096] The calculation formula of the dynamic potential energy can be but is not limited to:

[0097]

[0098] wherein w i (t) (0≤i≤3) are dynamic weight parameters, initial weight parameters, which can be initialized according to the robot control target, P ctrl is an energy consumption penalty, P contact is a foot contact force penalty, P Δy is a yaw penalty.

[0099] The calculation formula of the potential energy can be but is not limited to:

[0100]

[0101] wherein, is to encourage the robot to walk at the command speed v0, is to encourage the agent to walk in the command direction θ0, is a survival reward, is a foot sole orientation reward, which encourages the robot to put the foot sole flat and not to walk on tiptoe, σ0, σ1, σ2 are adjustment parameters, K is a normal number used to reward or punish whether the robot falls down, h is the height of the robot's center of gravity, h0 is the height of the robot's center of gravity at time 0, v x is the moving speed of the robot's center of gravity, v0 is the moving speed of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot's torso and the vertical direction, θ0 is the angle between the robot's torso and the vertical direction at time 0, fl or is the orientation of the robot's left foot represented by a quaternion, fl or0 is the orientation of the robot's left foot at time 0, fr or is the orientation of the robot's right foot represented by a quaternion, fr or0 is the orientation of the robot's right foot at time 0, s0 is the state information of the robot at time 0, s t is the state information of the robot at time t.

[0102] The calculation formula of the uniform random scaling can be but is not limited to:

[0103]

[0104] wherein a is the lower limit of scaling, b is the upper limit of scaling, U i is a uniform distribution in the interval [a, b];

[0105] The calculation formula of the Gaussian random scaling can be but is not limited to:

[0106]

[0107] wherein a is the mean of scaling, b is the variance of scaling, N iGaussian distribution with mean a and variance b;

[0108] The calculation formula of the quadratic random scaling can be but is not limited to:

[0109]

[0110] wherein a is a lower limit of scaling, b is an upper limit of scaling, P i is a quadratic distribution with lower limit a and upper limit b.

[0111] In some embodiments, the potential-based reward function of the embodiments of the present application can be but is not limited to:

[0112]

[0113] wherein Φ(s t ) is a potential-based reward function defined on state space S; i is the number of reward terms, ω i is the weight of the i-th reward term, is designed according to different objectives of reinforcement learning, and is calculated from the initial state s0 to the current state s t , which is a function of the potential size.

[0114] Further, the embodiments of the present application can calculate the dense reward sum R(M n ) of the entire M n , and the calculation formula can be but is not limited to:

[0115]

[0116] Further, the embodiments of the present application can calculate the PBRS dense reward, and the expression can be but is not limited to:

[0117] F(s t ,a t ,s t+1 )=γΦ(s t+1 )-Φ(s t )(0≤t<T n -1), (3)

[0118] wherein if R Φ (M n ) is the PBRS reward sum of the entire M n , then

[0119]

[0120] As can be seen from formulas (1)-(4), R Φ (M n ) is equivalent to R(M n) minus the potential energy reward of the initial state. Therefore, the potential energy based reward shaping can be achieved by using the potential energy based reward function shown in formula (1), under the premise of ensuring that the reinforcement learning converges to the optimal policy, the learning goal is achieved and the learning efficiency is improved by using each reward in formula (1) to guide the reinforcement learning process.

[0121] Further, in order to control the biped robot to advance at a constant speed and make the walking posture more stable and natural, each reward in formula (4) can be designed according to (5)-(8) to form a dense reward function constructed based on the potential energy difference between the current state and the expected state, and the expression can be but not limited to:

[0122]

[0123] wherein, ν x is the current moving speed of the robot's center of gravity, and σ0 is an adjustment parameter; ν z is the current yaw angle, and σ1 is an adjustment parameter; is the survival reward, H(s) judges whether the robot falls according to the current state s, a positive reward can be obtained if the robot does not fall, otherwise a larger negative reward will be given, which encourages the agent to maintain balance as much as possible and walk farther; is the foot sole orientation reward, which encourages the robot to put the foot sole flat and not to walk on tiptoe, fl or is the orientation of the robot's foot represented by a quaternion, fl or0 is the initial orientation of the foot.

[0124] In addition, in order to improve the efficiency of reinforcement learning, three parts of the penalty reward function are designed, the size of this part of the reward is negative, mainly composed of three parts, and the weights are p0, p1 and p2. The optimal strategy learned by the agent should make all three tend to 0, and the expressions are shown in (9)-(11).

[0125]

[0126] P Δy =p2|y-y0|, (11)

[0127] wherein, P ctrl is the energy consumption penalty, a i is the torque used by the i-th motor, this penalty term prevents violent motion to ensure the stability of the system, P contact is the foot contact force penalty, which encourages the agent to raise the foot as much as possible and helps exploration at the beginning of the experiment, Pn,i is the force received by the nth foot in the i direction; P Δy is the yaw penalty, y0 is the initial heading, y is the current heading, the agent usually deviates from the specified direction when walking, so the penalty is given to reduce the offset.

[0128] In addition, it should be noted that, in order to avoid the agent from useless exploration, when the absolute values of the pitch angle, roll angle and heading angle are greater than β0, the posture of the robot is already unnatural, in order to avoid invalid learning to terminate the current iteration learning, and a larger negative reward value of w4 is given. The final reward of the potential-based reward function is shown in formula (12).

[0129] When the agent achieves the goal, the reward obtained at each step will tend to 1.0.

[0130]

[0131] Further, the embodiment of the present application is based on the reward function of dynamic potential, which combines dynamic potential with PBRS to form DPBRS, denoted as Φ D (s t ), which can be but is not limited to:

[0132]

[0133] DPBRS dense reward, i.e. the dense reward function based on dynamic potential, which can be but is not limited to:

[0134] F D (s t ,α t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t<T n-1 ), (14)

[0135] Compared with the potential function Φ(s t ) in formula (4), the dynamic potential function Φ D (s t ) of the embodiment of the present application allows each part of the potential function to be variable with t, i.e. for any state s (s∈S), when appearing in different positions within the same MDP (Markov Decision Process) or in different MDPs, the dense reward value obtained by the agent is dynamically variable, and further makes the total dense reward of the entire MDP also dynamically variable. Based on the same reason of potential-based reward shaping, the dynamic potential-based reward function shown in formula (13) can achieve dynamic potential-based reward shaping, when w i(t) is a constant, the DPBRS degenerates to the PBRS, and the PBRS and the DPBRS are intuitively shown in a flowchart as shown in FIG. 8. Figure 3

[0136] Further, in the embodiment of the present application, w i (t) is dynamically adjustable by uniformly random scaling, Gaussian random scaling or quadratic random scaling of w i .

[0137] In the embodiment of the present application, the uniformly random scaling can be understood as determining the weight scaling value by uniformly random sampling, as shown in equation (15), which can be but is not limited to:

[0138]

[0139] The Gaussian random scaling can be understood as determining the weight scaling value by Gaussian random sampling, as shown in equation (16), which can be but is not limited to:

[0140]

[0141] The quadratic random scaling can be understood as determining the weight scaling value by parabolic random sampling, as shown in equation (17), which can be but is not limited to:

[0142]

[0143] For example, the parameters of the reward function in the embodiment of the present application are shown in Table 1, where a i and b i , (0≤i≤3), are respectively the lower limit and the upper limit of the random scaling of w . The embodiment of the present application can directly use the PBRS, the DPBRS_U, the DPBRS_N and the DPBRS_P to respectively represent the potential-based reward shaping, the dense reward function based on the uniformly random scaling dynamic potential, the dense reward function based on the Gaussian random scaling dynamic potential and the dense reward function based on the quadratic random scaling dynamic potential. Table 1 is a reward function parameter setting table provided according to an embodiment of the present application.

[0144] Table 1

[0145] Parameter name Value Parameter name Value Parameter name Value w0 7.5 σ2 1.0 a3 0.8 w1 2.0 p0 0.1 b0 1.37 w2 1.0 p1 5e-7 b1 1.6 w3 0.5 p2 0.3 b2 1.2 w4 -5.0 a0 0.67 b3 1.2 σ0 0.5 a1 0.4 σ1 0.667 a2 0.8

[0146] ​Further, the embodiment of the present application can obtain reinforcement learning training results of different reward functions. In the embodiment of the present application, the total number of samples is 1.6 million times, and the comparison test results of PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), TRPO (Trust Region Policy Optimization) and A2C reinforcement learning algorithms based on PBRS and DPBRS_U are shown in Table 2. Table 2 is a comparison table of cumulative rewards of common reinforcement learning algorithms based on PBRS and DPBRS_U according to an embodiment of the present application.

[0147] Table 2

[0148] PBRS DPBRS_U PPO 1831.8 1843.3 SAC 1759.9 1768.0 TRPO 1785.8 1720.4 A2C 456.0 906.5

[0149] From Table 2, it can be seen that the embodiment of the present application DPBRS_U can obtain higher cumulative rewards than PBRS, and can obtain certain performance improvement for SAC, PPO and A2C algorithms, but the best performance is PPO algorithm, so the subsequent test experiment adopts PPO algorithm. The test experiment is performed using PPO algorithm, and each interaction with the environment for 2048 times is a training round, then all samples of the round are used to train the neural network to update the policy, and then the fixed reward is calculated according to the formula (15) to evaluate the performance. The test shows that when the average reward of the robot per time step is above 0.9, the goal of walking at the command speed can be achieved, so the cumulative reward of 1800 at the maximum time of 2000 is taken as the benchmark to judge whether the robot achieves the experimental goal. The parameter settings of the PPO algorithm are shown in Table 3. Table 3 is a parameter setting table of the PPO algorithm according to an embodiment of the present application.

[0150] Table 3

[0151] Parameter name Value Value network hidden layer [256,128,32] Actor network hidden layer [256,128,32] Discount factor γ 0.99 Learning rate lr 1e-4 Truncation ε 0.1 Samples per episode n_steps 2048 Batch size mini batch 64 Updates per episode n_epoch 10

[0152] Since the agent needs to learn to control the robot to walk at multiple command speeds, the reward value will fluctuate greatly during training, so the average reward of 10 rounds of iterative learning is used as the evaluation reward for comparison test. Figure 4 is the comparison test result of the evaluation reward change of the PPO algorithm using PBRS, DPBRS_U, DPBRS_N and DPBRS_P reward functions respectively for 650 training rounds.

[0153] From Figure 4It can be seen that the PPO algorithm using three DPBRS reward functions in this application embodiment has a faster convergence speed and can reach the target (1800 cumulative reward value) more quickly than the PPO algorithm using the PBRS reward function. Specifically, the DPBRS_U reward function showed the best performance, achieving a maximum cumulative reward of 1801 in 538 training rounds; the DPBRS_N reward function achieved a maximum cumulative reward of 1754 in 632 training rounds; the DPBRS_P reward function resulted in the smoothest training process with relatively small curve fluctuations, achieving a maximum cumulative reward of 1793 in 645 training rounds; while the PBRS reward function exhibited the largest curve oscillation amplitude, achieving only a maximum cumulative reward of 1655 in 554 training rounds, indicating inferior training performance compared to the other three. Test results show that using the DPBRS reward function can improve exploration performance and accelerate training speed.

[0154] In step S102, based on robot information and initial state information, the robot is subjected to at least one round of robot control interaction. In each round of control interaction, based on the robot's control commands and state information, the dense reward value of the robot constructed based on dynamic potential energy is calculated.

[0155] As one possible implementation method, the embodiments of this application can perform at least one round, such as N (N≥1) rounds of robot control interaction. In each round of control interaction, the initial state information of the robot is first obtained, then control commands are sent, new state information is received, dense reward value constructed based on dynamic potential energy is calculated, and the interaction continues until the robot control interaction is completed.

[0156] Optionally, in one embodiment of this application, performing at least one round of robot control interaction on the robot includes: acquiring state information at each moment of the nth round during the robot control interaction; and determining the termination time T of the nth round of robot control interaction. n Based on the termination time T n The state information sequence M generated by the robot control interaction is obtained by combining the state information at each time step in the nth round. n Where n = 1, 2, 3, ..., N, M n For the nth round of robot control interaction, from 0 to T n The sequence of all state information at each time step, where N is the total number of training rounds; based on the state information sequence M... n The robot control training process generates a sequence of state information for all N rounds, {M1, M2, ..., M}. n ,…,M N}

[0157] In some embodiments of this application, a complete MDP can be established first. This can be understood as follows: in this embodiment, the reinforcement learning problem is usually modeled as an MDP, and its calculation formula can be, but is not limited to, expressed as:

[0158] M =<S,A,P,R> (18)

[0159] Where S is the set of observed states of the environment, A is the set of actions that the agent can take, P is the probability of environmental transition, and R is the reward function.

[0160] Furthermore, in this embodiment of the application, reinforcement learning can train actions through multiple rounds of iterative training, selecting an Action neural network. The parameter set of the Action neural network is denoted as θ, its input is state information s (s∈S), and its output is the action a (a∈A) taken under the current state information s. The action selection strategy described by the Action neural network is denoted as π (a|s), that is, how to select action a (a=π) based on state information s. θ (a|s). In the nth (n≥0) round of robot control interaction, the agent, under the initial state information s0, follows the current policy π. θ (a|s) The agent executes action a0, receives an immediate reward r0 from the environment, and its state information is transferred to s1 with probability P, completing the first control interaction (also known as sampling at time step t=0); under the new state information s1, the agent follows the current policy π. θ (a|s) Execute action a1, the environment provides an immediate reward r1, and the state information is transferred to s2 with probability P, completing the second control interaction (also known as sampling at time step t=1); repeat the above control interaction process until the termination time T. n The current iteration of learning ends when the robot falls or the time step exceeds the given maximum value, forming the MDP for the nth iteration, denoted as M. n ,

[0161]

[0162] Among them, T n It is the length of the MDP, and the maximum time step t. max =T n The goal of reinforcement learning is to learn the optimal strategy. This maximizes the cumulative reward of the trajectory.

[0163]

[0164] Where γ is the discount factor.

[0165] In step S103, the initial control policy neural network of the robot is trained based on the dense reward value constructed based on the dynamic potential energy and the training data generated in the control interaction process to obtain a final control policy neural network.

[0166] As a possible implementation manner, the initial control policy neural network π θ of the robot can be established by using an MLP neural network. φ Further, the embodiment of the application can obtain N pieces of data {s t , a t , r t} in combination with the dense reward value constructed based on the dynamic potential energy and the training data generated in the control interaction process, wherein r t is obtained by DPBRS reward calculation.

[0167] Further, the embodiment of the application calculates the advantage function and uses the following formula to update the initial control policy neural network parameter θ for M times, and the expression can be but is not limited to:

[0168]

[0169] Wherein, KL[π old |π θ ] is limited in a range [1 / α, α], and 1 / α is taken when it is lower than the lower limit, and α is taken when it is higher than the upper limit.

[0170] The initial value network parameter is updated using the following formula for B times, and the expression can be but is not limited to:

[0171]

[0172] Further, the final control policy neural network is obtained.

[0173] Optionally, in an embodiment of the application, the initial control policy neural network of the robot is trained based on the dense reward value constructed based on the dynamic potential energy and the training data generated in the control interaction process, comprising: obtaining a parameter set of the initial control policy neural network, and determining a target function of the parameter set based on the parameter set; generating an update rule for updating the parameter set based on the target function; and training the initial control policy neural network by the update rule in combination with the dense reward constructed based on the dynamic potential energy and the training data generated in the control training process. Wherein, the expression of the target function can be but is not limited to:

[0174]

[0175] Wherein, P(M n , θ D ) is the parameter set θ Dan initial control policy neural network controls the robot with a sequence of state information M n a probability of occurrence, π θ (a t | s t ) is a parameter set θ D of the initial control policy neural network, and a probability of outputting a robot control instruction a t when the state information is s t , R(M n ) is a dynamic dense reward sum of M n ;

[0176] The expression of the update rule can be but is not limited to:

[0177]

[0178] wherein, α is a step size, δ is an expected deviation of a dynamic weight relative to a standard weight, and is a random value obtained after sampling.

[0179] In some embodiments, the present application embodiment causes M n to be the n th (n≥0) round of iterative learning process, and the goal of reinforcement learning is to adjust the parameter set θ of the action selection neural network Action through multiple rounds of iterative learning, so that a higher reward value M n has a higher probability of occurrence. Therefore, the objective function J(θ) of the policy gradient of reinforcement learning can be but is not limited to:

[0180]

[0181] wherein, P(M n , θ) represents a probability of occurrence of the trajectory M n under the current behavior selection policy, and the calculation formula can be but is not limited to:

[0182]

[0183] wherein, P(s t+1 | s t , a t ) is a state transition probability determined by the environment. The derivative of the objective function J(θ) can be but is not limited to:

[0184]

[0185] It can be known that the present application embodiment is difficult to realize the summation of all possible M n , and the expectation of multiple trajectory samples is used to approximate in the reinforcement learning. Therefore, the rule of the policy gradient update can be but is not limited to:

[0186]

[0187] wherein, a is a step size.

[0188] Further, when the DPBRS reward function is used in the embodiment of the present application, the parameter set of the action neural network is denoted as θ D , the objective function is denoted as J(θ D ), and the following equation is obtained:

[0189]

[0190] At this time, in the embodiment of the present application, the corresponding policy gradient update rule can be, but is not limited to, the following equation:

[0191]

[0192] wherein, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

[0193] Therefore, due to the existence of the dynamic potential, in the iterative learning process of different rounds, even if the same state s (s∈S) appears, the calculated dense reward value is different, and the direction of policy update is also different, so that the learning process has better exploration.

[0194] According to the robot control reinforcement learning method based on dynamic potential reward provided in the embodiment of the present application, at least one round of robot control interaction can be performed on the robot according to the robot information and the initial state information of the robot, and in each round of control interaction, the dense reward value of the robot based on the dynamic potential is calculated, and then the initial control policy neural network of the robot is trained to obtain the final control policy neural network. The reward value obtained by dynamically adjusting the current action control of the robot based on the dense reward value constructed based on the dynamic potential, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the motion posture of the robot is more natural and stable. Therefore, the problems in the related art that the potential function is fixed, the reward obtained by each state in the entire training process is determined, the policy exploration is limited, and the robot cannot be controlled complexly are solved.

[0195] Secondly, the robot control reinforcement learning device based on dynamic potential reward provided in the embodiment of the present application is described with reference to the accompanying drawings.

[0196] Figure 5 The block schematic diagram of the robot control reinforcement learning device based on dynamic potential reward provided in the embodiment of the present application is shown.

[0197] As Figure 5As shown, the robot control reinforcement learning device 50 based on dynamic potential energy reward is applied in the model training stage. The device 50 includes: a first acquisition module 501, a first calculation module 502, and a training module 503.

[0198] The first acquisition module 501 is used to acquire robot information and initial state information of the robot.

[0199] The first calculation module 502 is used to perform at least one round of robot control interaction on the robot based on robot information and initial state information. In each round of control interaction, the dense reward value of the robot based on dynamic potential energy is calculated based on the robot's control instructions and state information.

[0200] Training module 503 is used to train the robot's initial control policy neural network by combining dense reward values ​​constructed based on dynamic potential energy and training data generated by the control interaction process, so as to obtain the final control policy neural network.

[0201] Optionally, in one embodiment of this application, the first calculation module 502 includes: a first acquisition unit, a determination unit, and a second generation unit.

[0202] The first acquisition unit is used to acquire the state information of each moment in the nth round of robot control interaction.

[0203] The determining unit is used to determine the termination time T of the nth round of robot control interaction. n The first generation unit is used to generate data based on the termination time T. n The state information sequence M generated by the robot control interaction is obtained by combining the state information at each time step in the nth round. n Where n = 1, 2, 3, ..., N, M n For the nth round of robot control interaction, from 0 to T n The sequence of all state information at each time step, where N is the total number of training rounds.

[0204] The second generation unit is used to generate data based on the state information sequence M. n The robot control training process generates a sequence of state information for all N rounds, {M1, M2, ..., M}. n ,…,M N}

[0205] Optionally, in one embodiment of this application, it further includes: a second calculation module, a first determining module, a second determining module, or a third determining module, a first building module, and a second building module.

[0206] The second calculation module is configured to calculate the potential energy of the robot according to current state information of the robot before calculating the dense reward value of the robot based on the dynamic potential energy.

[0207] The first determination module is configured to determine a weight parameter of the potential energy by uniform random scaling based on the potential energy.

[0208] Alternatively, the second determination module is configured to determine the weight parameter of the potential energy by Gaussian random scaling based on the potential energy.

[0209] Alternatively, the third determination module is configured to determine the weight parameter of the potential energy by quadratic random scaling based on the potential energy.

[0210] The first construction module is configured to construct the dynamic potential energy of the robot based on the potential energy and the weight parameter.

[0211] The second construction module is configured to construct the dense reward function based on the dynamic potential energy.

[0212] Optionally, in an embodiment of the present application, the dense reward function based on the dynamic potential energy can be but is not limited to:

[0213] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t<T n-1 ),

[0214] wherein F D (s t ,a t ,s t+1 ) is the dense reward based on the dynamic potential energy, s t is state information of the robot at time t, a t is a control instruction sent to the robot for s t , s t+1 is new state information of the robot after responding to a t , γ is a discount factor, and Φ D is the dynamic potential energy, that is, the reward r t at time t.

[0215] Optionally, in an embodiment of the present application, the calculation formula of the dynamic potential energy can be but is not limited to:

[0216]

[0217] wherein w i(t) (0≤i≤3) are dynamic weight parameters, initial weight parameters, which can be initialized according to robot control objectives, P ctrl energy consumption penalty, P contact foot contact force penalty, P Δy is a yaw penalty.

[0218] Optionally, in an embodiment of the present application, the calculation formula of potential energy can be but is not limited to:

[0219]

[0220] wherein, is to encourage the robot to walk at the command speed v0, is to encourage the agent to walk in the command direction θ0, is a survival reward, is a foot sole orientation reward, which encourages the robot to put the foot sole flat and not to walk on tiptoe, σ0, σ1, σ2 are adjustment parameters, K is a normal number used to reward or punish whether the robot falls down, h is the height of the robot's center of gravity, h0 is the height of the robot's center of gravity at time 0, v x is the moving speed of the robot's center of gravity, v0 is the moving speed of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot's torso and the vertical direction, θ0 is the angle between the robot's torso and the vertical direction at time 0, fl or is the left foot orientation of the robot represented by a quaternion, fl or0 is the left foot orientation of the robot at time 0, fr or is the right foot orientation of the robot represented by a quaternion, fr or0 is the right foot orientation of the robot at time 0, s0 is the state information of the robot at time 0, s t is the state information of the robot at time t.

[0221] Optionally, in an embodiment of the present application, the calculation formula of uniform random scaling can be but is not limited to:

[0222]

[0223] wherein a is the lower limit of scaling, b is the upper limit of scaling, U i is a uniform distribution on the interval [a, b];

[0224] The calculation formula of Gaussian random scaling can be but is not limited to:

[0225]

[0226] wherein a is the mean of scaling, b is the variance of scaling, N iA Gaussian distribution with mean a and variance b;

[0227] The calculation formula of the quadratic random scaling can be but is not limited to:

[0228]

[0229] wherein a is a lower limit of scaling, b is an upper limit of scaling, P i is a quadratic distribution with a lower limit a and an upper limit b.

[0230] Optionally, in an embodiment of the present application, the training module 300 comprises a second acquisition unit, a generation unit and a training unit.

[0231] The second acquisition unit is configured to acquire a parameter set of an initial control policy neural network, and determine a target function of the parameter set based on the parameter set.

[0232] The generation unit is configured to generate an update rule for updating the parameter set based on the target function.

[0233] The training unit is configured to train the initial control policy neural network through the update rule in combination with dense rewards constructed based on dynamic potential energy and training data generated in a control training process.

[0234] Optionally, in an embodiment of the present application, the expression of the target function can be but is not limited to:

[0235]

[0236] wherein P(M n , θ D ) is a probability of occurrence of the state information sequence M n when the initial control policy neural network with the parameter set θ D is used to control the robot, π θ (a t | s t ) is a probability of outputting the robot control instruction a t when the initial control policy neural network with the parameter set θ D is in the state information s t , and R(M n ) is a dynamic dense reward sum of M n ;

[0237] The expression of the update rule can be but is not limited to:

[0238]

[0239] wherein α is a step size, δ is an expected deviation of a dynamic weight relative to a standard weight, and is a random value obtained after sampling.

[0240] It should be noted that the foregoing explanation of the embodiment of the robot control reinforcement learning method based on dynamic potential energy reward also applies to the embodiment of the robot control reinforcement learning device based on dynamic potential energy reward, which will not be described here.

[0241] The robot control reinforcement learning device based on dynamic potential energy reward provided by the embodiment of the application can perform at least one round of robot control interaction on the robot according to the robot information and the initial state information of the robot, and in each round of control interaction, the dense reward value constructed based on the dynamic potential energy of the robot is calculated, and then the initial control policy neural network of the robot is trained to obtain the final control policy neural network. The reward value obtained by dynamically adjusting the current action control of the robot based on the dense reward value constructed based on the dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the motion posture of the robot is more natural and stable. Thus, the problems in the related art that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, the strategy exploration is limited, and the robot cannot be controlled complexly are solved.

[0242] The above embodiment describes the model training phase, and the following describes the embodiment of the model application phase.

[0243] Figure 6 The flowchart of another robot control reinforcement learning method based on dynamic potential energy reward provided by the embodiment of the application is shown.

[0244] As shown in Figure 6 , the robot control reinforcement learning method based on dynamic potential energy reward is applied to the model application phase, and the method includes the following steps:

[0245] In step S601, the actual robot information and the actual initial state information of the target robot are obtained.

[0246] In step S602, the actual robot information and the actual initial state information are input into the pre-trained final control policy neural network to obtain the control instruction of the target robot, wherein the pre-trained final control policy neural network is obtained by training based on the dense reward constructed based on the dynamic potential energy.

[0247] In the actual execution process, the actual robot information and the actual initial state s t of the target robot can be input into the pre-trained final control policy neural network, and the control action a t is output, the state information of the robot is changed to s t+1 , and the process is repeated continuously, so that the continuous walking control of the target robot can be realized.

[0248] For example, the embodiment of the present application can analyze the walking test results of the robot by building an environment for different reward functions.

[0249] The test experiment compares and tests from three aspects of instruction speed tracking accuracy, walking straightness and stability. After 10 rounds of testing, the average reward is 1831.8 when the PBRS reward function is used, and the average reward is 1843.3 when the DPBRS_U reward function is used.

[0250] Among them, the analysis of the instruction speed tracking accuracy test result of the embodiment of the present application is as follows:

[0251] The average speed and standard deviation of the embodiment of the present application when the DPBRS_U and PBRS reward functions are used under five instruction speeds are shown in Table 4, and the maximum error is 1.4% and 2.4%, respectively. In addition, the instruction speed is gradually increased at intervals of 400 time steps in the same test to detect the accuracy of speed tracking when the two reward functions are used, and the test result is shown in Table 6. Figure 7 From Table 4 and Figure 7 It can be seen that the embodiment of the present application executes the instruction speed more accurately when the DPBRS_U reward function is used, and the walking speed is more stable, and the maximum speed fluctuation is about 8.5%; and the walking speed is not stable when the PBRS reward function is used, and the maximum speed fluctuation is 13.8%. Table 4 is a table of average speed and standard deviation of robot walking when different reward functions are used according to an embodiment of the present application.

[0252] Table 4

[0253] v0 DPBRS_U Vx (m / s) PBRS Vx (m / s) 0.3 0.303±0.026 0.305±0.042 0.4 0.402±0.025 0.404±0.038 0.5 0.488±0.030 0.493±0.045 0.6 0.598±0.038 0.592±0.061 0.7 0.692±0.045 0.693±0.053

[0254] The analysis of the walking straightness test result of the embodiment of the present application is as follows:

[0255] The walking straightness of the embodiment of the present application mainly tests the ability of the robot to walk in a specified direction. The average value and standard deviation of the yaw angle when the DPBRS_U and PBRS reward functions are used under five instruction speeds are shown in Table 5. From Table 5, it can be seen that the maximum yaw angle is 0.035 and the minimum yaw angle is 0.006 when the DPBRS_U reward function is used; and the maximum yaw angle is 0.040 and the minimum yaw angle is -0.005 when the PBRS reward function is used. Too large or too small speed will cause instability of the yaw angle. The yaw is smaller when the DPBRS_U reward function is used. Table 5 is a table of average yaw angle and standard deviation of robot walking when different reward functions are used according to an embodiment of the present application.

[0256] Table 5

[0257] v0 DPBRS_U x (rad) PBRS x (rad) 0.3 0.024±0.015 -0.034±0.018 0.4 0.026±0.017 -0.005±0.014 0.5 -0.006±0.016 0.02±0.015 0.6 -0.012±0.017 0.035±0.017 0.7 0.035±0.018 0.04±0.024

[0258] The robot center of gravity yaw test results when the instruction speed gradually increases are shown in Figure 8 Figure 5 It can be seen that when the DPBRS_U reward function is used, the robot can maintain good straightness of walking, the center of gravity offset reaches a maximum of -0.0755 m at 201 steps, and then gradually decreases to 0.070 m at 1501 steps, and then gradually approaches 0, the center of gravity offset range angle is small, and there is no large fluctuation; when the PBRS reward function is used, the center of gravity offset reaches a maximum of -0.201 m at 896 steps, and then gradually decreases to 0.035 m at 1985 steps, the overall center of gravity offset range is larger, and the fluctuation is larger.

[0259] The walking stability test results of the embodiments of the present application are analyzed as follows:

[0260] When a human walks, the body center of gravity height will oscillate regularly to maintain the stability of walking and save energy, so the walking stability of the robot can be measured by testing the change of the center of gravity height of the robot when walking. The initial center of gravity height of the robot is 0.47, and the center of gravity height change curves of the robot when the instruction speed gradually increases and the DPBRS_U and PBRS reward functions are used are shown in Figure 9

[0261] From Figure 9 It can be seen that when the DPBRS_U reward function is used, the center of gravity height of the robot oscillates while the height is basically stable, and under the gradually increasing five instruction speeds, the average center of gravity height of the robot is 0.476, 0.480, 0.481, 0.484, and 0.487, respectively, the center of gravity height of the robot increases slightly while the walking speed increases, and the robot walks in a fully stretched leg manner; and when the PBRS reward function is used, the center of gravity height of the robot oscillates while gradually decreasing, and under the gradually increasing five instruction speeds, the average center of gravity height of the robot is 0.461, 0.460, 0.455, 0.450, and 0.438, respectively, when the instruction speed increases to 0.7, the center of gravity height decreases greatly, the robot cannot stretch the legs, and the walking stability decreases.

[0262] Both the two reward functions do not explicitly guide the robot to walk in which gait, but the test results show that the robot using the DPBRS_U reward function can learn a more natural and coordinated gait from the implicit information of the reward function. Figure 10 and Figure 11 are respectively the gait effect diagrams of the robot when the PBRS and DPBRS_U reward functions are used (the instruction speed is 0.4).

[0263] Comparing Figure 10 and Figure 11 ​​It can be seen that, when the robot adopts the DPBRS_U reward function, the gait of the robot is closer to the human walking gait, and when the robot adopts the PBRS reward function, the gait of the robot is similar to single-foot jumping, which also shows that the DPBRS_U reward function can encourage the agent to explore more aspects in the reinforcement learning process, so that the robot can walk in a more natural and stable gait

[0264] The robot control reinforcement learning method based on dynamic potential energy reward provided in the embodiments of the present application can input the actual robot information and the actual initial state information into the pre-trained final control policy neural network, and then obtain the control instruction of the target robot. The reward value obtained by the current action control of the robot is dynamically adjusted based on the dense reward value constructed based on the dynamic potential energy, so as to improve the exploration ability of the learning process, effectively improve the training speed, and make the motion posture of the robot more natural and stable. Therefore, the problems in the related art that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, the strategy exploration is limited, and the robot cannot be controlled complexly are solved.

[0265] Secondly, the robot control reinforcement learning device based on dynamic potential energy reward provided in the embodiments of the present application is described with reference to the accompanying drawings.

[0266] Figure 12 The block schematic diagram of the robot control reinforcement learning device based on dynamic potential energy reward provided for another embodiment of the present application is shown.

[0267] As Figure 12 shown, the robot control reinforcement learning device 120 based on dynamic potential energy reward is applied to a model application stage, wherein the device 120 comprises a second acquisition module 1201 and an input module 1202.

[0268] The second acquisition module 1201 is configured to acquire the actual robot information and the actual initial state information of the target robot.

[0269] The input module 1202 is configured to input the actual robot information and the actual initial state information into a pre-trained final control policy neural network to obtain the control instruction of the target robot, wherein the pre-trained final control policy neural network is trained based on the dense reward constructed based on the dynamic potential energy.

[0270] It should be noted that the foregoing explanation and description of the robot control reinforcement learning method based on dynamic potential energy reward also apply to the robot control reinforcement learning device based on dynamic potential energy reward, which will not be described here again.

[0271] The robot control reinforcement learning device based on dynamic potential energy reward provided by the embodiment of the application can input actual robot information and actual initial state information into a pre-trained final control strategy neural network, and then obtain a control instruction of a target robot. The reward value obtained by dynamically adjusting the current action control of the robot based on the dense reward value constructed based on the dynamic potential energy is used, so as to improve the exploration ability of the learning process, effectively improve the training speed, and make the motion posture of the robot more natural and stable. Therefore, the problems in the related art that the potential energy function is fixed, the reward obtained by each state in the entire training process is determined, the strategy exploration is limited, and the robot cannot be controlled in a complex manner are solved.

[0272] Figure 13 A structural schematic diagram of an electronic device provided by the embodiment of the application is shown. The electronic device can include:

[0273] The memory 1301, the processor 1302, and a computer program stored in the memory 1301 and executable on the processor 1302.

[0274] The processor 1302 implements the robot control reinforcement learning method based on dynamic potential energy reward provided in the above embodiment when executing the program.

[0275] Further, the electronic device further includes:

[0276] The communication interface 1303 is used for communication between the memory 1301 and the processor 1302.

[0277] The memory 1301 is used to store a computer program executable on the processor 1302.

[0278] The memory 1301 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0279] If the memory 1301, the processor 1302, and the communication interface 1303 are independently implemented, the communication interface 1303, the memory 1301, and the processor 1302 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation,Figure 13 Only one bus or bus type is used in some of the drawings to illustrate the architecture, the bus can be replaced by any other type of communications link, such as a digital network.

[0280] Optionally, in a specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a chip, the memory 1301, the processor 1302 and the communication interface 1303 can complete the communication among each other through an internal interface.

[0281] The processor 1302 can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the application.

[0282] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the robot control reinforcement learning method based on dynamic potential energy reward.

[0283] The embodiment of the application further provides a computer program product, which comprises a computer program, and the program is executed to implement the robot control reinforcement learning method based on dynamic potential energy reward.

[0284] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0285] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one feature. In the description of the application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0286] Any processes or methods described in the flowcharts or elsewhere herein can be understood as representing a module, segment, or portion of code that includes one or N steps for implementing the specified logical functions or processes. The scope of a preferred embodiment of the present application encompasses combinations with additional steps as appropriate, and the steps performed in the manner described or otherwise discussed herein, including in an order other than as described, including in an order substantially simultaneous with one another, or in the reverse order, depending upon the functionality involved. These and other variations are considered to be within the scope of the present application.

[0287] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing, and can be a machine-readable storage medium (alternatively, the medium can be a machine-readable signal medium). The computer-readable medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the medium is a suitable medium with the program printed on the medium. The computer-readable medium can be a non-transitory computer-readable medium. The term "non-transitory" can also mean "not entirely transitory" or "not entirely non-transitory," which can include, for example, a transitory signal that can be embodied as a carrier wave, a bit, a data packet, and / or other permeable data transmission signals that can be a basis, at least in part, for a program product.

[0288] It should be understood that aspects of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0289] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0290] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0291] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A robot control reinforcement learning method based on dynamic potential reward, characterized in that, The method is applied to a model training stage, and the method comprises the following steps: Obtain robot information and initial state information of a robot; Based on the robot information and the initial state information, at least one round of robot control interaction is performed on the robot, and in each round of control interaction, a dense reward value of the robot based on dynamic potential energy is calculated based on control instructions and state information of the robot; The initial control policy neural network of the robot is trained in combination with the dense reward value based on dynamic potential energy and training data generated in the control interaction process, so as to obtain a final control policy neural network.

2. The method of claim 1, wherein, The at least one round of robot control interaction comprises: Obtain state information at each time point in the n th round of the robot control interaction process; determining a termination time T at which the nth round of robot-controlled interaction terminates n ; based on the termination time T n and the state information of each time in the nth round, the state information sequence M generated by the robot control interaction is obtained n , wherein n=1, 2, 3,..., N, M n is a sequence composed of all state information from 0 to T n in the nth round of robot control interaction, and N is the total training rounds; Based on the state information sequence M n Get N rounds of robot control training to generate all round state information sequences {M1, M2, …, M n ,…,M N}.

3. The method of claim 1, wherein, Before calculating the dense reward value of the robot based on dynamic potential energy, the method further comprises the following steps: Calculate the potential energy of the robot according to the current state information of the robot; Based on the potential energy, the weight parameter of the potential energy is determined through uniform random scaling; Or, based on the potential energy, the weight parameter of the potential energy is determined through Gaussian random scaling; Or, based on the potential energy, the weight parameter of the potential energy is determined through quadratic random scaling; Based on the potential energy and the weight parameter, the dynamic potential energy of the robot is constructed. The dense reward function based on dynamic potential energy is constructed based on the dynamic potential energy.

4. The method of claim 3, wherein, The calculation formula of the dynamic potential energy is: F D (s t , a t , s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t<T n-1 ), where F D (s t , a t ) is the dense reward based on dynamic potential, s t+1 is the state information of the robot at time t, a t is the control instruction sent to the robot for s t , s t is the new state information of the robot after responding to a t+1 , γ is the discount factor, Φ t is the dynamic potential, that is, the reward r D at time t. t ​ 5. The method of claim 3, wherein, The calculation formula of the potential energy is: where w i (t) are dynamic weight parameters, initial weight parameters, which can be initialized according to the robot control objectives, P ctrl is an energy consumption penalty, P contact is a foot contact force penalty, P Δy is a yaw penalty.

6. The method of claim 3, wherein, Wherein, wherein, for encouraging the robot to walk at the commanded velocity v0, for encouraging the agent to walk in the commanded direction θ0, for the survival reward, for the foot placement reward, encouraging the robot to place the feet flat on the ground and not to walk on tiptoes, σ0, σ1, σ2 are tuning parameters, K is a positive number used to reward or penalize the robot for falling down, h is the height of the robot's center of mass, h0 is the height of the robot's center of mass at time 0, v x is the velocity of the robot's center of mass, v0 is the velocity of the robot's center of mass at time 0, x is the walking direction, θ z is the angle between the robot's torso and the vertical direction, θ0 is the angle between the robot's torso and the vertical direction at time 0, fl or is the orientation of the robot's left foot represented by a quaternion, fl or0 is the orientation of the robot's left foot at time 0, fr or is the orientation of the robot's right foot represented by a quaternion, fr or0 is the orientation of the robot's right foot at time 0, s0 is the state information of the robot at time 0, s t is the state information of the robot at time t.

7. The method of claim 3, wherein, The calculation formula of the uniform random scaling is: The calculation formula of the Gaussian random scaling is: where a is a lower bound of scaling, b is an upper bound of scaling, U i is a uniform distribution over the interval [a, b] The calculation formula of the quadratic random scaling is: where a is a scaling mean and b is a scaling variance, N i is a Gaussian distribution with mean a and variance b. The initial control policy neural network of the robot is trained in combination with the dense reward value based on dynamic potential energy and training data generated in the control interaction process, so as to obtain a final control policy neural network. where a is a lower bound of scaling, b is an upper bound of scaling, P i is a quadratic distribution with lower bound a and upper bound b.

8. The method of claim 1, wherein, Obtain a parameter set of the initial control policy neural network, and determine a target function of the parameter set based on the parameter set; Based on the target function, an update rule for updating the parameter set is generated; The initial control policy neural network is trained through the update rule in combination with the dense reward based on dynamic potential energy and the training data generated in the control training process. Wherein, 9. The method of claim 8, wherein, The expression of the target function is: The expression of the update rule is: wherein P(M n , θ D ) is the probability of the state information sequence M D occurring when the initial control policy neural network controls the robot using the parameter set θ n , π θ (a t |s t ) is the probability of the initial control policy neural network outputting the robot control instruction a D when the state information is s t , and the parameter set is θ t , and R(M n ) is the dynamic dense reward sum of M n . Wherein, α is a step length, δ is an expected deviation of a dynamic weight relative to a standard weight, and is a random value obtained after sampling. The method is applied to a model application stage, and the method comprises the following steps:

10. A robot control reinforcement learning method based on dynamic potential reward, characterized in that, Obtain actual robot information and actual initial state information of a target robot; The actual robot information and the actual initial state information are input into a pre-trained final control policy neural network to obtain control instructions of the target robot, wherein the pre-trained final control policy neural network is trained based on a dense reward based on dynamic potential energy. ​

Images