Robot control reinforcement learning method and device based on dynamic potential energy reward

Abstract

The invention relates to the technical field of computer science, in particular to a robot control reinforcement learning method and device based on dynamic potential energy reward, and the method comprises the steps: obtaining the robot information and initial state information of a robot; based on the robot information and the initial state information, at least one round of robot control interaction is carried out on the robot, and in each round of control interaction, a dense reward value constructed based on dynamic potential energy of the robot is calculated based on the control instruction and the state information of the robot; and training an initial control strategy neural network of the robot by combining a dense reward value constructed based on the dynamic potential energy and training data generated in a control interaction process so as to obtain a final control strategy neural network. Therefore, the problems that in the related technology, the potential energy function is fixed, so that the award obtained by each state in the whole training process is determined, strategy exploration is limited, and complex control cannot be carried out on the robot are solved.

Description

Technical Field

[0001] The present application relates to the field of computer science and technology, and in particular to a robot control reinforcement learning method and device based on dynamic potential energy rewards. Background Art

[0002] Reinforcement learning has achieved great success in video games, chess, financial modeling, and language models with its powerful feature extraction and decision-making capabilities. Using deep reinforcement learning-based methods to generate robot control strategies has been a research hotspot in recent years, but in the design process of the entire system, how to design a reward function that can clearly reflect the researcher's expectations so that the robot can learn various skills is a challenging task.

[0003] In the related technologies, Siekmann et al. designed a reward function based on reference trajectory deviation, so that the Cassie robot learned to walk; Peng et al. used the motion redirection method to generate animation, designed a reward function that combined imitation trajectory and reinforcement goals, and enabled the robot to learn various actions; Kalashnikov et al. collected data as priors in a real robotic arm, and used sparse rewards 1 and 0 to assign rewards to whether the robotic arm successfully grasped; Gym's standard reinforcement learning environment BipedalWalker designed positive rewards based on the moving distance, and in order to avoid invalid actions, a penalty term was designed based on the torque size of each action; PBRS (Pontential-Base Reward Shaping) proposed a normative reward function design method, which can accelerate the convergence of the intelligent agent to the optimal strategy and achieve multi-objective learning.

[0004] However, in the related technology, since the potential energy function is fixed, the reward obtained for each state during the entire training process is determined, which in turn limits strategy exploration and makes it impossible to perform complex control of the robot, which urgently needs improvement. Summary of the invention

[0005] The present application provides a robot control reinforcement learning method and device based on dynamic potential energy rewards to solve the problem in related technologies that, due to the fixed potential energy function, the reward obtained for each state in the entire training process is fixed, which in turn limits strategy exploration and makes it impossible to perform complex control of the robot.

[0006] The first aspect of the present application provides a robot control reinforcement learning method based on dynamic potential energy rewards, which is applied to the model training stage, wherein the method includes the following steps: obtaining robot information and initial state information of the robot; based on the robot information and the initial state information, performing at least one round of robot control interaction on the robot, and in each round of control interaction, calculating the dense reward value of the robot constructed based on dynamic potential energy based on the control instructions and state information of the robot; training the initial control strategy neural network of the robot in combination with the dense reward value constructed based on dynamic potential energy and the training data generated by the control interaction process to obtain the final control strategy neural network.

[0007] Optionally, in one embodiment of the present application, performing at least one round of robot control interaction on the robot includes: obtaining state information of each moment of the nth round of the robot control interaction; determining the termination time T of the nth round of robot control interaction termination; n Based on the termination time T n and the state information at each moment in the nth round to obtain the state information sequence M generated by the robot control interaction n , where n = 1, 2, 3, ..., N, M n 0 to T in the nth round of robot control interaction n The sequence of all state information at the moment, N is the total number of training rounds; based on the state information sequence M n Get N rounds of robot control training to generate all round state information sequences {M 1 ,M 2 ,…,M n ,…,M N}.

[0008] Optionally, in one embodiment of the present application, before calculating the dense reward value of the robot constructed based on dynamic potential energy, it also includes: calculating the potential energy of the robot according to the current state information of the robot; based on the potential energy, determining the weight parameters of the potential energy through uniform random scaling; or, based on the potential energy, determining the weight parameters of the potential energy through Gaussian random scaling; or, based on the potential energy, determining the weight parameters of the potential energy through quadratic random scaling; constructing the dynamic potential energy of the robot based on the potential energy and the weight parameters; and constructing a dense reward function based on dynamic potential energy based on the dynamic potential energy.

[0009] Optionally, in one embodiment of the present application, the intensive reward function based on dynamic potential energy may be, but is not limited to:

[0010] F D (s t ,a t,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t <T n-1 ),

[0011] Among them, F D (s t ,a t ,s t+1 ) is a dense reward based on dynamic potential, s t is the robot status information at time t, a t For s t The control instructions issued to the robot, s t+1 For the robot to respond to a t The new state information after, γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r at time t t .

[0012] Optionally, in one embodiment of the present application, the calculation formula of the dynamic potential energy may be, but is not limited to,:

[0013]

[0014] Among them, w i (t)(0≤i≤3) is the dynamic weight parameter, The initial weight parameter can be initialized according to the robot control target, P ctrl is the energy penalty, P contact Foot contact force penalty, P Δy It is a yaw penalty.

[0015] Optionally, in one embodiment of the present application, the calculation formula of the potential energy may be, but is not limited to,:

[0016]

[0017] in, To encourage the robot to move at the command speed v 0 walk, To encourage the agent to move in the direction of the instruction θ 0 walk, For survival rewards, The reward for the paw direction encourages the robot to keep its paw flat and not walk on tiptoes. 0 , σ 1 , σ 2 is the adjustment parameter, K is a positive constant used to reward or punish the robot if it falls, h is the height of the robot's center of gravity, and h 0is the height of the robot's center of gravity at time 0, ν x is the speed of the robot's center of gravity, ν 0 is the velocity of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot trunk and the vertical direction, θ 0 is the angle between the robot’s trunk and the vertical direction at time 0, fl or is the orientation of the robot's left foot expressed in quaternion, fl or0 is the direction of the robot's left foot at time 0, fr or is the orientation of the robot's right foot expressed as a quaternion, fr or0 is the direction of the robot's right foot at time 0, s 0 is the status information of the robot at time 0, s t is the status information of the robot at time t.

[0018] Optionally, in one embodiment of the present application, the calculation formula of the uniform random scaling may be but is not limited to:

[0019]

[0020] Among them, a is the lower limit of scaling, b is the upper limit of scaling, and U i is a uniform distribution on the interval [a,b];

[0021] The calculation formula of the Gaussian random scaling may be, but is not limited to,:

[0022]

[0023] Among them, a is the scaled mean, b is the scaled variance, N i is a Gaussian distribution with mean a and variance b;

[0024] The calculation formula of the secondary random scaling may be, but is not limited to,:

[0025]

[0026] Among them, a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with a lower bound and b upper bound.

[0027] Optionally, in one embodiment of the present application, the initial control strategy neural network of the robot is trained in combination with the dense reward value constructed based on dynamic potential energy and the training data generated in the control interaction process, including: obtaining a parameter set of the initial control strategy neural network, and determining an objective function of the parameter set based on the parameter set; generating an update rule for updating the parameter set based on the objective function; and training the initial control strategy neural network through the update rule in combination with the dense reward constructed based on dynamic potential energy and the training data generated in the control training process.

[0028] Optionally, in one embodiment of the present application, the expression of the objective function may be, but is not limited to:

[0029]

[0030] Among them, P(M n ,θ D ) is the parameter set θ D The initial control strategy of the neural network when controlling the robot is the state information sequence M n The probability of occurrence, π θ (a t |s t ) is the parameter set θ D The initial control strategy of the neural network is s t When the robot control instruction a is output t The probability of R(M n ) is M n The sum of dynamic intensive rewards;

[0031] The expression of the update rule may be, but is not limited to:

[0032]

[0033] Among them, α is the step size, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

[0034] The second aspect of the present application provides a robot control reinforcement learning method based on dynamic potential energy rewards, which is applied to the model application stage, wherein the method includes the following steps: obtaining actual robot information and actual initial state information of the target robot; inputting the actual robot information and the actual initial state information into a pre-trained final control strategy neural network to obtain control instructions for the target robot, wherein the pre-trained final control strategy neural network is obtained by intensive reward training based on dynamic potential energy construction.

[0035] The third aspect of the present application provides a robot control reinforcement learning device based on dynamic potential energy rewards, which is applied to the model training stage, wherein the device includes: a first acquisition module, used to acquire the robot information and initial state information of the robot; a first calculation module, used to perform at least one round of robot control interaction on the robot based on the robot information and the initial state information, and in each round of control interaction, calculate the dense reward value of the robot constructed based on dynamic potential energy based on the control instructions and state information of the robot; a training module, used to train the initial control strategy neural network of the robot in combination with the dense reward value constructed based on dynamic potential energy and the training data generated by the control interaction process to obtain the final control strategy neural network.

[0036] Optionally, in one embodiment of the present application, the first calculation module includes: a first acquisition unit, used to acquire the state information of each moment of the nth round of the robot control interaction process; a determination unit, used to determine the termination time T of the nth round of robot control interaction termination. n ; A first generating unit, for based on the termination time T n and the state information at each moment in the nth round to obtain the state information sequence M generated by the robot control interaction n , where n = 1, 2, 3, ..., N, M n 0 to R in the nth round of robot control interaction n A sequence of all state information at the moment, N is the total number of training rounds; a second generating unit, for generating a sequence of all state information at the moment, N is the total number of training rounds; n Get N rounds of robot control training to generate all round state information sequences {M 1 ,M 2 ,…,M n ,…,M N}.

[0037] Optionally, in one embodiment of the present application, it also includes: a second calculation module, used to calculate the potential energy of the robot according to the current state information of the robot before calculating the dense reward value of the robot constructed based on dynamic potential energy; a first determination module, used to determine the weight parameters of the potential energy based on the potential energy through uniform random scaling; or, a second determination module, used to determine the weight parameters of the potential energy based on the potential energy through Gaussian random scaling; or, a third determination module, used to determine the weight parameters of the potential energy based on the potential energy through quadratic random scaling; a first construction module, used to construct the dynamic potential energy of the robot based on the potential energy and the weight parameters; a second construction module, used to construct a dense reward function based on dynamic potential energy based on the dynamic potential energy.

[0038] Optionally, in one embodiment of the present application, the intensive reward function based on dynamic potential energy may be, but is not limited to:

[0039] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t <T n-1 ),

[0040] Among them, F D (s t ,a t ,s t+1 ) is a dense reward based on dynamic potential, s t is the robot status information at time t, a t For s t The control instructions issued to the robot, s t+1 For the robot to respond to a t The new state information after, γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r at time t t .

[0041] Optionally, in one embodiment of the present application, the calculation formula of the dynamic potential energy may be, but is not limited to,:

[0042]

[0043] Among them, w i (t)(0≤i≤3) is the dynamic weight parameter, The initial weight parameter can be initialized according to the robot control target, P ctrl is the energy penalty, P contact Foot contact force penalty, P Δy It is a yaw penalty.

[0044] Optionally, in one embodiment of the present application, the calculation formula of the potential energy may be, but is not limited to,:

[0045]

[0046] in, To encourage the robot to move at the command speed v 0 walk, To encourage the agent to move in the direction of the instruction θ 0 walk, For survival rewards, The reward for the paw direction encourages the robot to keep its paw flat and not walk on tiptoes.0 , σ 1 , σ 2 is the adjustment parameter, K is a positive constant used to reward or punish the robot if it falls, h is the height of the robot's center of gravity, and h 0 is the height of the robot's center of gravity at time 0, ν x is the speed of the robot's center of gravity, ν 0 is the velocity of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot trunk and the vertical direction, θ 0 is the angle between the robot’s trunk and the vertical direction at time 0, fl or is the orientation of the robot's left foot expressed in quaternion, fl or0 is the direction of the robot's left foot at time 0, fr or is the orientation of the robot's right foot expressed as a quaternion, fr or0 is the direction of the robot's right foot at time 0, s 0 is the status information of the robot at time 0, s t is the status information of the robot at time t.

[0047] Optionally, in one embodiment of the present application, the calculation formula of the uniform random scaling may be but is not limited to:

[0048]

[0049] Among them, a is the lower limit of scaling, b is the upper limit of scaling, and U i is a uniform distribution on the interval [a,b];

[0050] The calculation formula of the Gaussian random scaling may be, but is not limited to,:

[0051]

[0052] Among them, a is the scaled mean, b is the scaled variance, N i is a Gaussian distribution with mean a and variance b;

[0053] The calculation formula of the secondary random scaling may be, but is not limited to,:

[0054]

[0055] Among them, a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with a lower bound and b upper bound.

[0056] Optionally, in one embodiment of the present application, the training module includes: a second acquisition unit, used to acquire a parameter set of the initial control strategy neural network, and determine the objective function of the parameter set based on the parameter set; a generation unit, used to generate an update rule for updating the parameter set based on the objective function; a training unit, used to combine the dense reward constructed based on dynamic potential energy and the training data generated during the control training process, and train the initial control strategy neural network through the update rule.

[0057] Optionally, in one embodiment of the present application, the expression of the objective function may be, but is not limited to:

[0058]

[0059] Among them, P(M n ,θ D ) is the parameter set θ D The initial control strategy of the neural network when controlling the robot is the state information sequence M n The probability of occurrence, π θ (a t |s t ) is the parameter set θ D The initial control strategy of the neural network is s t When the robot control instruction a is output t The probability of R(M n ) is M n The sum of dynamic intensive rewards;

[0060] The expression of the update rule may be, but is not limited to:

[0061]

[0062] Among them, α is the step size, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

[0063] The fourth aspect of the present application provides a robot control reinforcement learning device based on dynamic potential energy rewards, which is applied to the model application stage, wherein the device includes: a second acquisition module, used to obtain actual robot information and actual initial state information of the target robot; an input module, used to input the actual robot information and the actual initial state information into a pre-trained final control strategy neural network to obtain the control instructions of the target robot, wherein the pre-trained final control strategy neural network is obtained by intensive reward training based on dynamic potential energy construction.

[0064] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot control reinforcement learning method based on dynamic potential energy reward as described in the above embodiment.

[0065] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned robot control reinforcement learning method based on dynamic potential energy reward.

[0066] The seventh aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned robot control reinforcement learning method based on dynamic potential energy reward.

[0067] The embodiment of the present application can perform at least one round of robot control interaction on the robot according to the robot information and initial state information of the robot, and in each round of control interaction, calculate the robot's dense reward value constructed based on dynamic potential energy, and then train the robot's initial control strategy neural network to obtain the final control strategy neural network, and dynamically adjust the reward value obtained by the robot's current action control by the dense reward value constructed based on dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's movement posture is more natural and stable. Therefore, the problem in the related technology that the potential energy function is fixed, resulting in the reward obtained by each state in the entire training process being determined, limiting the strategy exploration, and being unable to perform complex control on the robot is solved.

[0068] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0070] Figure 1 A flowchart of a robot control reinforcement learning method based on dynamic potential energy reward provided according to an embodiment of the present application;

[0071] Figure 2 A schematic block diagram of size parameters of a Bolt robot provided according to an embodiment of the present application;

[0072] Figure 3A block diagram showing the intuitive functions of PBRS and DPBRS (Dynamic Potential-Based Reward Shaping) provided according to an embodiment of the present application;

[0073] Figure 4 A schematic block diagram of an evaluation reward change curve when different reward functions are used according to an embodiment of the present application;

[0074] Figure 5 A block diagram of a robot control reinforcement learning device based on dynamic potential energy rewards according to an embodiment of the present application;

[0075] Figure 6 A flowchart of another robot control reinforcement learning method based on dynamic potential energy reward provided according to an embodiment of the present application;

[0076] Figure 7 A schematic block diagram of a speed tracking test result provided according to an embodiment of the present application;

[0077] Figure 8 A schematic block diagram of a center of gravity yaw test result provided according to an embodiment of the present application;

[0078] Fig. 9 A schematic block diagram of a centroid height test result provided according to an embodiment of the present application;

[0079] Fig.10 A schematic block diagram of a robot gait when a PBRS reward function is used according to an embodiment of the present application;

[0080] Fig.11 A schematic block diagram of a robot gait when the DPBRS_U reward function is used according to an embodiment of the present application;

[0081] Fig.12 A block diagram of a robot control reinforcement learning device based on dynamic potential energy rewards according to another embodiment of the present application;

[0082] Fig.13 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0083] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0084] The following describes the robot control reinforcement learning method and device based on dynamic potential energy reward of the embodiment of the present application with reference to the accompanying drawings. In view of the problem mentioned in the above background technology that the potential energy function is fixed, the reward obtained by each state in the whole training process is determined, which limits the strategy exploration and cannot perform complex control on the robot, the present application provides a robot control reinforcement learning method based on dynamic potential energy reward, in which the robot can be subjected to at least one round of robot control interaction according to the robot information and initial state information of the robot, and in each round of control interaction, the robot's intensive reward value constructed based on dynamic potential energy is calculated, and then the robot's initial control strategy neural network is trained to obtain the final control strategy neural network, and the reward value obtained by the robot's current action control is dynamically adjusted by the intensive reward value constructed based on dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's motion posture is more natural and stable. Thus, the problem that the potential energy function is fixed in the related technology, the reward obtained by each state in the whole training process is determined, which limits the strategy exploration and cannot perform complex control on the robot is solved.

[0085] Specifically, Figure 1 The present invention provides a flowchart of a robot control reinforcement learning method based on dynamic potential energy reward according to an embodiment of the present application.

[0086] like Figure 1 As shown, the robot control reinforcement learning method based on dynamic potential energy reward is applied to the model training stage, wherein the method includes the following steps:

[0087] In step S101 , robot information and initial state information of the robot are acquired.

[0088] It is understandable that the robot may be the ODRI (Open Dynamics Robotics Initiative) open source point-legged bipedal robot Bolt, or other robots may be used. The specific configuration may be made by technicians in this field according to actual conditions, and this application does not impose any specific restrictions.

[0089] Furthermore, in an embodiment of the present application, the state information may include, but is not limited to, information consisting of 48 dimensions of information, including posture information, auxiliary information, and external instructions. The present application does not impose specific restrictions. The posture information may include, but is not limited to, the Euler angle of the base body toward the center of gravity (3 dimensions), the speed of the center of gravity (3 dimensions) and the acceleration (3 dimensions), which reflects the stability of the Bolt's walking and the stability of the posture, the angles (6 dimensions) and angular velocities (6 dimensions) of the 6 joints. The intelligent agent can use the joint information to learn the relationship between the posture and the control torque of each joint, the joint angles (12 dimensions) of the past two time steps and the joint angular velocity information (6 dimensions) of the past one time step, which are used to improve the intelligent agent's The present application does not impose any specific restrictions on the speed instructions and the ability to perceive external changes, etc.; the auxiliary information may include but is not limited to the relative displacement of the two feet and the center of gravity (6 dimensions). The robot should have different gaits under different speed instructions. Using this information, the robot can perceive the current action state, etc., which are not specifically limited in the present application; the external instruction information may include but is not limited to randomly extracting a speed instruction (1 dimension) from the instruction speed set v0 = {0.3, 0.4, 0.5, 0.6, 0.7} as the target speed of the robot's walking, and the yaw angle instruction (1 dimension) as the target direction of the robot's walking, etc. before the start of each round of iterative learning process, which are not specifically limited in the present application.

[0090] As a possible implementation method, the embodiment of the present application can obtain the robot information and initial state information of the robot.

[0091] For example, Figure 2 As shown, the embodiment of the present application can adopt the ODRI open source point-footed bipedal robot Bolt. The Bolt control strategy learned by the reinforcement learning algorithm in the physical simulation environment can be transferred to the physical robot with only fine-tuning. The structure of Bolt makes it lightweight and flexible, but the point-footed sole increases the difficulty of control, making it difficult to maintain a straight line and balance when walking, so it is difficult to control it using traditional control methods. In the experiment, physical simulation was achieved by importing the Bolt model in the xml file format into the Mujoco physical simulation engine. The reinforcement learning algorithm was written in Python, and the robot's status information was obtained and control instructions were sent through the Python programming interface provided by the Mujoco physical simulation engine.

[0092] Furthermore, in the embodiment of the present application, the Bolt robot has 6 degrees of freedom, including left and right hip pitch, roll and left and right knee pitch. The angle ranges of left and right hip pitch, roll and left and right knee pitch are [-10°, 10°], [-30°, 15°] and [-5°, 45°] respectively. The action space of the deep reinforcement learning algorithm for controlling the Bolt robot is 6-dimensional, corresponding to the control torque of the 6 degrees of freedom of the Bolt robot, and the range is [-1N·m, 1N·m]. The Actor neural network of the deep reinforcement learning algorithm directly outputs a 6-dimensional control torque. The experimental goal is to control the Bolt to achieve 5 different walking speeds. The state space consists of 48 dimensions of information, including posture information, auxiliary information and external instructions.

[0093] Optionally, in one embodiment of the present application, before calculating the dense reward value of the robot constructed based on dynamic potential energy, it also includes: calculating the potential energy of the robot according to the current state information of the robot; determining the weight parameter of the potential energy by uniform random scaling based on the potential energy; or determining the weight parameter of the potential energy by Gaussian random scaling based on the potential energy; or determining the weight parameter of the potential energy by quadratic random scaling based on the potential energy; constructing the dynamic potential energy of the robot based on the potential energy and the weight parameter; constructing a dense reward function based on the dynamic potential energy based on the dynamic potential energy. Among them, the dense reward function based on the dynamic potential energy can be, but is not limited to:

[0094] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t <T n-1 ),

[0095] Among them, F D (s t ,a t ,s t+1 ) is a dense reward based on dynamic potential, s t is the robot status information at time t, a t For s t The control instructions issued to the robot, s t+1 For the robot to respond to a t The new state information after, γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r at time t t .

[0096] The calculation formula of dynamic potential energy can be but is not limited to:

[0097]

[0098] Among them, w i (t)(0≤i≤3) is the dynamic weight parameter, The initial weight parameter can be initialized according to the robot control target, P ctrl is the energy penalty, P contact Foot contact force penalty, P Δy It is a yaw penalty.

[0099] The calculation formula of potential energy can be but is not limited to:

[0100]

[0101] in, To encourage the robot to move at the command speed v 0 walk, To encourage the agent to move in the direction of the instruction θ 0 walk, For survival rewards, The reward for the paw direction encourages the robot to keep its paw flat and not walk on tiptoes. 0 , σ 1 , σ 2 is the adjustment parameter, K is a positive constant used to reward or punish the robot if it falls, h is the height of the robot's center of gravity, and h 0 is the height of the robot's center of gravity at time 0, ν x is the speed of the robot's center of gravity, ν 0 is the velocity of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot trunk and the vertical direction, θ 0 is the angle between the robot’s trunk and the vertical direction at time 0, fl or is the orientation of the robot's left foot expressed in quaternion, fl or0 is the direction of the robot's left foot at time 0, fr or is the orientation of the robot's right foot expressed as a quaternion, fr or0 is the direction of the robot's right foot at time 0, s 0 is the status information of the robot at time 0, s t is the status information of the robot at time t.

[0102] The calculation formula of uniform random scaling can be but is not limited to:

[0103]

[0104] Among them, a is the lower limit of scaling, b is the upper limit of scaling, and U i is a uniform distribution on the interval [a,b];

[0105] The calculation formula of Gaussian random scaling can be but is not limited to:

[0106]

[0107] Among them, a is the scaled mean, b is the scaled variance, N i is a Gaussian distribution with mean a and variance b;

[0108] The calculation formula of the secondary random scaling can be but is not limited to:

[0109]

[0110] Among them, a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with a lower bound and b upper bound.

[0111] In some embodiments, the potential energy-based reward function of the present application embodiment may be, but is not limited to:

[0112]

[0113] Among them, Φ(s t ) is a potential-based reward function defined in the state space S; i is the number of reward items, ω i is the weight of the i-th reward, Designed according to different objectives of reinforcement learning, the initial state s 0 Calculate the current state s t A function of the potential energy.

[0114] Then, the embodiment of the present application can calculate the entire M n The total amount of dense rewards R(M n ), the calculation formula can be but not limited to:

[0115]

[0116] Furthermore, the embodiment of the present application can calculate the PBRS intensive reward, and its expression can be but not limited to:

[0117] F(s t ,a t ,s t+1 )=γΦ(s t+1 )-Φ(s t )(0≤t <T n -1), (3)

[0118] Among them, if R Φ (M n ) is the entire M n The total PBRS reward is,

[0119]

[0120] From formulas (1)-(4), we can see that R Φ (M n ) is equivalent to R(M n ) minus the potential energy reward of the initial state. Therefore, the potential energy-based reward function shown in formula (1) can be used to achieve potential energy-based reward shaping. Under the premise of ensuring that reinforcement learning can converge to the optimal strategy, the various rewards in formula (1) are used to guide the reinforcement learning process, achieve the learning goal, and improve learning efficiency.

[0121] Furthermore, in order to control the bipedal robot to move forward at a constant speed and make the walking posture more stable and natural, the embodiments of the present application can respectively design the rewards in formula (4) according to (5)-(8) to form a dense reward function constructed based on the potential energy difference between the current state and the expected state, and its expression can be but is not limited to:

[0122]

[0123] in, To encourage the robot to move at the command speed v 0 Walking, x is the current moving speed of the robot's center of gravity, σ 0 To adjust the parameters; Encourage the agent to move in the direction of the instruction θ 0 Walking, θ z is the current yaw angle, σ 1 To adjust the parameters; For the survival reward, H(s) determines whether the robot falls according to the current state s. If the robot does not fall, it can get a positive reward. Otherwise, it will give a large negative reward. This reward encourages the agent to keep balance as much as possible and walk further. The reward for the paw facing direction encourages the robot to keep its paws flat and not walk on tiptoes. or is the robot foot orientation expressed as a quaternion, fl or0 is the initial foot orientation.

[0124] In addition, in order to improve the efficiency of reinforcement learning, the embodiment of the present application designs a three-part penalty reward function, the size of which is a negative value and is mainly composed of three parts, with a weight of p 0 、p 1 and p 2 The optimal strategy learned by the agent should make all three items approach 0, and their expressions are shown in (9)-(11).

[0125]

[0126] P Δy=p 2 |yy 0 |, (11)

[0127] Among them, P ctrl is the energy penalty, a i is the torque used by the i-th motor. This penalty term prevents violent motion to ensure the stability of the system. contact is the foot contact force penalty, which encourages the agent to lift its feet as high as possible and helps exploration in the early stages of the experiment. is the force on the nth foot in the i direction; P Δy is the yaw penalty, y 0 is the initial heading, y is the current heading. The agent usually deviates from the specified direction when walking, so this penalty is given to reduce the deviation.

[0128] In addition, it should be noted that in order to avoid useless exploration by the intelligent agent, when the absolute values ​​of the pitch angle, roll angle and heading angle are greater than β 0 When the robot’s posture is unnatural, the current round of iterative learning is terminated in advance to avoid invalid learning, and w is given 4 The final reward based on the potential energy reward function is shown in formula (12).

[0129] As the agent achieves its goal, the reward it receives at each step approaches 1.0.

[0130]

[0131] Furthermore, the embodiment of the present application is based on the reward function of dynamic potential energy, combining dynamic potential energy with PBRS to form DPBRS, denoted as Φ D (s t ), which may be, but is not limited to:

[0132]

[0133] DPBRS dense reward, i.e., a dense reward function based on dynamic potential energy, can be but is not limited to:

[0134] F D (s t ,α t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t <T n-1 ), (14)

[0135] Compared with the potential energy function Φ(s t) compared to the dynamic potential energy function Φ of the embodiment of the present application D (s t ) allows all parts of the potential energy function to change with t, that is, for any state s (s∈S), when it appears in different positions in the same MDP (Markov Decision Process) or in different MDPs, the dense reward value obtained by the agent is dynamically variable, which makes the total dense reward of the entire MDP also dynamically variable. Similar to the potential energy-based reward shaping, the dynamic potential energy-based reward function shown in formula (13) can be used to achieve dynamic potential energy-based reward shaping. When w i When (t) is a constant value, DPBRS degenerates into PBRS. The intuitive action flow chart of PBRS and DPBRS is as follows: Figure 3 shown.

[0136] Further, in the embodiment of the present application, w i The dynamics of (t) can be determined by i Uniform random scaling, Gaussian random scaling, or quadratic random scaling is achieved by:

[0137] In the embodiment of the present application, uniform random scaling can be understood as determining the weight scaling value by uniform random sampling, as shown in formula (15), which can be but is not limited to:

[0138]

[0139] Gaussian random scaling can be understood as determining the weight scaling value by Gaussian random sampling, as shown in formula (16), which can be but is not limited to:

[0140]

[0141] The secondary random scaling can be understood as determining the weight scaling value by parabolic random sampling, as shown in formula (17), which can be but is not limited to:

[0142]

[0143] For example, the parameters of the reward function in the embodiment of the present application are shown in Table 1, where a i and b i , (0≤i≤3), respectively The random scaling lower and upper limits of the algorithm are shown in Table 1. The embodiment of the present application can directly use PBRS, DPBRS_U, DPBRS_N and DPBRS_P to represent the potential-based reward shaping, the dense reward function based on uniform random scaling dynamic potential, the dense reward function based on Gaussian random scaling dynamic potential, and the dense reward function based on quadratic random scaling dynamic potential. Table 1 is a reward function parameter setting table provided according to an embodiment of the present application.

[0144] Table 1

[0145] Parameter name value Parameter name value Parameter name value w0 7.5 σ2 1.0 a3 0.8 w1 2.0 p0 0.1 b0 1.37 w2 1.0 p1 5e-7 b1 1.6 w3 0.5 p2 0.3 b2 1.2 w4 -5.0 a0 0.67 b3 1.2 σ0 0.5 a1 0.4 σ1 0.667 a2 0.8

[0146] Furthermore, the embodiment of the present application can obtain reinforcement learning training results of different reward functions, wherein the total number of sampling times of the embodiment of the present application is 1.6 million times, and the comparative test results of PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), TRPO (Trust Region Policy Optimization) and A2C reinforcement learning algorithms based on PBRS and DPBRS_U are shown in Table 2, wherein Table 2 is a cumulative reward comparison table of common reinforcement learning algorithms based on PBRS and DPBRS_U provided according to an embodiment of the present application.

[0147] Table 2

[0148] PBRS DPBRS_U PPO 1831.8 1843.3 SAC 1759.9 1768.0 TRPO 1785.8 1720.4 A2C 456.0 906.5

[0149] As can be seen from Table 2, the embodiment of the present application DPBRS_U can obtain a higher cumulative reward than PBRS, and can achieve certain performance improvements for SAC, PPO, and A2C algorithms, but the best performing algorithm is the PPO algorithm, so the subsequent test experiments all use the PPO algorithm. The PPO algorithm was used for the test experiment, and each 2048 interactions with the environment was a training round, and then all samples of the round were used to train the neural network to update the strategy, and then the fixed reward was calculated according to the formula (15) in the manual for performance evaluation. The test shows that when the robot's average reward per time step is above 0.9, the goal of walking at the command speed can be achieved. Therefore, the cumulative reward of 1800 at the maximum time of 2000 is used as a benchmark to judge whether the robot has achieved the experimental goal. The parameter settings of the PPO algorithm are shown in Table 3. Among them, Table 3 is a PPO algorithm parameter setting table provided according to an embodiment of the present application.

[0150] Table 3

[0151] Parameter name value Value network hidden layer [256,128,32] Actor Network Hidden Layer [256,128,32] Discount factor γ 0.99 Learning rate lr 1e-4 Truncation∈ 0.1 Sample n_steps per round 2048 Batch size mini_batch 64 Number of updates per round n_epoch 10

[0152] Since the agent needs to learn to control the robot to walk at a variety of command speeds, the reward value will fluctuate greatly during training, so the average reward of 10 rounds of iterative learning is used as the evaluation reward for comparative testing. Figure 4 The figure shows the comparative test results of evaluating reward changes in 650 training rounds of the PPO algorithm using PBRS, DPBRS_U, DPBRS_N, and DPBRS_P reward functions respectively.

[0153] from Figure 4 It can be seen that the PPO algorithm using three DPBRS reward functions in the embodiment of the present application has a faster convergence speed than the PPO algorithm using the PBRS reward function, and can reach the goal (1800 cumulative reward value) faster. Among them, the performance is best when the DPBRS_U reward function is used, and the maximum cumulative reward of 1801 is obtained in 538 training rounds; the maximum cumulative reward of 1754 is obtained in 632 training rounds using DPBRS_N; the training process is the most stable when the DPBRS_P reward function is used, and the curve fluctuations are relatively small, and the maximum cumulative reward of 1793 is obtained in 645 training rounds; and the curve oscillation amplitude is the largest when the PBRS reward function is used, and only the maximum cumulative reward of 1655 is obtained in 554 training rounds, and the training effect is not as good as the first three. The test experimental results show that the use of the DPBRS reward function can improve the exploration performance and accelerate the training speed.

[0154] In step S102, based on the robot information and the initial state information, at least one round of robot control interaction is performed on the robot. In each round of control interaction, based on the robot's control instructions and state information, a dense reward value of the robot constructed based on dynamic potential energy is calculated.

[0155] As a possible implementation method, the embodiment of the present application can perform at least one round, such as N (N≥1) rounds of robot control interaction. In each round of control interaction, the initial state information of the robot is first obtained, and then control instructions are sent, new state information is received, and a dense reward value constructed based on dynamic potential energy is calculated. The interaction continues until the robot control interaction is completed.

[0156] Optionally, in one embodiment of the present application, performing at least one round of robot control interaction on the robot includes: obtaining state information at each moment of the nth round of robot control interaction; determining the termination time T of the nth round of robot control interaction termination; n ; Based on the termination time T n The state information sequence M generated by the robot control interaction is obtained by combining the state information at each moment in the nth round n , where n = 1, 2, 3, ..., N, M n 0 to T in the nth round of robot control interaction nThe sequence of all state information at the moment, N is the total number of training rounds; based on the state information sequence M n Get N rounds of robot control training to generate all round state information sequences {M 1 ,M 2 ,…,M n ,…,M N}.

[0157] In some embodiments, the embodiment of the present application may first establish a complete MDP. It can be understood that, in the embodiment of the present application, the reinforcement learning problem is usually modeled as an MDP, and its calculation formula can be, but is not limited to, expressed as:

[0158] M=<S,A,P,R> , (18)

[0159] Among them, S is the set of observed states of the environment, A is the set of actions that the agent can take, P is the probability of environment transition, and R is the reward function.

[0160] Furthermore, the reinforcement learning of the embodiment of the present application can select an Action neural network through multiple rounds of iterative training actions, and the parameter set of the Action neural network is denoted as θ, and its input is the state information s (s∈S), and the output is the action a (a∈A) taken under the current state information s. The action selection strategy described by the Action neural network is denoted as π(a|s), that is, how to select action a (a=π according to the state information s θ (a|s). In the nth (n≥0) round of robot control interaction, the agent uses the current strategy π under the initial state information s0. θ (a|s) performs action a0, the environment gives an immediate reward r0, and the state information is transferred to s1 with probability P, completing the first control interaction (also called sampling at time step t=0); the agent responds to the new state information s1 according to the current strategy π θ (a|s) executes action a1, the environment gives an immediate reward r1, and the state information is transferred to s2 with probability P, completing the second control interaction (also called sampling at time step t=1); the above control interaction process is repeated until the termination time T n (the moment when the robot falls or the time step exceeds the given maximum value), then this round of iterative learning ends, forming the MDP of the nth round of iterative learning, denoted as M n ,

[0161]

[0162] Among them, T n is the length of the MDP, the maximum time step t max =T n , the goal of reinforcement learning is to learn the optimal strategy Maximize the cumulative reward of the trajectory.

[0163]

[0164] Among them, γ is the discount factor.

[0165] In step S103, the initial control strategy neural network of the robot is trained by combining the dense reward value constructed based on the dynamic potential energy and the training data generated by the control interaction process to obtain the final control strategy neural network.

[0166] As a possible implementation method, the embodiment of the present application can use the MLP neural network to establish the initial control strategy neural network π θ , initial value network V φ Furthermore, the embodiment of the present application can combine the dense reward value constructed based on dynamic potential energy and the training data generated by the control interaction process to obtain N data {s t ,a t ,r t}, where r t Calculated by DPBRS rewards.

[0167] Furthermore, the advantage function is calculated in the embodiment of the present application The following formula is used to initialize the control strategy neural network parameter θ for a total of M times, and its expression can be but is not limited to:

[0168]

[0169] Among them, KL[π old |π θ ] is limited to a range of [1 / α,α], with 1 / α being taken below the lower limit and α being taken above the upper limit.

[0170] Use the following formula to update the initial value network parameters for a total of B times, and the expression can be but is not limited to:

[0171]

[0172] Then the final control strategy neural network is obtained.

[0173] Optionally, in one embodiment of the present application, the initial control strategy neural network of the robot is trained by combining the dense reward value constructed based on the dynamic potential energy and the training data generated in the control interaction process, including: obtaining the parameter set of the initial control strategy neural network, and determining the objective function of the parameter set based on the parameter set; generating an update rule for updating the parameter set based on the objective function; combining the dense reward constructed based on the dynamic potential energy and the training data generated in the control training process, and training the initial control strategy neural network through the update rule. Wherein, the expression of the objective function can be but is not limited to:

[0174]

[0175] Among them, P(M n ,θ D ) is the parameter set θ D The initial control strategy of the neural network when controlling the robot is the state information sequence M n The probability of occurrence, π θ (a t |s t ) is the parameter set θ D The initial control strategy of the neural network is s t When the robot control instruction a is output t The probability of R(M n ) is M n The sum of dynamic intensive rewards;

[0176] The expression of the update rule can be, but is not limited to:

[0177]

[0178] Among them, α is the step size, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

[0179] In some embodiments, the present application embodiment makes M n is the nth (n≥0) round of iterative learning process, then the goal of reinforcement learning is to adjust the parameter set θ of the action selection neural network Action through multiple rounds of iterative learning, so that M with higher reward value n The probability of occurrence is higher. Therefore, the objective function J(θ) of the reinforcement learning policy gradient can be but is not limited to:

[0180]

[0181] Among them, P(M n ,θ) represents the trajectory M n The probability of occurrence under the current behavior selection strategy can be calculated by the formula but is not limited to:

[0182]

[0183] Among them, P(s t+1 |s t ,a t ) is the state transition probability determined by the environment. The derivative of the objective function J(θ) can be, but is not limited to:

[0184]

[0185] It can be seen that the embodiments of the present application are difficult to realize all possible Mn In reinforcement learning, the expectation of multiple trajectory samples is used to approximate the sum. Therefore, the policy gradient update rule can be but is not limited to:

[0186]

[0187] Among them, α is the step size.

[0188] Furthermore, in the embodiment of the present application, when the DPBRS reward function is used, the parameter set of the Action neural network is denoted as θ D , the objective function is recorded as J(θ D ), then:

[0189]

[0190] At this time, in the embodiment of the present application, the corresponding policy gradient update rule may be, but is not limited to:

[0191]

[0192] Among them, δ is the expected deviation of the dynamic weight relative to the standard weight, which is a random value obtained after sampling.

[0193] Therefore, due to the existence of dynamic potential energy, in different rounds of iterative learning, even if the same state s (s∈S) appears, the calculated dense reward value is different and the direction of strategy update is also different, making the learning process more exploratory.

[0194] According to the robot control reinforcement learning method based on dynamic potential energy reward proposed in the embodiment of the present application, at least one round of robot control interaction can be performed on the robot according to the robot information and initial state information of the robot, and in each round of control interaction, the robot's dense reward value constructed based on dynamic potential energy is calculated, and then the robot's initial control strategy neural network is trained to obtain the final control strategy neural network, and the reward value obtained by the robot's current action control is dynamically adjusted by the dense reward value constructed based on dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's movement posture is more natural and stable. Therefore, the problem in the related technology that the potential energy function is fixed, resulting in the reward obtained for each state in the entire training process being determined, limiting the strategy exploration, and being unable to perform complex control of the robot is solved.

[0195] Next, a robot control reinforcement learning device based on dynamic potential energy rewards proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0196] Figure 5 A block diagram of a robot control reinforcement learning device based on dynamic potential energy rewards provided according to an embodiment of the present application.

[0197] like Figure 5 As shown, the robot control reinforcement learning device 50 based on dynamic potential energy reward is applied to the model training stage, wherein the device 50 includes: a first acquisition module 501, a first calculation module 502 and a training module 503.

[0198] The first acquisition module 501 is used to acquire the robot information and initial state information of the robot.

[0199] The first calculation module 502 is used to perform at least one round of robot control interaction on the robot based on the robot information and the initial state information. In each round of control interaction, based on the control instructions and state information of the robot, a dense reward value of the robot constructed based on dynamic potential energy is calculated.

[0200] The training module 503 is used to train the initial control strategy neural network of the robot by combining the dense reward value constructed based on the dynamic potential energy and the training data generated by the control interaction process to obtain the final control strategy neural network.

[0201] Optionally, in one embodiment of the present application, the first calculation module 502 includes: a first acquisition unit, a determination unit and a second generation unit.

[0202] The first acquisition unit is used to acquire the state information of each moment in the nth round during the robot control interaction process.

[0203] A determination unit is used to determine the termination time T of the nth round of robot control interaction termination n ; A first generating unit, for based on the termination time T n The state information sequence M generated by the robot control interaction is obtained by combining the state information at each moment in the nth round n , where n = 1, 2, 3, ..., N, M n 0 to T in the nth round of robot control interaction n The sequence of all state information at the moment, N is the total number of training rounds.

[0204] The second generating unit is used to generate a state information sequence M based on the state information sequence M n Get N rounds of robot control training to generate all round state information sequences {M 1 ,M 2 ,…,M n ,…,M N}.

[0205] Optionally, in one embodiment of the present application, it further includes: a second calculation module, a first determination module, a second determination module, or a third determination module, a first construction module and a second construction module.

[0206] The second calculation module is used to calculate the potential energy of the robot according to the current state information of the robot before calculating the dense reward value of the robot constructed based on the dynamic potential energy.

[0207] The first determination module is used to determine a weight parameter of the potential energy by uniform random scaling based on the potential energy.

[0208] Alternatively, the second determination module is used to determine the weight parameter of the potential energy through Gaussian random scaling based on the potential energy.

[0209] Alternatively, the third determination module is used to determine the weight parameter of the potential energy through secondary random scaling based on the potential energy.

[0210] The first building module is used to build the dynamic potential energy of the robot based on the potential energy and the weight parameter.

[0211] The second building block is used to construct a dense reward function based on dynamic potential energy.

[0212] Optionally, in one embodiment of the present application, the intensive reward function based on dynamic potential energy may be, but is not limited to:

[0213] F D (s t ,a t ,s t+1 )=γΦ D (s t+1 )-Φ D (s t )(0≤t <T n-1 ),

[0214] Among them, F D (s t ,a t ,s t+1 ) is a dense reward based on dynamic potential, s t is the robot status information at time t, a t For s t The control instructions issued to the robot, s t+1 For the robot to respond to a t The new state information after, γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r at time t t .

[0215] Optionally, in one embodiment of the present application, the calculation formula of dynamic potential energy may be, but is not limited to,:

[0216]

[0217] Among them, w i(t)(0≤i≤3) is the dynamic weight parameter, The initial weight parameter can be initialized according to the robot control target, P ctrl is the energy penalty, P contact Foot contact force penalty, P Δy It is a yaw penalty.

[0218] Optionally, in one embodiment of the present application, the calculation formula of potential energy may be, but is not limited to,:

[0219]

[0220] in, To encourage the robot to move at the command speed v 0 walk, To encourage the agent to move in the direction of the instruction θ 0 walk, For survival rewards, The reward for the paw direction encourages the robot to keep its paw flat and not walk on tiptoes. 0 , σ 1 , σ 2 is the adjustment parameter, K is a positive constant used to reward or punish the robot if it falls, h is the height of the robot's center of gravity, and h 0 is the height of the robot's center of gravity at time 0, ν x is the speed of the robot's center of gravity, ν 0 is the velocity of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot trunk and the vertical direction, θ 0 is the angle between the robot’s trunk and the vertical direction at time 0, fl or is the orientation of the robot's left foot expressed in quaternion, fl or0 is the direction of the robot's left foot at time 0, fr or is the orientation of the robot's right foot expressed as a quaternion, fr or0 is the direction of the robot's right foot at time 0, s 0 is the status information of the robot at time 0, s t is the status information of the robot at time t.

[0221] Optionally, in one embodiment of the present application, the calculation formula for uniform random scaling may be, but is not limited to,:

[0222]

[0223] Among them, a is the lower limit of scaling, b is the upper limit of scaling, and U i is a uniform distribution on the interval [a,b];

[0224] The calculation formula of Gaussian random scaling can be but is not limited to:

[0225]

[0226] Among them, a is the scaled mean, b is the scaled variance, N i is a Gaussian distribution with mean a and variance b;

[0227] The calculation formula of the secondary random scaling can be but is not limited to:

[0228]

[0229] Among them, a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with a lower bound and b upper bound.

[0230] Optionally, in one embodiment of the present application, the training module 300 includes: a second acquisition unit, a generation unit and a training unit.

[0231] The second acquisition unit is used to acquire a parameter set of the initial control strategy neural network and determine the objective function of the parameter set based on the parameter set.

[0232] A generation unit is used to generate an update rule for updating a parameter set based on an objective function.

[0233] The training unit is used to combine the dense rewards constructed based on dynamic potential energy and the training data generated during the control training process to train the initial control policy neural network through the update rule.

[0234] Optionally, in one embodiment of the present application, the expression of the objective function may be, but is not limited to:

[0235]

[0236] Among them, P(M n ,θ D ) is the parameter set θ D The initial control strategy of the neural network when controlling the robot is the state information sequence M n The probability of occurrence, π θ (a t |s t ) is the parameter set θ D The initial control strategy of the neural network is s t When the robot control instruction a is output t The probability of R(M n ) is M n The sum of dynamic intensive rewards;

[0237] The expression of the update rule can be, but is not limited to:

[0238]

[0239] Among them, α is the step size, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

[0240] It should be noted that the aforementioned explanation of the embodiment of the robot control reinforcement learning method based on dynamic potential energy reward is also applicable to the robot control reinforcement learning device based on dynamic potential energy reward in this embodiment, and will not be repeated here.

[0241] According to the robot control reinforcement learning device based on dynamic potential energy rewards proposed in the embodiment of the present application, at least one round of robot control interaction can be performed on the robot according to the robot information and initial state information of the robot, and in each round of control interaction, the robot's dense reward value constructed based on dynamic potential energy is calculated, and then the robot's initial control strategy neural network is trained to obtain the final control strategy neural network, and the reward value obtained by the robot's current action control is dynamically adjusted by the dense reward value constructed based on dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's movement posture is more natural and stable. Therefore, the problem in the related technology that the potential energy function is fixed, resulting in the reward obtained for each state in the entire training process being determined, limiting the strategy exploration, and being unable to perform complex control of the robot is solved.

[0242] The above embodiment describes the model training phase, and the following describes an embodiment of the model application phase.

[0243] Figure 6 This is a flowchart of another robot control reinforcement learning method based on dynamic potential energy rewards provided according to an embodiment of the present application.

[0244] like Figure 6 As shown, the robot control reinforcement learning method based on dynamic potential energy reward is applied to the model application stage, wherein the method includes the following steps:

[0245] In step S601, actual robot information and actual initial state information of the target robot are acquired.

[0246] In step S602, the actual robot information and the actual initial state information are input into the pre-trained final control strategy neural network to obtain the control instructions of the target robot, wherein the pre-trained final control strategy neural network is obtained by intensive reward training based on dynamic potential energy construction.

[0247] In the actual execution process, the embodiment of the present application can be used to tInput into the pre-trained final control strategy neural network and output the control action a t , the robot changes its status information to s t+1 , repeating continuously, the continuous walking control of the target robot can be achieved.

[0248] For example, the embodiments of the present application can analyze the robot walking test results of different reward functions by building an environment.

[0249] The test experiment conducted comparative tests on the accuracy of command speed tracking, walking straightness and stability. After 10 rounds of testing, the average reward was 1831.8 when the PBRS reward function was used, and the average reward was 1843.3 when the DPBRS_U reward function was used.

[0250] The test results of the instruction speed tracking accuracy in the embodiment of the present application are analyzed as follows:

[0251] The average speed and standard deviation of the embodiment of the present application under 5 instruction speeds when using the DPBRS_U and PBRS reward functions are shown in Table 4, and the maximum errors are 1.4% and 2.4% respectively. In addition, the instruction speed is gradually increased in the same test at intervals of 400 time steps to detect the accuracy of speed tracking when using the two reward functions respectively. The test results are shown in Table 4. Figure 7 As shown in Table 4 and Figure 7 It can be seen that when the DPBRS_U reward function is used in the embodiment of the present application, the command speed is executed more accurately, the walking speed is more stable, and the maximum speed fluctuation is about 8.5%; while when the PBRS reward function is used, the walking speed is not stable enough, and the maximum speed fluctuation is 13.8%. Among them, Table 4 is a table of the average walking speed and standard deviation of the robot when different reward functions are used according to an embodiment of the present application.

[0252] Table 4

[0253] v0 DPBRS_U Vx(m / s) PBRS Vx(m / s) 0.3 0.303±0.026 0.305±0.042 0.4 0.402±0.025 0.404±0.038 0.5 0.488±0.030 0.493±0.045 0.6 0.598±0.038 0.592±0.061 0.7 0.692±0.045 0.693±0.053

[0254] The analysis of the walking straightness test results in the present application example is as follows:

[0255] The walking straightness of the embodiment of the present application mainly tests the ability of the robot to walk in a specified direction. Under 5 command speeds, the average and standard deviation of the yaw angle when using the DPBRS_U and PBRS reward functions are shown in Table 5. It can be seen from Table 5 that when the DPBRS_U reward function is used, the maximum yaw angle is 0.035 and the minimum yaw angle is 0.006; when the PBRS reward function is used, the maximum yaw angle is 0.040 and the minimum yaw angle is -0.005. Too large or too small speed will lead to instability of the yaw angle. The yaw is smaller when the DPBRS_U reward function is used. Among them, Table 5 is a table of the average yaw angle and standard deviation of the robot walking when different reward functions are used according to an embodiment of the present application.

[0256] Table 5

[0257] v0 DPBRS_U x(rad) PBRS x(rad) 0.3 0.024±0.015 -0.034±0.018 0.4 0.026±0.017 -0.005±0.014 0.5 -0.006±0.016 0.02±0.015 0.6 -0.012±0.017 0.035±0.017 0.7 0.035±0.018 0.04±0.024

[0258] The robot's center of gravity yaw test results are as follows when the command speed is gradually increased: Figure 8 As shown. Figure 5 It can be seen that when the DPBRS_U reward function is used, the robot can maintain good walking straightness. The center of gravity offset reaches a maximum value of -0.0755m at 201 steps, and then gradually decreases to 0.070m at 1501 steps, and then gradually approaches 0. The center of gravity offset range is small and there is no large fluctuation. When the PBRS reward function is used, the center of gravity offset reaches a maximum value of -0.201m at 896 steps, and then gradually decreases to 0.035m at 1985 steps. The overall center of gravity offset range is larger and the fluctuation is greater.

[0259] The analysis of the walking stability test results in the present application example is as follows:

[0260] In the embodiment of the present application, when humans walk, the height of the center of gravity of the body will oscillate regularly to maintain walking stability and save energy. Therefore, the walking stability of the robot can be measured by testing the change of the center of gravity height when the robot walks. The initial center of gravity height of the robot is 0.47. When the command speed is gradually increased, the center of gravity height change curve of the robot when using the DPBRS_U and PBRS reward functions is as follows: Fig. 9 shown.

[0261] from Fig. 9It can be seen that when the DPBRS_U reward function is adopted in the embodiment of the present application, the height of the robot's center of gravity oscillates while the height remains basically stable. At five gradually increasing instruction speeds, the average heights of the robot's center of gravity are 0.476, 0.480, 0.481, 0.484, and 0.487, respectively. As the walking speed increases, the height of the robot's center of gravity increases slightly, and the robot walks with its legs fully extended. When the PBRS reward function is adopted, the height of the robot's center of gravity gradually decreases while oscillating. At five gradually increasing instruction speeds, the average heights of the robot's center of gravity are 0.461, 0.460, 0.455, 0.450, and 0.438, respectively. When the instruction speed increases to 0.7, the height of the center of gravity drops significantly, the robot's legs cannot be extended, and the walking stability is reduced.

[0262] Neither reward function explicitly guides the robot to walk in which gait, but the test results show that when the DPBRS_U reward function is used, the robot can learn a more natural and coordinated gait from the implicit information of the reward function. Fig.10 and Fig.11 These are the gait effect diagrams of the robot when using PBRS and DPBRS_U reward functions (the command speed is 0.4).

[0263] contrast Fig.10 and Fig.11 It can be seen that when the DPBRS_U reward function is used in the embodiment of the present application, the gait of the robot is closer to the human walking gait, while when the PBRS reward function is used, it is similar to a single-leg jump. This also shows that when the DPBRS_U reward function is used, the intelligent agent can be encouraged to explore more aspects in the reinforcement learning process, so that the robot can walk with a more natural and stable gait.

[0264] According to the robot control reinforcement learning method based on dynamic potential energy reward proposed in the embodiment of the present application, the actual robot information and the actual initial state information can be input into the pre-trained final control strategy neural network, and then the control instructions of the target robot can be obtained. The reward value obtained by the robot's current action control can be dynamically adjusted by the dense reward value constructed based on the dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's movement posture is more natural and stable. Therefore, the problem in the related technology that the potential energy function is fixed, resulting in the determination of the reward obtained for each state during the entire training process, limiting the strategy exploration, and making it impossible to perform complex control of the robot is solved.

[0265] Next, a robot control reinforcement learning device based on dynamic potential energy rewards proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0266] Fig.12A block diagram of a robot control reinforcement learning device based on dynamic potential energy rewards provided in another embodiment of the present application.

[0267] like Fig.12 As shown, the robot control reinforcement learning device 120 based on dynamic potential energy reward is applied to the model application stage, wherein the device 120 includes: a second acquisition module 1201 and an input module 1202.

[0268] The second acquisition module 1201 is used to acquire actual robot information and actual initial state information of the target robot.

[0269] The input module 1202 is used to input the actual robot information and the actual initial state information into the pre-trained final control strategy neural network to obtain the control instructions of the target robot, wherein the pre-trained final control strategy neural network is obtained by intensive reward training based on dynamic potential energy construction.

[0270] It should be noted that the aforementioned explanation of the embodiment of the robot control reinforcement learning method based on dynamic potential energy reward is also applicable to the robot control reinforcement learning device based on dynamic potential energy reward in this embodiment, and will not be repeated here.

[0271] According to the robot control reinforcement learning device based on dynamic potential energy rewards proposed in the embodiment of the present application, the actual robot information and the actual initial state information can be input into the pre-trained final control strategy neural network, and then the control instructions of the target robot can be obtained. The reward value obtained by the robot's current action control can be dynamically adjusted by the dense reward value constructed based on the dynamic potential energy, thereby improving the exploration ability of the learning process, effectively improving the training speed, and the robot's movement posture is more natural and stable. In this way, the problem in the related art that the potential energy function is fixed, resulting in the determination of the reward obtained for each state during the entire training process, limiting the strategy exploration, and making it impossible to perform complex control of the robot is solved.

[0272] Fig.13 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0273] A memory 1301 , a processor 1302 , and a computer program stored in the memory 1301 and executable on the processor 1302 .

[0274] When the processor 1302 executes the program, the robot control reinforcement learning method based on dynamic potential energy reward provided in the above embodiment is implemented.

[0275] Furthermore, the electronic device further comprises:

[0276] The communication interface 1303 is used for communication between the memory 1301 and the processor 1302 .

[0277] The memory 1301 is used to store computer programs that can be executed on the processor 1302 .

[0278] The memory 1301 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0279] If the memory 1301, the processor 1302 and the communication interface 1303 are implemented independently, the communication interface 1303, the memory 1301 and the processor 1302 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0280] Optionally, in a specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a chip, the memory 1301, the processor 1302 and the communication interface 1303 can communicate with each other through an internal interface.

[0281] The processor 1302 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0282] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned robot control reinforcement learning method based on dynamic potential energy reward.

[0283] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned robot control reinforcement learning method based on dynamic potential energy reward.

[0284] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0285] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0286] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0287] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0288] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0289] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0290] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0291] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A robot control reinforcement learning method based on dynamic potential energy reward, characterized in that: Applied to the model training stage, wherein the method comprises the following steps: Get the robot information and initial state information of the robot; Based on the robot information and the initial state information, performing at least one round of robot control interaction on the robot, and in each round of control interaction, calculating a intensive reward value of the robot constructed based on dynamic potential energy based on the control instructions and state information of the robot; The initial control strategy neural network of the robot is trained by combining the dense reward value constructed based on dynamic potential energy and the training data generated by the control interaction process to obtain a final control strategy neural network.

2. The method according to claim 1, characterized in that The performing at least one round of robot control interaction on the robot comprises: Obtaining state information at each moment of the nth round during the robot control interaction process; Determine the termination time T of the nth round of robot control interaction n ; Based on the termination time T n and the state information at each moment in the nth round to obtain the state information sequence M generated by the robot control interaction n , where n = 1, 2, 3, ..., N, M n 0 to T in the nth round of robot control interaction n The sequence of all state information at the moment, N is the total number of training rounds; Based on the state information sequence M n Get N rounds of robot control training to generate all round state information sequences {M1,M2,…,M n ,…,M N }.

3. The method according to claim 1, characterized in that Before calculating the dense reward value of the robot based on dynamic potential energy, it also includes: Calculating the potential energy of the robot according to the current state information of the robot; Based on the potential energy, determining a weight parameter of the potential energy by uniform random scaling; Alternatively, based on the potential energy, a weight parameter of the potential energy is determined by Gaussian random scaling; Alternatively, based on the potential energy, a weight parameter of the potential energy is determined by secondary random scaling; Constructing the dynamic potential energy of the robot based on the potential energy and the weight parameter; A dynamic potential based dense reward function is constructed based on the dynamic potential.

4. The method according to claim 3, characterized in that The dynamic potential based dense reward function is: F D (with t ,and t ,with t+1 )=γΦ D (with t+1 )-Φ D (with t )(0≤t <T n-1 ), Among them, F D (s t ,a t ,s t+1 ) is a dense reward based on dynamic potential, s t is the robot status information at time t, a t For s t Control instructions issued to the robot, s t+1 For the robot to respond to a t The new state information after, γ is the discount factor, Φ D is the dynamic potential energy, that is, the reward r at time t t .

5. The method according to claim 3, characterized in that: The calculation formula of the dynamic potential energy is: Among them, w i (t)(0≤i≤3) is the dynamic weight parameter, The initial weight parameter can be initialized according to the robot control target, P ctrl is the energy penalty, P contact Foot contact force penalty, P Δy It is a yaw penalty.

6. The method according to claim 3, characterized in that: The calculation formula of the potential energy is: in, To encourage the robot to walk at the command speed v0, To encourage the agent to walk in the command direction θ0, For survival rewards, is the reward for the sole of the foot facing the robot, which encourages the robot to put its soles flat and not walk on tiptoe. σ0, σ1, and σ2 are adjustment parameters. K is a positive constant used to reward or punish the robot if it falls. h is the height of the robot's center of gravity, h0 is the height of the robot's center of gravity at time 0, and ν x is the moving speed of the robot's center of gravity, ν0 is the moving speed of the robot's center of gravity at time 0, x is the walking direction, θ z is the angle between the robot trunk and the vertical direction, θ0 is the angle between the robot trunk and the vertical direction at time 0, and fl or is the orientation of the robot's left foot expressed in quaternion, fl or0 is the direction of the robot's left foot at time 0, fr or is the orientation of the robot's right foot expressed as a quaternion, fr or0 is the direction of the robot's right foot at time 0, s0 is the robot's state information at time 0, and s t is the status information of the robot at time t.

7. The method according to claim 3, characterized in that in, The calculation formula of the uniform random scaling is: Among them, a is the lower limit of scaling, b is the upper limit of scaling, and U i is a uniform distribution on the interval [a,b]; The calculation formula of the Gaussian random scaling is: Among them, a is the scaled mean, b is the scaled variance, N i is a Gaussian distribution with mean a and variance b; The calculation formula of the secondary random scaling is: Among them, a is the lower limit of scaling, b is the upper limit of scaling, P i is a quadratic distribution with a lower bound and b upper bound.

8. The method according to claim 1, characterized in that The initial control strategy neural network of the robot is trained by combining the dense reward value constructed based on dynamic potential energy and the training data generated by the control interaction process, including: Acquire a parameter set of an initial control strategy neural network, and determine an objective function of the parameter set based on the parameter set; Generate an update rule for updating the parameter set based on the objective function; The initial control strategy neural network is trained by the update rule in combination with the dense reward constructed based on dynamic potential energy and the training data generated during the control training process.

9. The method according to claim 8, characterized in that in, The expression of the objective function is: Among them, P(M n ,θ D ) is the parameter set θ D The initial control strategy of the neural network when controlling the robot is the state information sequence M n The probability of occurrence, π θ (a t |s t ) is the parameter set θ D The initial control strategy of the neural network is s t When the robot control instruction a is output t The probability of R(M n ) is M n The sum of dynamic intensive rewards; The expression of the update rule is: Among them, α is the step size, δ is the expected deviation of the dynamic weight relative to the standard weight, and is a random value obtained after sampling.

10. A robot control reinforcement learning method based on dynamic potential energy reward, characterized in that: Applied to the model application stage, wherein the method comprises the following steps: Acquire actual robot information and actual initial state information of the target robot; The actual robot information and the actual initial state information are input into a pre-trained final control strategy neural network to obtain control instructions for the target robot, wherein the pre-trained final control strategy neural network is obtained by intensive reward training based on dynamic potential energy construction.

Images