Four-legged robot reinforcement learning control method and device, electronic equipment and storage medium

Abstract

This invention provides a reinforcement learning control method, device, electronic device, and storage medium for a quadruped robot, relating to the field of robot control methods. Within one control cycle, the method includes the following steps: constructing a state observation vector, which includes the robot's linear velocity, angular velocity, gravity projection vector, joint position deviation, joint angular velocity, action command from the previous control cycle, desired velocity, and gait phase encoding; inputting the state observation vector into a pre-trained reinforcement learning policy network, which then outputs action commands; and having servos drive the joints to complete the action based on the action commands. This invention can improve the gait stability of quadruped robots and has strong practicality.

Description

Technical Field

[0001] This application relates to the field of robot control methods, specifically to a reinforcement learning control method, device, electronic equipment, and storage medium for a quadruped robot. Background Technology

[0002] With the development of smart home and security monitoring technologies, the demand for automated inspection equipment in homes and small offices is increasing. Currently, mobile monitoring equipment on the market is mainly wheeled or tracked robots. For example, the existing technology disclosed "A Wheeled Home Monitoring Robot" (application number: CN201520268574.7) drives the wheels through a chassis, motor, synchronous belt transmission mechanism and universal coupling, and achieves mobile monitoring by combining anti-roll bars and shock absorbers. Although this solution has a simple structure and low cost, it has problems such as poor terrain adaptability, lack of autonomous recovery capability and large monitoring blind spots.

[0003] Existing technologies also disclose a variety of high-end quadruped robots, but they have poor gait stability when adapting to complex and ever-changing terrain environments, making it difficult for them to cope with home environments with many obstacles. Summary of the Invention

[0004] To address the aforementioned deficiencies in the prior art, this application provides a reinforcement learning control method, device, electronic device, and storage medium for quadruped robots, which can improve the gait stability of quadruped robots and has strong practicality.

[0005] To achieve the above objectives, the present invention employs the following techniques: A reinforcement learning control method for a quadruped robot, comprising the following steps within one control cycle: A state observation vector is constructed, comprising fuselage linear velocity, fuselage angular velocity, gravity projection vector, joint position deviation, joint angular velocity, action command from the previous control cycle, desired velocity, and gait phase encoding. When constructing the gait phase encoding within the state observation vector, the gait phase variable ϕ1 within the current control cycle is calculated. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle; Gait phase coding is constructed based on ϕ1. The formula for constructing gait phase coding is as follows: Among them, P phase Indicates gait phase encoding; The state observation vector is input into a pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions. The servo motor drives the joints to complete the movement based on the motion command.

[0006] A reinforcement learning control device for a quadruped robot includes: a vector construction module, an instruction output module, and an instruction processing module; The vector construction module is used to construct a state observation vector that includes fuselage linear velocity, fuselage angular velocity, gravity projection vector, joint position deviation, joint angular velocity, motion command from the previous control cycle, desired velocity, and gait phase encoding. When constructing the gait phase encoding in the state observation vector, the gait phase variable ϕ1 within the current control cycle is calculated. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle; Gait phase coding is constructed based on ϕ1. The formula for constructing gait phase coding is as follows: Among them, P phase Indicates gait phase encoding; The instruction output module is used to input the state observation vector into the pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions. The instruction processing module is used to enable the servo motor to drive the joints to complete actions based on the action instructions.

[0007] An electronic device includes: at least one processor and a memory; wherein the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to execute the quadruped robot reinforcement learning control method.

[0008] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, controls the device containing the computer-readable storage medium to perform the quadruped robot reinforcement learning control method.

[0009] The beneficial effects of this invention are as follows: 1. This invention uses a state observation vector containing gait phase encoding as input to a reinforcement learning policy network, enabling the network to perceive the rhythm of motion. This effectively reduces gait disorder and discontinuity, making the quadruped robot's motion more stable and with stronger periodic regularity.

[0010] 2. By pre-training the reinforcement learning policy network, target motion patterns with phase differences are generated for different legs. Gait prior knowledge is integrated into the reinforcement learning process, making it easier for the policy to converge to a diagonal gait that conforms to bio-inspired laws during the exploration process, thereby improving the motion efficiency and stability of the quadruped robot.

[0011] 3. By mapping the action commands output by the reinforcement learning policy network to the target joint angles within the actual physical control range according to the action scaling factor, the adaptability of the control method to different robot structures and actuators is improved. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the reinforcement learning control method for a quadruped robot according to an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the process for constructing gait phase codes according to an embodiment of this application.

[0014] Figure 3 This is a flowchart illustrating the pre-trained reinforcement learning policy network according to an embodiment of this application.

[0015] Figure 4 This is a schematic diagram illustrating the process of a servo motor driving a joint to complete an action based on an action command, according to an embodiment of this application.

[0016] Figure 5 This is a schematic diagram of the structure of a quadruped robot reinforcement learning control device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the implementation methods of the present invention will be described in detail below with reference to the accompanying drawings. However, the embodiments described in this invention are only some embodiments of the present invention, and not all embodiments.

[0018] like Figure 1 As shown, one aspect of this application provides a reinforcement learning control method for a quadruped robot, which includes the following steps within one control cycle: S100, Construct the state observation vector.

[0019] S200. Input the state observation vector into the pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions.

[0020] The S300 servo drives the joints to complete actions based on motion commands.

[0021] Preferably, the state observation vector o t Defined as: o t =[v b , ω b , g b , q-q0, ω j , a t-1 , v cmd , p phase ] vb This represents the linear velocity of the fuselage, and v b ∈R 3 .

[0022] ω b This represents the fuselage angular velocity, and ω b ∈R 3 .

[0023] g b Let g represent the gravity projection vector, and g b ∈R 3 .

[0024] q represents the current joint angle and q∈R n q0 represents the default joint angle and q0∈R n q-q0 represents the joint position deviation and q-q0∈R n .

[0025] ω j This represents the fuselage angular velocity, and ω j ∈R n .

[0026] a t-1 This indicates the action command from the previous control cycle, and a t-1 ∈R n .

[0027] v cmd This indicates the desired speed command, and v cmd ∈R 3 .

[0028] p phase This indicates gait phase encoding, and p phase ∈R 2 .

[0029] Where n represents the number of joints.

[0030] In a typical case, the state dimension of the state observation vector is dim(o t )for: dim(o t ) = 3 + 3 + 3 + n + n + n + 3 + 2 For example, for a quadruped robot with eight joints, the state dimension of its state observation vector is 38.

[0031] For v b ω b g b q-q0 and ω j This is usually obtained by collecting current motion state information using various sensors; a t-1 Derived from the historical record of the previous control cycle; vcmd It is given by external input; p phase It is generated by the system.

[0032] Preferred, such as Figure 2 As shown, p phase The construction includes the following steps: S101. Calculate the gait phase variable ϕ1 within the current control cycle. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle.

[0033] S102. Construct gait phase coding based on ϕ1. The formula for constructing gait phase coding is: Among them, P phase This indicates gait phase encoding.

[0034] In typical cases, the control period ranges from 0.01s to 0.03s, and the gait period T ranges from 0.3s to 0.8s. The control period refers to the time required for the quadruped robot to complete one step S100 to S300. The gait period T refers to the time required for the robot to complete a complete and repeatable leg movement cycle. Multiple control periods constitute a complete gait period T. The control moment refers to the time point corresponding to the current control period, that is, the discrete sampling instant on the time axis.

[0035] This design, by performing periodic normalization on the current control moment and using trigonometric function encoding to generate corresponding sine and cosine values ​​for gait phase coding, avoids discontinuities in gait phase variables at gait cycle boundaries. This allows the reinforcement learning policy network to explicitly perceive gait cycle changes, thereby forming a periodic movement pattern during the learning process. This enhances the reinforcement learning policy network's ability to perceive periodic movements and improves the stability and consistency of gait.

[0036] Reinforcement learning policy networks are neural networks that take states as input and actions as output. Their application in the field of robot motion control is existing technology, so it will not be elaborated on here. In this example, before using the reinforcement learning policy network to output action commands to control the movement of the quadruped robot, the reinforcement learning policy network was simulated and trained in advance to enable the reinforcement learning policy network to output stable and efficient action commands.

[0037] Preferred, such as Figure 3 As shown, the pre-training of the reinforcement learning policy network includes the following steps: S400. Calculate the total reward r based on the pre-built reward function, which is: r=w1r vei +w2r orient +w3r phase +w4r smooth Where w1, w2, w3, and w4 are weighting coefficients, r vei For the velocity tracking reward function, r orient For the attitude-stable reward function, r phase Let r be the phase consistency reward function. smooth This is the motion smoothness constraint function.

[0038] S401. Using r as the reward signal, the reinforcement learning policy network is simulated and trained using the proximal policy optimization algorithm.

[0039] Preferably, the velocity tracking reward function r vei for: r vei =exp(-k v ||vv cmd || 2 ) Where v is the actual speed, v cmd For the desired speed, k v This is the speed tracking penalty coefficient; in typical cases, -k v =1.0~3.0; The velocity tracking reward function is used to measure the error between the actual movement speed and the expected speed of the quadruped robot, thereby guiding the robot to complete the specified movement task.

[0040] Attitude-stable reward function r orient for: r orient =exp(-k o (g x 2 +g y 2 )) Among them, g x Let g be the component of the gravity vector along the fuselage's X-axis. y Let k be the component of the gravity vector along the Y-axis of the fuselage. o -k is the attitude stability penalty coefficient; in typical cases, -k o =3~8; The attitude stabilization reward function is used to evaluate the deviation between the body attitude and the desired stable attitude, thereby improving the stability of the robot during movement.

[0041] Action smoothness constraint function r smooth for: r smooth =-ks ||a t -a t-1 || 2 Among them, a t For the action command of the current control cycle, a t-1 k is the action command from the previous control cycle. s k is the motion smoothness penalty coefficient; in typical cases, k s =0.01~0.1; The motion smoothness constraint function is used to constrain the rate of change of control input, thereby reducing energy consumption and improving motion continuity.

[0042] Phase Consistency Reward Function r phase for: Where, q i Let q be the actual angle of the i-th joint. i ref Let n be the reference angle for the i-th joint, n be the total number of joints, and k be the reference angle for the i-th joint. p k is the phase consistency penalty coefficient; in typical cases, k p =2~10; The gait phase consistency reward function is used to compare the difference between the current joint state and the target state generated based on the phase, thereby guiding the robot to form a periodic gait.

[0043] Preferred, q i ref The calculation formula is: q i ref =q offset +A·sin(ϕ2+δ i ) Where, q offset Let A be the joint reference offset, ϕ2 be the amplitude, ϕ2 be the gait phase variable within the current control cycle during simulation training, and δ be the gait phase variable. i Let be the phase offset of the i-th joint.

[0044] The formula for calculating ϕ2 is: Where t2 is the control time of the current control cycle during the simulation training process, and T is the gait cycle.

[0045] This design allows for the generation of target motion patterns with phase differences for different legs during simulation training, based on gait phase variables. Typically, for a pair of diagonally opposite legs (left foreleg and right hind leg), the same phase function is used to generate the target trajectory, δ. iThe value is 0; for the other pair of diagonal leg groups (right front leg and left hind leg), a phase shift is introduced to achieve alternating swing, δ i The value of is π; in this way, gait prior knowledge is integrated into the reinforcement learning process, making it easier for the strategy to converge to a diagonal gait that conforms to biological inspiration during the exploration process, thereby improving movement efficiency and stability.

[0046] The Proximal Policy Optimization (PPO) algorithm in step S401 is existing technology. In this example, its workflow generally includes: based on the total reward r for each control cycle, using r as the reward signal, and calculating and constructing the advantage function A according to the discount factor γ and the GAE parameter λ. t In typical cases, γ=0.99 and λ=0.95.

[0047] Based on the advantage function A t Calculate the objective function L for constructing the near-end policy optimization algorithm. CLIP L CLIP The calculation formula is: L CLIP =E[min(r t (θ)A t ,clip(r t (θ), 1-ϵ, 1+ϵ)A t )] In a typical case, the cutting range ϵ = 0.2.

[0048] Based on L CLIP With a pre-set learning rate, the reinforcement learning policy network π θ The parameters θ are updated, which completes one training of the reinforcement learning policy network. Then, the updated reinforcement learning policy network is trained again. This process is repeated until the reinforcement learning network gradually learns to output high-reward action commands, that is, stable and efficient action commands, enabling the quadruped robot to walk stably.

[0049] Preferred, such as Figure 4 As shown, S300 includes the following steps: S301. Calculate the target joint angle q based on the motion command. target q target The calculation formula is: q target =q0+α·a t Where q0 is the default joint angle, α is the motion scaling factor, and a t The action command for the current control cycle.

[0050] S302. Input the target joint angle to the servo motor, and the servo motor drives the joint to complete the action.

[0051] In this example, a t This can be expressed as the target position offset or standardized control quantity for each joint, and a t ∈[-1,1] n Furthermore, by setting α, the output α of the reinforcement learning policy network can be adjusted. t This is mapped to the target joint angle within the actual physical control range to adapt to different robot structures and actuator capabilities; in typical cases, α = 0.2 rad ~ 0.5 rad.

[0052] A further preferred approach is to introduce a low-level controller to decouple high-level strategies from low-level execution, thereby improving system stability and control accuracy; that is, inputting control commands to the servo motor directly executes the commands based on the target joint angle. q cmd =q target The control commands are then sent to each servo motor via the position control interface to drive the joints to complete the action.

[0053] like Figure 5 As shown in the figure, another aspect of the embodiments of this application provides a quadruped robot reinforcement learning control device, including a vector construction module, an instruction output module, and an instruction processing module.

[0054] The vector construction module is used to construct a state observation vector that includes fuselage linear velocity, fuselage angular velocity, gravity projection vector, joint position deviation, joint angular velocity, motion command from the previous control cycle, desired velocity, and gait phase encoding. When constructing the gait phase encoding in the state observation vector, the gait phase variable ϕ1 within the current control cycle is calculated. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle; Gait phase coding is constructed based on ϕ1. The formula for constructing gait phase coding is as follows: Among them, P phase This indicates gait phase encoding.

[0055] The instruction output module is used to input the state observation vector into the pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions.

[0056] The instruction processing module is used to enable the servo motor to drive the joints to complete actions based on the action instructions.

[0057] In another aspect of this application, an electronic device is provided, comprising: at least one processor and a memory; wherein the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the quadruped robot reinforcement learning control method described in the preceding embodiments.

[0058] In another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to perform the quadruped robot reinforcement learning control method described in the preceding embodiments.

[0059] The above description is only a preferred embodiment of this application and is not intended to limit this application. Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application.

Claims

1. A reinforcement learning control method for a quadruped robot, characterized in that, The method includes the following steps in each control cycle: Construct a state observation vector that includes fuselage linear velocity, fuselage angular velocity, gravity projection vector, joint position deviation, joint angular velocity, action command of the previous control cycle, desired velocity, and gait phase encoding; When constructing the gait phase encoding in the state observation vector, the gait phase variable ϕ1 within the current control cycle is calculated. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle; Gait phase coding is constructed based on ϕ1. The formula for constructing gait phase coding is as follows: Among them, P phase Indicates gait phase encoding; The state observation vector is input into a pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions. The servo motor drives the joints to complete the movement based on the motion command.

2. The reinforcement learning control method for a quadruped robot according to claim 1, characterized in that, Pre-training of reinforcement learning policy networks includes the following steps: The total reward r is calculated based on a pre-built reward function, which is: r=w1r vei +w2r orient +w3r phase +w4r smooth Where w1, w2, w3, and w4 are weighting coefficients, r vei For the velocity tracking reward function, r orient For the attitude-stable reward function, r phase Let r be the phase consistency reward function. smooth This is the motion smoothness constraint function; Using r as the reward signal, the reinforcement learning policy network is simulated and trained using a proximal policy optimization algorithm.

3. The reinforcement learning control method for a quadruped robot according to claim 2, characterized in that, The speed tracking reward function is: r vei =exp(-k v ||vv cmd || 2 ) Where v is the actual speed, v cmd For the desired speed, k v This is the speed tracking penalty coefficient; The attitude-stabilized reward function is: r orient =exp(-k o (g x 2 +g y 2 )) Among them, g x Let g be the component of the gravity vector along the fuselage's X-axis. y Let k be the component of the gravity vector along the Y-axis of the fuselage. o This is the attitude stability penalty coefficient; The phase consistency reward function is: Where, q i Let q be the actual angle of the i-th joint. i ref Let n be the reference angle for the i-th joint, n be the total number of joints, and k be the reference angle for the i-th joint. p This is the phase consistency penalty coefficient; The motion smoothness constraint function is: r smooth =-k s ||a t -a t-1 || 2 Among them, a t For the action command of the current control cycle, a t-1 k is the action command from the previous control cycle. s This is the penalty coefficient for motion smoothness.

4. The reinforcement learning control method for a quadruped robot according to claim 3, characterized in that, q i ref The calculation formula is: q i ref =q offset +A·sin(ϕ2+δ i ) Where, q offset Let A be the joint reference offset, ϕ2 be the amplitude, ϕ2 be the gait phase variable within the current control cycle during simulation training, and δ be the gait phase variable. i This represents the phase offset of the i-th joint; The formula for calculating ϕ2 is: Where t2 is the control time of the current control cycle during the simulation training process, and T is the gait cycle.

5. The reinforcement learning control method for a quadruped robot according to claim 1, characterized in that, The servo motor drives the joints to complete actions based on motion commands, including the following steps: Calculate the target joint angle q based on the motion command. target q target The calculation formula is: q target =q0+α·a t Where q0 is the default joint angle, α is the motion scaling factor, and a t The action command for the current control cycle; The target joint angle is input to the servo motor, which then drives the joint to complete the movement.

6. A reinforcement learning control device for a quadruped robot, characterized in that, include: Vector construction module, instruction output module, instruction processing module; The vector construction module is used to construct state observation vectors including fuselage linear velocity, fuselage angular velocity, gravity projection vector, joint position deviation, joint angular velocity, action command of the previous control cycle, desired velocity, and gait phase encoding; When constructing the gait phase encoding in the state observation vector, the gait phase variable ϕ1 within the current control cycle is calculated. The formula for calculating ϕ1 is: Where t1 is the control time of the current control cycle, and T is the gait cycle; Gait phase coding is constructed based on ϕ1. The formula for constructing gait phase coding is as follows: Among them, P phase Indicates gait phase encoding; The instruction output module is used to input the state observation vector into the pre-trained reinforcement learning policy network, and the reinforcement learning policy network outputs action instructions. The instruction processing module is used to enable the servo motor to drive the joints to complete actions based on the action instructions.

7. An electronic device, comprising: At least one processor and a memory; wherein the memory stores computer-executable instructions; characterized in that the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the quadruped robot reinforcement learning control method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it controls the device containing the computer-readable storage medium to perform the quadruped robot reinforcement learning control method as described in any one of claims 1 to 5.

Images