Quadruped robot gait generation method and system based on improved SAC algorithm

Abstract

The invention discloses a quadruped robot gait generation method based on an improved SAC algorithm. The quadruped robot gait generation method comprises the steps that current state data and motion data of a quadruped robot are obtained; obtaining new state data based on the action data; setting a multi-dimensional reward function, and forming tuple data with the current state data, the action data and the new state data; inputting the tuple data into an intelligent agent formed by an actor network and a multi-reviewer network to obtain a dynamic weight and an action value; obtaining a mixed strategy gradient based on the dynamic weight and the action value; based on the mixed strategy gradient and the loss function, training the actor network and the multi-head reviewer network to obtain a trained actor network and a trained multi-head reviewer network; and obtaining current state data of the quadruped robot, and inputting the current state data into the trained actor network and the trained multi-head commentator network to obtain optimized gait data. And the gait generation efficiency and the gait robustness of the quadruped robot are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of gait generation of a quadruped robot, and more particularly to a gait generation method and system of a quadruped robot based on an improved SAC algorithm. Background Art

[0002] In the study of quadruped robots, the robot's gait is crucial. Gait refers to the coordinated movement pattern of the robot's four legs. It selects a reasonable leg trajectory to prevent the robot from tipping over. It determines the robot's movement speed, stability and energy efficiency. Optimizing the gait can significantly improve the robot's performance and adaptability. However, the quadruped robot is a multi-degree-of-freedom, complex system, and the modeling process requires a lot of time and computing resources. Secondly, the model-based algorithm has high requirements for model accuracy, but in actual applications, the robot's physical parameters and environmental conditions may change, resulting in the actual system not matching the model, reducing the system's robustness.

[0003] For the movement of quadruped robots, reinforcement learning is used for gait design, which can generate effective joint motion trajectories without in-depth knowledge of the robot's gait; and the core of reinforcement learning is to train the robot to take actions to maximize the expected cumulative reward. In the past, the design of deep reinforcement learning reward functions usually learned a single value function by adding multiple parts, each of which quantitatively described an aspect of the quality of the walking task, such as body balance maintenance, motor torque, direction of movement, etc. This reward function design method may destroy the correlation between different rewards, thereby limiting learning efficiency.

[0004] Therefore, how to improve the gait generation efficiency and gait robustness of quadruped robots is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a quadruped robot gait generation method and system based on an improved SAC algorithm, which improves the gait generation efficiency and gait robustness of the quadruped robot.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A gait generation method for a quadruped robot based on an improved SAC algorithm, comprising:

[0008] Obtain the current state data and motion data of the quadruped robot;

[0009] Obtain new state data based on the action data;

[0010] Setting a multi-dimensional reward function and forming tuple data with the current state data, the action data and the new state data;

[0011] Based on the tuple data, the tuple data is input into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values;

[0012] Obtaining a hybrid policy gradient based on the dynamic weight and the action value;

[0013] Based on the hybrid strategy gradient and the loss function, the actor network and the multi-head critic network are trained respectively to obtain a trained actor network and a trained multi-head critic network;

[0014] The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-critic network to obtain optimized gait data.

[0015] Preferably, the multidimensional reward function includes multiple sub-reward functions, and the sub-reward functions are: forward reward function, energy consumption reward function, expected height reward function, angle reward function between the forward direction of the robot body and the forward direction of movement, drift reward function and shaking reward function.

[0016] Preferably, the dynamic weight acquisition method is:

[0017] The tuple data is input into the agent;

[0018] The agent normalizes the multi-dimensional reward function in the tuple data to a specified range and calculates a reward sample mean and a reward sample variance;

[0019] The dynamic weight is obtained based on the reward sample mean and the reward sample variance.

[0020] Preferably, the dynamic weight is specifically:

[0021]

[0022] Among them, m k represents the dynamic weight, K represents the number of sub-reward functions, μ k represents the reward sample mean, σ k represents the reward sample variance.

[0023] Preferably, the multi-head critic network includes a plurality of head networks with the same structure;

[0024] The head network includes: a splicing module, a first hidden layer, a first activation layer, a second hidden layer and a second activation layer;

[0025] The current state data and the action data in the tuple data are input into the splicing module to obtain splicing data;

[0026] The spliced ​​data is sequentially input into the first hidden layer, the first activation layer, the second hidden layer and the second activation layer to obtain the action value.

[0027] Preferably, each of the head networks learns the corresponding action value based on the corresponding sub-reward function:

[0028] Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )]

[0029] Among them, Q t,ki (s t ,a t ) represents the action value of the kth sub-reward function, r tk represents the kth sub-reward function, s t Indicates the current state data, a t Represents action data, s t+1 Represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, Qt,ki(st +1 ,at +1 ) represents the Q value of the next state-action pair, logπ(a t+1 |s t+1 ) means in state s t+1 Next select action a t+1 The logarithm of the probability of .

[0030] Preferably, the hybrid strategy gradient acquisition method is:

[0031] Obtaining a corresponding policy gradient based on the action value;

[0032] Based on the dynamic weights, all policy gradients are weighted and fused to obtain the mixed policy gradient:

[0033]

[0034] in, represents the partial derivative of the actor network parameter φ, π φ (a t |s t ) represents a policy function determined by parameter φ in state s t Take action a tThe probability of Indicates action a t Find the partial derivative, f φ (ε t ;s t ) represents the actor network according to the current state s t and noise ε t The generated action, Indicates that in the current state s t Select action a t The minimum Q value when

[0035] Preferably, the loss function is specifically:

[0036]

[0037] in, Represents the sampled state-action pair (s t ,a t ) expectations, represents the action value, θ ki represents the parameters of the multi-head critic network, y t,k Indicates the target value.

[0038] Preferably, the current state data includes: joint angle, joint angular velocity, joint torque and body posture.

[0039] A quadruped robot gait generation system based on an improved SAC algorithm includes: a data acquisition module, a gradient acquisition module, a network optimization module and a result output module;

[0040] The data acquisition module is used to obtain the current state data and action data of the quadruped robot; obtain new state data based on the current state data; set a multi-dimensional reward function, and form tuple data with the current state data, the action data and the new state data;

[0041] The gradient acquisition module is used to input the tuple data into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values; and obtain a hybrid strategy gradient based on the dynamic weights and the action values;

[0042] The network optimization module is used to train the actor network and the multi-critic network respectively based on the hybrid strategy gradient and the loss function to obtain a trained actor network and a trained multi-critic network;

[0043] The result output module is used to obtain the current state data of the quadruped robot and input it into the trained actor network and the trained multi-critic network to obtain optimized gait data.

[0044] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for generating gait of a quadruped robot based on an improved SAC algorithm. The present invention constructs a multi-head critic network in the SAC framework. The multi-head critic network learns a value function for each sub-reward function component separately, instead of simply weighting and summing each reward function in the traditional method. This design enables the system to retain the unique information of each control target and avoids the loss of information in the summation process; the present invention adjusts the optimization priority of different gradients through dynamic weights, improves the learning efficiency based on the dynamic adjustment of priorities, and enables the quadruped robot to better complete complex motion control tasks; the present invention separates the training of the actor network from the multi-head critic network, and the training of the multi-head critic network is updated based on the original reward signal, which can ensure the accuracy and stability of the action value estimation. The training of the actor network is adjusted according to the weighted mixed strategy gradient to adapt to the learning priority of different components. This separation makes the training of the actor network and the multi-head critic network independent and non-interfering, thereby improving the training stability of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flow chart of a method for generating gait of a quadruped robot based on an improved SAC algorithm provided by the present invention.

[0047] Figure 2 This is a schematic diagram of the actor network structure provided by the present invention.

[0048] Figure 3 A schematic diagram of the multi-critic network structure provided by the present invention.

[0049] Figure 4 This is a schematic diagram of the cumulative reward curve provided by the present invention.

[0050] Figure 5 This is a schematic diagram of the strategy loss curve provided by the present invention.

[0051] Figure 6 A schematic diagram of the cumulative reward value for each part of the reward of the MH-SAC algorithm provided by the present invention.

[0052] Figure 7A schematic diagram of the cumulative reward value for each part of the reward of the MHD-SAC algorithm provided by the present invention.

[0053] Figure 8 A schematic diagram of the structure of a quadruped robot gait generation system based on an improved SAC algorithm provided by the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Example 1

[0056] like Figure 1 As shown, the embodiment of the present invention discloses a gait generation method for a quadruped robot based on an improved SAC algorithm, comprising:

[0057] Obtain the current state data and motion data of the quadruped robot;

[0058] Get new state data based on current state data;

[0059] Set a multi-dimensional reward function and form a tuple data with the current state data, action data and new state data;

[0060] Based on the tuple data input to the intelligent agent composed of the actor network and the multi-head critic network, dynamic weights and action values ​​are obtained;

[0061] Get hybrid policy gradient based on dynamic weights and action values;

[0062] Based on the hybrid strategy gradient and loss function, the actor network and the multi-head critic network are trained respectively to obtain a trained actor network and a trained multi-head critic network;

[0063] The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-critic network to obtain the optimized gait data.

[0064] Example 2

[0065] In order to avoid the complex modeling process of high-dimensional nonlinear dynamic systems, it is very useful to develop a control strategy that does not rely on the precise knowledge of the system dynamics and can handle stochastic dynamics and object model adaptation. To meet these challenges, reinforcement learning-based control, as a data-driven approach, has been shown to be a good alternative to traditional control methods.

[0066] The SAC algorithm contains an entropy-based objective and two critic networks to address overestimation bias. SAC uses a random strategy to enhance stability with more random exploration and faster convergence, thereby producing a robust strategy. The performance of the algorithm is less sensitive to different hyperparameter values, which greatly reduces the time spent on tuning hyperparameters.

[0067] Based on this, an embodiment of the present invention discloses a gait generation method for a quadruped robot based on an improved SAC algorithm, comprising:

[0068] Get the current state data and action data of the quadruped robot:

[0069] Preferably, the current state data of the quadruped robot is obtained based on the simulation environment, and the current state data includes: joint angles, joint angular velocities, joint torques and body postures.

[0070] Preferably, based on the current state data s t Input into the actor network to obtain action data a t .

[0071] Preferably, Figure 2 As shown, the actor network includes a third hidden layer, a third activation layer, a fourth hidden layer and a fourth activation layer; the current state data s t As input, the state is extracted and nonlinearly transformed in turn through the third hidden layer, the third activation layer, the fourth hidden layer and the fourth activation layer, and the mean and standard deviation of the Gaussian distribution are output, which correspond to the expected value and randomness of taking actions in the current state. The action data a is obtained based on the mean and standard deviation of the Gaussian distribution. t .

[0072] Preferably, the current state data includes: joint angle, joint angular velocity, joint torque and body posture.

[0073] Preferably, in this embodiment, the third hidden layer and the fourth hidden layer both have 256 neurons, and the third activation layer and the fourth activation layer both use the ReLU activation function.

[0074] Get new state data based on action data:

[0075] Preferably, based on the action data a t Input to the quadruped robot in the simulation environment, the quadruped robot performs corresponding actions and generates new state data s t+1 .

[0076] Set up a multi-dimensional reward function and form a tuple data with the current state data, action data, and new state data:

[0077] Preferably, the multidimensional reward function includes multiple sub-reward functions, which are: forward reward function, energy consumption reward function, expected height reward function, angle reward function between the forward direction of the robot body and the forward direction of movement, drift reward function and sway reward function.

[0078] Preferably, in reinforcement learning, the reward function can encourage beneficial behaviors while inhibiting undesirable behaviors. The reward function provides the target signal for learning. The multi-dimensional reward function is defined as:

[0079]

[0080] Preferably, forward motion means that the robot keeps moving forward. By giving positive rewards, the robot can be guided to move forward continuously and improve its motion efficiency, that is, the forward reward function is set

[0081]

[0082] Among them, x t represents the displacement of the robot on the X axis at time t, ζ f Represents the weight of the robot's forward motion reward function.

[0083] Preferably, the energy consumption reward function Specifically:

[0084]

[0085] Among them, e represents the energy consumption weight, τ i represents the torque of the i-th joint motor, ω i represents the angular velocity of the i-th joint motor, and Δt represents the time interval between two adjacent control moments.

[0086] Preferably, the expected height reward function is Specifically:

[0087]

[0088] Among them, h represents the weight of the expected height reward function, z represents the trunk height of the quadruped robot in the current state, and z d represents the desired torso height of the quadruped robot.

[0089] Preferably, in order to prevent the quadruped robot from walking backwards, the present invention also provides a reward function for the angle between the forward direction of the robot body and the forward direction of the movement: Specifically:

[0090]

[0091] Among them, a represents the weight of the reward function for the angle between the robot's forward direction and the forward direction of motion, Represents the angle between the forward direction of the robot body and the forward direction of movement.

[0092] Preferably, the drift reward function Used to punish the robot's lateral displacement (Y-axis direction). We hope that the robot will try to walk in the forward direction (X-axis direction) instead of drifting too much in the lateral direction. Based on this, the drift reward function is set

[0093]

[0094] Among them, d represents the weight of the drift reward function, and y represents the displacement in the Y-axis direction.

[0095] Preferably, the shaking reward function Used to punish the robot's shaking in the vertical direction (Z-axis direction). By setting the shaking reward function Ensure that the robot remains relatively stable while walking and avoid excessive up and down shaking. Specifically:

[0096]

[0097] Among them, u represents the actual orientation of the robot's torso, and v represents the vertical vector of the robot in an ideal stable state. When the robot remains completely stable without any tilt or shake, its vertical vector u should be completely aligned with the v vector.

[0098] Preferably, when setting the reward function, it is necessary to weigh the importance of each part, i.e., the weight, and adjust it according to the specific task requirements and environmental conditions. For example, in tasks with high speed requirements, the weight of forward motion is increased accordingly; while on rough terrain, more attention needs to be paid to the robot's stability and leg coordination, and the weight of the corresponding parts is increased accordingly.

[0099] Preferably, the tuple data is (s t ,a t ,s t+1 ,r t ) and stored in the playback buffer.

[0100] Based on the tuple data input to the agent composed of the actor network and the multi-head critic network, the dynamic weights and action values ​​are obtained:

[0101] Preferably, in this embodiment, the actor network and the multi-head critic network together constitute the intelligent agent.

[0102] Preferably, the dynamic weight acquisition method is:

[0103] Tuple data is input to the agent;

[0104] The agent normalizes the multi-dimensional reward function in the tuple data to a specified range and calculates the reward sample mean and reward sample variance;

[0105] The dynamic weight is obtained based on the reward sample mean and reward sample variance.

[0106] Preferably, this embodiment normalizes the multidimensional reward function to the range of (0,1), and then calculates the reward sample mean μ = [μ 1 ,...,μ k ,μ K ] and reward sample variance σ=[σ 1 ,...,σ k ,...,σ K ].

[0107] Preferably, the reward sample mean μ k It reflects the difficulty of the kth part to obtain a larger reward. A large mean value indicates that this part is more likely to obtain a larger reward. k It reflects the degree of change of the reward of the kth part. A large variance indicates that the learning process of this part is unstable.

[0108] Preferably, the reward sample mean μ is taken into consideration when determining the dynamic weight. k and reward sample variance σ k For larger μ k We give higher priority to the part with larger σ to improve the overall performance of the learning process faster. k We also increase the priority of the part so that the agent can learn how to obtain rewards stably as early as possible and reduce the fluctuation and uncertainty of the learning process. In summary, the priority weight of each policy gradient part is the dynamic weight m k for:

[0109]

[0110] Among them, m k represents the dynamic weight, k∈1,2,...K, K represents the number of sub-reward functions, μ k represents the reward sample mean, σ k represents the reward sample variance.

[0111] Preferably, the dynamic weight m kIt is calculated based on the experience samples obtained from the interaction between the agent and the environment, so it can be adjusted adaptively to adapt to task changes and the learning process of the agent. By introducing a priority learning strategy based on difficulty and stability, it can help the agent learn multiple reward parts more efficiently and stably, and accelerate its learning process in complex tasks. This strategy fully considers the dependencies and difficulty differences between different components, making the learning process of the agent more intelligent and humane.

[0112] Preferably, Figure 3 As shown, the multi-head critic network includes multiple head networks with the same structure;

[0113] The head network includes: a splicing module, a first hidden layer, a first activation layer, a second hidden layer, and a second activation layer;

[0114] The current state data and action data in the tuple data are input into the splicing module to obtain splicing data;

[0115] The concatenated data are sequentially input into the first hidden layer, the first activation layer, the second hidden layer, and the second activation layer to obtain the action value.

[0116] Preferably, it can be represented as a vector: Where i=1,2.

[0117] Preferably, each head network learns the corresponding action value based on the corresponding sub-reward function, represented as a vector: Where i = 1, 2, where:

[0118] Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )];

[0119] Among them, Q t,ki (s t ,a t ) represents the action value of the kth sub-reward function, r tk represents the kth sub-reward function, s t Indicates the current state data, a t Represents action data, s t+1 Represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, Q t,ki (s t+1 ,at+1 ) represents the Q value of the next state-action pair, which represents the long-term value of taking the corresponding action. t+1 |s t+1 ) means in state s t+1 Next select action a t+1 The logarithm of the probability of .

[0120] Preferably, the action value Q of the kth sub-reward function is t,ki (s t ,a t ) depends not only on the current reward value r t , also depends on the next state s t+1 Take the best action a t+1 This update strategy can make the multi-critic converge to the value of a consistent strategy, and all sub-reward functions are optimized in the same direction.

[0121] Preferably, the entire action value is defined as t,ki The sum of

[0122] Preferably, each head network independently learns a different sub-reward function corresponding to an action value. Through this multi-head structure, the value of the action in different aspects can be evaluated simultaneously, thereby guiding the strategy learning process more comprehensively.

[0123] Preferably, the multi-head critic network includes an online network and a target network, which have the same structure but independent parameters. Meanwhile, the actor network converts the current state s t As input, the state is feature extracted and nonlinearly transformed through multiple hidden layers, and the mean and standard deviation of the action distribution are output, which represents the expected value and randomness of taking actions in the current state. The action is sampled from the Gaussian distribution using the reparameterization technique to ensure the conductivity of the policy gradient.

[0124] Preferably, each critic network consists of two parts: an online network and a target network: the online critic network is responsible for calculating the Q value of the current state-action in real time and directly participates in training updates, while the target critic network is used to calculate the target Q value to provide a stable learning target.

[0125] Preferably, during the training process, a batch of tuple data is randomly extracted from the experience replay buffer as training data, and K head networks process the sampled data in parallel, and each outputs a corresponding action value.

[0126] Based on the dynamic weights and action values, a hybrid policy gradient is obtained:

[0127] Preferably, the hybrid strategy gradient acquisition method is:

[0128] Get the corresponding policy gradient based on the action value:

[0129] Based on the dynamic weights, all policy gradients are weighted and fused to obtain a mixed policy gradient.

[0130]

[0131]

[0132] in, represents the partial derivative of the actor network parameter φ, π φ (a t |s t ) represents a policy function determined by parameter φ in state s t Take action a t The probability of Indicates action a t Find the partial derivative, f φ (ε t ;s t ) represents the actor network according to the current state s t and noise ε t The generated action, ε t is noise sampled from a standard normal distribution to increase exploration, Indicates that in the current state s t Select action a t The minimum Q value when

[0133] Preferably, the learning priority is defined according to the difficulty of obtaining rewards under the current strategy, first encouraging the agent to learn those parts that are easy to obtain rewards, and then gradually learning those parts with less rewards.

[0134] Preferably, the dynamic weight m k Weighted Policy Gradients, instead of directly weighting the rewards, we separate the training of the actor network from the training of the multi-headed critic network. The training of the multi-headed critic network is no longer directly affected by the reward value adjustment, but is updated based on the original reward signal, which ensures the accuracy and stability of the action value estimation. The training of the actor network is adjusted according to the weighted mixed policy gradient to adapt to the learning priority of different components. This separation makes the training of the actor network and the multi-headed critic network independent and non-interfering, thereby improving the training stability of the entire system.

[0135] Preferably, hybrid policy gradient Used to update the parameters of the actor network and guide the behavior selection of the agent. It is worth noting that the hybrid policy gradient is updated dynamically, and each update is based on the current dynamic weight m kThis dynamic adjustment mechanism allows the agent to adapt to changes in tasks and its own learning process, and continuously optimize its learning strategy.

[0136] Based on the mixed strategy gradient and loss function, the actor network and the multi-head critic network are trained respectively to obtain the trained actor network and the trained multi-head critic network:

[0137] Preferably, the loss function is specifically:

[0138]

[0139] in, Represents the sampled state-action pair (s t ,a t ) expectations, represents the action value, θ ki represents the parameters of the multi-head critic network, y t,k Indicates the target value.

[0140] Preferably, the target value Among them, r(s t ,a t ) represents the immediate reward, γ represents the discount factor, Indicates that in the next state s t+1 Select action a t+1 The minimum action value when α represents the entropy regularization coefficient, log(π φ (a t+1 |s t+1 ) represents the entropy regularization term of the strategy, which is used to encourage exploration.

[0141] Preferably, the termination condition is checked to see whether the maximum number of training rounds has been reached: if the termination condition has not been reached, training continues; if the termination condition has been reached, the final model is saved and training ends.

[0142] The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-critic network to obtain the optimized gait data.

[0143] Optimally, the multi-dimensional reward function provides the target signal for learning, the multi-head critic network ensures that these target signals can be accurately evaluated and learned, and the dynamic policy gradient ensures the rationality and efficiency of the learning process. This design not only solves the information loss problem caused by simple summation in traditional methods, but also improves the learning efficiency through dynamic adjustment of priorities, enabling the quadruped robot to better complete complex motion control tasks.

[0144] Preferably, the quadruped robot achieves preset motion modes such as smooth and straight forward motion based on optimized gait data.

[0145] Example 3

[0146] Example verification

[0147] The action space of the quadruped robot t It is composed of the joint angles of each leg. In this embodiment, each leg has 3 degrees of freedom, corresponding to the hip joint, thigh joint and knee joint. Therefore, the complete action space is a 12-dimensional continuous space, and each dimension represents the control instruction of a joint. The joint angle is used as the action to directly control the movement of the robot.

[0148] For the control task of quadruped robots, an improved SAC algorithm is used, combined with the actor-critic framework to design a neural network structure suitable for quadruped robots.

[0149] This example compares the performance of three algorithms: SAC, multi-head review SAC (MH-SAC), and multi-head review and dynamic policy gradient SAC (MHD-SAC) in the task of generating gait for a quadruped robot. To ensure the fairness of the experiment, the hyperparameter settings of the three algorithms are kept consistent, as shown in Table 1:

[0150] Table 1 Hyperparameters and reward function weights of the three algorithms

[0151]

[0152]

[0153] By analyzing the reward curve, loss curve, and the curve of each reward function in the MH and MHD algorithms during training, we can comprehensively evaluate the applicability and effectiveness of these algorithms on complex control problems.

[0154] like Figure 4 As shown, from the reward curve Figure 4 It can be seen that within the first 1500000 rounds of training, the MHD-SAC algorithm showed the fastest convergence speed and the highest reward value, indicating that it has achieved a better balance between exploration and utilization, and its learning efficiency and policy quality are better than the other two algorithms. The performance of the MH-SAC algorithm is second, while the convergence speed and reward value of the SAC algorithm are relatively low. This shows that the introduction of multi-head structure and dynamic policy gradient can significantly improve the performance of the algorithm in continuous control tasks. After 1500000 rounds, the change of the reward curve tends to be flat, but the reward values ​​of the MHD-SAC and MH-SAC algorithms are still stably higher than the SAC algorithm, further confirming the superiority of the MHD method.

[0155] like Figure 5 As shown, from the strategy loss curve Figure 5It can be seen that the loss values ​​of the MHD-SAC and MH-SAC algorithms gradually stabilize after 1,500,000 rounds, while the SAC algorithm needs about 2,000,000 rounds to reach a stable state. In addition, the policy loss curves of the MHD-SAC and MH-SAC algorithms change faster, indicating that they can use empirical data more effectively and accelerate the policy learning process. This also shows that the multi-head structure and dynamic policy gradient play an important role in improving the learning efficiency and training stability of the algorithm.

[0156] like Figure 6-Figure 7 As shown in the figure, by observing the reward value curves of each reward part in the MH-SAC and MHD-SAC algorithms, it is found that both algorithms can achieve reward convergence within 2,500,000 rounds. This shows that the multi-head structure can effectively handle multi-objective optimization problems, allowing the algorithm to achieve a good balance in each subtask.

[0157] Example 4

[0158] like Figure 8 As shown, a quadruped robot gait generation system based on an improved SAC algorithm includes: a data acquisition module, a gradient acquisition module, a network optimization module and a result output module;

[0159] The data acquisition module is used to obtain the current state data and action data of the quadruped robot; obtain new state data based on the current state data; set a multi-dimensional reward function, and form tuple data with the current state data, action data and new state data;

[0160] A gradient acquisition module is used to obtain dynamic weights and action values ​​based on tuple data input to the intelligent agent composed of the actor network and the multi-head critic network; and obtain a hybrid policy gradient based on the dynamic weights and action values;

[0161] A network optimization module is used to train the actor network and the multi-head critic network based on the hybrid policy gradient and loss function to obtain a trained actor network and a trained multi-head critic network;

[0162] The result output module is used to obtain the current state data of the quadruped robot and input it into the trained actor network and the trained multi-critic network to obtain the optimized gait data.

[0163] Preferably, the function implementation methods of the above-mentioned functional modules in this embodiment correspond one-to-one to the contents of the above-mentioned methods, and will not be repeated here one by one.

[0164] Example 5

[0165] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0166] Memory, used to store computer programs;

[0167] The processor, when used to execute the program stored in the memory, can implement a quadruped robot gait generation method based on an improved SAC algorithm as in Example 1 or 2.

[0168] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory to execute a quadruped robot gait generation method based on an improved SAC algorithm in Embodiment 1 or 2.

[0169] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0170] Through the above technical solutions, it can be known that the present invention discloses a method and system for generating gait of a quadruped robot based on an improved SAC algorithm. The present invention constructs a multi-head critic network in the SAC framework. The multi-head critic network learns a value function for each sub-reward function component separately, instead of simply weighting and summing each reward function in the traditional method. This design enables the system to retain the unique information of each control target and avoids the loss of information in the summation process; the present invention adjusts the optimization priority of different gradients through dynamic weights, improves the learning efficiency based on the dynamic adjustment of priorities, and enables the quadruped robot to better complete complex motion control tasks; the present invention separates the training of the actor network from the multi-head critic network, and the training of the multi-head critic network is updated based on the original reward signal, which can ensure the accuracy and stability of the action value estimation. The training of the actor network is adjusted according to the weighted mixed strategy gradient to adapt to the learning priority of different components. This separation makes the training of the actor network and the multi-head critic network independent and non-interfering, thereby improving the training stability of the entire system.

[0171] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0172] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gait generation method for a quadruped robot based on an improved SAC algorithm, characterized in that: include: Obtain the current state data and motion data of the quadruped robot; Obtain new state data based on the action data; Setting a multi-dimensional reward function and forming tuple data with the current state data, the action data and the new state data; Based on the tuple data, the tuple data is input into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values; Obtaining a hybrid policy gradient based on the dynamic weight and the action value; Based on the hybrid strategy gradient and the loss function, the actor network and the multi-head critic network are trained respectively to obtain a trained actor network and a trained multi-head critic network; The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-critic network to obtain optimized gait data.

2. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 1, characterized in that: The multidimensional reward function includes multiple sub-reward functions, which are: forward reward function, energy consumption reward function, expected height reward function, angle reward function between the forward direction of the robot body and the forward direction of movement, drift reward function and shaking reward function.

3. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 2, characterized in that: The dynamic weight acquisition method is: The tuple data is input into the agent; The agent normalizes the multi-dimensional reward function in the tuple data to a specified range and calculates a reward sample mean and a reward sample variance; The dynamic weight is obtained based on the reward sample mean and the reward sample variance.

4. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 3, characterized in that: The dynamic weight is specifically: Among them, m k represents the dynamic weight, K represents the number of sub-reward functions, μ k represents the reward sample mean, σ k represents the reward sample variance.

5. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 4, characterized in that: The multi-head critic network includes multiple head networks with the same structure; The head network includes: a splicing module, a first hidden layer, a first activation layer, a second hidden layer and a second activation layer; The current state data and the action data in the tuple data are input into the splicing module to obtain splicing data; The spliced ​​data is sequentially input into the first hidden layer, the first activation layer, the second hidden layer and the second activation layer to obtain the action value.

6. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 5, characterized in that: Each of the head networks learns the corresponding action value based on the corresponding sub-reward function: Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )] Among them, Q t,ki (s t ,a t ) represents the action value of the kth sub-reward function, r tk represents the kth sub-reward function, s t Indicates the current state data, a t Represents action data, s t+1 Represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, Qt,ki(st +1 ,at +1 ) represents the Q value of the next state-action pair, logπ(a t+1 |s t+1 ) means in state s t+1 Next select action a t+1 The logarithm of the probability of .

7. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 6, characterized in that: The hybrid strategy gradient acquisition method is: Obtaining a corresponding policy gradient based on the action value; Based on the dynamic weights, all policy gradients are weighted and fused to obtain the mixed policy gradient: Among them, φ represents the partial derivative of the actor network parameter φ, π φ (a t |s t ) represents a policy function determined by parameter φ in state s t Take action a t The probability of Indicates action a t Find the partial derivative, f φ (ε t ;s t ) represents the actor network according to the current state s t and noise ε t The generated action, Indicates that in the current state s t Select action a t The minimum Q value when 8. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 6, characterized in that: The loss function is specifically: in, Represents the sampled state-action pair (s t ,a t ) expectations, represents the action value, θ ki represents the parameters of the multi-head critic network, y t,k Indicates the target value.

9. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 1, characterized in that: The current state data includes: joint angle, joint angular velocity, joint torque and body posture.

10. A quadruped robot gait generation system based on an improved SAC algorithm, applied to a quadruped robot gait generation method based on an improved SAC algorithm as claimed in any one of claims 1 to 9, characterized in that: include: Data acquisition module, gradient acquisition module, network optimization module and result output module; The data acquisition module is used to obtain the current state data and motion data of the quadruped robot; and obtain new state data based on the current state data; Setting a multi-dimensional reward function and forming tuple data with the current state data, the action data and the new state data; The gradient acquisition module is used to input the tuple data into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values; and obtain a hybrid strategy gradient based on the dynamic weights and the action values; The network optimization module is used to train the actor network and the multi-critic network respectively based on the hybrid strategy gradient and the loss function to obtain a trained actor network and a trained multi-critic network; The result output module is used to obtain the current state data of the quadruped robot and input it into the trained actor network and the trained multi-critic network to obtain optimized gait data.

Images