A quadruped robot gait generation method and system based on an improved SAC algorithm

Abstract

The application discloses a kind of based on the gait generation method of quadruped robot of improved SAC algorithm, comprising: obtaining the current state data and action data of quadruped robot;Based on action data, obtain new state data;Set up multidimensional reward function, and with current state data, action data and new state data form tuple data;Based on tuple data input to the intelligent agent composed of actor network and multi-head critic network, obtain dynamic weight and action value;Based on dynamic weight and action value, obtain mixed policy gradient;Based on mixed policy gradient and loss function, respectively train actor network and multi-head critic network, obtain trained actor network and trained multi-head critic network;The current state data of quadruped robot is obtained and input to trained actor network and trained multi-head critic network, and optimization gait data is obtained.The gait generation efficiency and gait robustness of quadruped robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of quadruped robot gait generation, and more particularly to a quadruped robot gait generation method and system based on an improved SAC algorithm. BACKGROUND

[0002] In the research of quadruped robots, the gait of the robot is crucial. The gait refers to the coordinated movement pattern of the four legs of the robot, and selecting a reasonable leg trajectory can prevent the robot from falling over. It determines the movement speed, stability and energy efficiency of the robot. Optimizing the gait can significantly improve the performance and adaptability of the robot. However, a quadruped robot is a multi-degree-of-freedom, complex system, and the modeling process requires a large amount of time and computing resources. Secondly, model-based algorithms have high requirements for model accuracy, but in actual applications, the physical parameters of the robot and environmental conditions may change, resulting in a discrepancy between the actual system and the model, and reducing the robustness of the system.

[0003] For the motion of a quadruped robot, reinforcement learning is used for gait design, which can generate effective joint motion trajectories without deep understanding of the knowledge related to the gait of the robot. The core of reinforcement learning is to train the robot to take actions to maximize the expected cumulative reward. In the past, the design of the reward function for deep reinforcement learning usually adds multiple parts to learn a single value function, and each part quantitatively describes an aspect of the walking task quality, such as body balance maintenance, motor torque, motion direction, etc. This design method of the reward function may destroy the correlation between different rewards, thereby limiting the learning efficiency.

[0004] Therefore, how to improve the gait generation efficiency and gait robustness of a quadruped robot is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a quadruped robot gait generation method and system based on an improved SAC algorithm, which improves the gait generation efficiency and gait robustness of a quadruped robot.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A quadruped robot gait generation method based on an improved SAC algorithm, comprising:

[0008] Obtaining current state data and action data of a quadruped robot;

[0009] Obtaining new state data based on the action data;

[0010] Setting a multi-dimensional reward function, and forming a tuple data with the current state data, the action data and the new state data;

[0011] inputting the tuple data into an agent composed of an actor network and a multi-headed critic network to obtain a dynamic weight and an action value;

[0012] obtaining a hybrid policy gradient based on the dynamic weight and the action value;

[0013] training the actor network and the multi-headed critic network based on the hybrid policy gradient and a loss function respectively to obtain a trained actor network and a trained multi-headed critic network;

[0014] obtaining optimized gait data by inputting current state data of the quadruped robot into the trained actor network and the trained multi-headed critic network.

[0015] Preferably, the multi-dimensional reward function comprises a plurality of sub-reward functions, and the sub-reward functions are respectively: an advancing reward function, an energy consumption reward function, an expected height reward function, an angle reward function between a forward direction of the robot body and a moving forward direction, a drift reward function, and a sway reward function.

[0016] Preferably, the dynamic weight obtaining method is:

[0017] the tuple data is inputted into the agent;

[0018] the multi-dimensional reward function in the tuple data is normalized to a specified range by the agent, and a reward sample mean and a reward sample variance are calculated;

[0019] the dynamic weight is obtained based on the reward sample mean and the reward sample variance.

[0020] Preferably, the dynamic weight is specifically:

[0021]

[0022] wherein, m k represents the dynamic weight, K represents the number of sub-reward functions, μ k represents the reward sample mean, and σ k represents the reward sample variance.

[0023] Preferably, the multi-headed critic network comprises a plurality of head networks with the same structure;

[0024] The head networks each comprise a concatenation module, a first hidden layer, a first activation layer, a second hidden layer, and a second activation layer.

[0025] The current state data and the action data in the tuple data are inputted into the concatenation module to obtain concatenated data.

[0026] The spliced data is sequentially input to the first hidden layer, the first activation layer, the second hidden layer and the second activation layer to obtain the action value.

[0027] Preferably, each of the head networks learns the corresponding action value based on the corresponding sub-reward function:

[0028] Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )]

[0029] Wherein, Q t,ki (s t ,a t ) represents the action value of the kth sub-reward function, r tk represents the kth sub-reward function, s t represents the current state data, a t represents the action data, s t+1 represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, Qt,ki(st +1 ,at +1 ) represents the Q value of the next state-action pair, logπ(a t+1 |s t+1 ) represents the logarithm of the probability of selecting action a t+1 under state s t+1 .

[0030] Preferably, the hybrid policy gradient acquisition method is:

[0031] Based on the action value, the corresponding policy gradient is obtained;

[0032] Based on the dynamic weight, all the policy gradients are weighted and fused to obtain the hybrid policy gradient:

[0033]

[0034] Wherein, represents the partial derivative of the parameter φ of the actor network, π φ (a t |s t ) represents a policy function determined by the parameter φ taking action a t under state s tthe probability of the action a denotes the action a t derivative, f φ (ε t ; s t ) denotes the action generated by the actor network according to the current state s t and the noise ε t , denotes the minimum Q value when the action a t is selected in the current state s t .

[0035] Preferably, the loss function is specifically:

[0036]

[0037] wherein, denotes the expectation of the sampled state-action pair (s t , a t ), denotes the action value, θ ki denotes the parameters of the multi-headed critic network, y t,k denotes the target value.

[0038] Preferably, the current state data includes: joint angle, joint angular velocity, joint torque and body posture.

[0039] A quadruped robot gait generation system based on an improved SAC algorithm, comprising: a data acquisition module, a gradient acquisition module, a network optimization module and a result output module;

[0040] The data acquisition module is configured to acquire current state data and action data of a quadruped robot, obtain new state data based on the current state data, set a multi-dimensional reward function, and form a tuple data together with the current state data, the action data and the new state data.

[0041] The gradient acquisition module is configured to input the tuple data into an agent composed of an actor network and a multi-headed critic network to obtain a dynamic weight and an action value, and obtain a hybrid policy gradient based on the dynamic weight and the action value.

[0042] The network optimization module is configured to train the actor network and the multi-headed critic network based on the hybrid policy gradient and a loss function, respectively, to obtain a trained actor network and a trained multi-headed critic network.

[0043] The result output module is configured to acquire the current state data of the quadruped robot and input the current state data into the trained actor network and the trained multi-headed critic network to obtain optimized gait data.

[0044] Through the above technical solutions, compared with the prior art, the application provides a four-legged robot gait generation method and system based on an improved SAC algorithm, the application constructs a multi-head critic network in the SAC framework, the multi-head critic network learns a value function for each sub-reward function component, instead of simply weighting and summing the various reward functions in the traditional method, this design enables the system to retain the unique information of each control target and avoids the loss of information in the summation process; the application adjusts the optimization priority of different gradients through dynamic weights, improves the learning efficiency based on priority dynamic adjustment, and enables the four-legged robot to better complete complex motion control tasks; the application separates the training of the actor network and the multi-head critic network, the training of the multi-head critic network is updated based on the original reward signal, which can ensure the accuracy and stability of the action value estimation, and the training of the actor network is adjusted based on the weighted mixed policy gradient to adapt to the learning priority of different components. This separation enables the training of the actor network and the multi-head critic network to be independent of each other and not to interfere with each other, thereby improving the training stability of the entire system. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below, and obviously, the drawings in the following description are only embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0046] Figure 1 A flowchart of a four-legged robot gait generation method based on an improved SAC algorithm is provided for the application.

[0047] Figure 2 An actor network structure diagram is provided for the application.

[0048] Figure 3 A multi-head critic network structure diagram is provided for the application.

[0049] Figure 4 A cumulative reward curve diagram is provided for the application.

[0050] Figure 5 A policy loss curve diagram is provided for the application.

[0051] Figure 6 A cumulative reward value diagram of each part of the reward of the MH-SAC algorithm is provided for the application.

[0052] Figure 7This is a schematic diagram of the cumulative reward value of each part of the MHD-SAC algorithm provided by the present invention.

[0053] Figure 8 This is a structural schematic diagram of a quadruped robot gait generation system based on an improved SAC algorithm provided by the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1

[0056] like Figure 1 As shown, the embodiment of the present invention discloses a method for generating gait of a quadruped robot based on an improved SAC algorithm, comprising:

[0057] Obtain the current state data and motion data of the quadruped robot;

[0058] Get new state data based on current state data;

[0059] Set a multi-dimensional reward function and form a tuple data with the current state data, action data and new state data;

[0060] Based on the tuple data input to the intelligent agent composed of the actor network and the multi-head critic network, dynamic weights and action values ​​are obtained;

[0061] Get hybrid policy gradient based on dynamic weights and action values;

[0062] Based on the hybrid policy gradient and loss function, the actor network and the multi-head critic network are trained respectively to obtain the trained actor network and the trained multi-head critic network;

[0063] The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-head critic network to obtain the optimized gait data.

[0064] Example 2

[0065] In order to avoid the complex modeling process of high-dimensional nonlinear dynamic systems, it is very useful to develop a control strategy that does not rely on the precise knowledge of the system dynamics and can handle stochastic dynamics and object model adaptation. To meet these challenges, reinforcement learning-based control, as a data-driven approach, has been proven to be a good alternative to traditional control methods.

[0066] The SAC algorithm includes an entropy-based objective and two critic networks to address overestimation bias. SAC uses a stochastic policy to enhance stability with more random exploration and faster convergence, resulting in a robust policy. The performance of the algorithm is less sensitive to different hyperparameter values, greatly reducing the time spent adjusting hyperparameters.

[0067] Based on this, the embodiment of the application discloses a quadruped robot gait generation method based on an improved SAC algorithm, comprising:

[0068] Obtain the current state data and action data of the quadruped robot:

[0069] Preferably, the current state data of the quadruped robot is obtained based on a simulation environment, and the current state data includes joint angles, joint angular velocities, joint torques, and body postures.

[0070] Preferably, based on the current state data s t Input into the actor network to obtain action data a t .

[0071] Preferably, as shown in Figure 2 The actor network includes a third hidden layer, a third activation layer, a fourth hidden layer, and a fourth activation layer; the current state data s t is input, and the state is sequentially subjected to feature extraction and nonlinear transformation by the third hidden layer, the third activation layer, the fourth hidden layer, and the fourth activation layer, and the mean and standard deviation of the Gaussian distribution are output, which correspond to the expected value and randomness of taking action under the current state, and the action data a t is obtained based on the mean and standard deviation of the Gaussian distribution.

[0072] Preferably, the current state data includes joint angles, joint angular velocities, joint torques, and body postures.

[0073] Preferably, the third hidden layer and the fourth hidden layer each have 256 neurons, and the third activation layer and the fourth activation layer each adopt a ReLU activation function.

[0074] Obtain new state data based on the action data:

[0075] Preferably, based on the action data a t Input into the quadruped robot in the simulation environment, and the quadruped robot performs corresponding actions to generate new state data s t+1 .

[0076] Set a multi-dimensional reward function, and form a tuple data with the current state data, the action data, and the new state data:

[0077] Preferably, the multi-dimensional reward function comprises a plurality of sub-reward functions, and the sub-reward functions are respectively: an advancing reward function, an energy consumption reward function, an expected height reward function, a robot body forward direction and motion forward direction angle reward function, a drift reward function and a sway reward function.

[0078] Preferably, in the reinforcement learning, the reward function can encourage beneficial behavior while suppressing undesirable behavior, and the reward function provides a learning target signal, and the multi-dimensional reward function is defined as:

[0079]

[0080] Preferably, the advancing motion represents that the robot keeps moving forward, and by giving a positive reward, the robot can be guided to keep moving forward and improve the motion efficiency, that is, the advancing reward function is set.

[0081]

[0082] Wherein, x t represents the displacement of the robot on the X axis at the t time, ζ f represents the weight of the robot forward motion reward function.

[0083] Preferably, the energy consumption reward function Specifically:

[0084]

[0085] Wherein, ζ e represents the energy consumption weight, τ i represents the torque of the i th joint motor, ω i represents the angular velocity of the i th joint motor, and Δt represents the time interval between two adjacent control time points.

[0086] Preferably, the expected height reward function Specifically:

[0087]

[0088] Wherein, ζ h represents the weight of the expected height reward function, z represents the height of the torso of the quadruped robot in the current state, and z d represents the expected height of the torso of the quadruped robot.

[0089] Preferably, in order to avoid the quadruped robot walking forward in reverse, the robot body forward direction and motion forward direction angle reward function is further set. Specifically:

[0090]

[0091] wherein ζ a represents the weight of the reward function of the angle between the forward direction of the robot body and the forward direction of the motion, represents the angle between the forward direction of the robot body and the forward direction of the motion.

[0092] Preferably, the drift reward function is used to punish the robot for displacement in the lateral direction (Y-axis direction). We want the robot to walk as much as possible in the forward direction (X-axis direction) rather than producing too much drift in the lateral direction. Based on this, the drift reward function is set as

[0093]

[0094] wherein ζ d represents the weight of the drift reward function, and y represents the displacement in the Y-axis direction.

[0095] Preferably, the sway reward function is used to punish the robot for swaying in the vertical direction (Z-axis direction). By setting the sway reward function ensure that the robot remains relatively stable when walking, avoiding excessive up-and-down swaying, Specifically:

[0096]

[0097] wherein u represents the actual orientation of the robot body, and v represents an ideal vertical vector of the robot in a stable state. When the robot remains completely stable without any tilting or swaying, its vertical vector u should be completely aligned with the v vector.

[0098] Preferably, when setting the reward function, the importance of each part, i.e., the weight, needs to be balanced, and adjusted according to the specific task requirements and environmental conditions. For example, in tasks with high speed requirements, the weight of the forward motion is correspondingly increased; while on rough terrain, more attention needs to be paid to the stability and leg coordination of the robot, and the weights of the corresponding parts are correspondingly increased.

[0099] Preferably, the tuple data is (s t ,a t ,s t+1 ,r t ) and is stored in the replay buffer.

[0100] Based on the tuple data input into the agent composed of the actor network and the multi-headed critic network, the dynamic weight and the action value are obtained:

[0101] Preferably, in this embodiment, the actor network and the multi-headed critic network together constitute the agent.

[0102] Preferably, the dynamic weight acquisition method is:

[0103] Tuple data is input to the agent;

[0104] The agent normalizes the multi-dimensional reward function in the tuple data to a specified range and calculates the reward sample mean and reward sample variance;

[0105] The dynamic weight is obtained based on the reward sample mean and reward sample variance.

[0106] Preferably, this embodiment normalizes the multi-dimensional reward function to the range of (0,1), and then calculates the reward sample mean μ=[μ1,...,μ k ,μ K ] and reward sample variance σ=[σ1,...,σ k ,...,σ K ].

[0107] Preferably, the reward sample mean μ k Reflects the difficulty of obtaining a larger reward for the kth part. A large mean value indicates that the part is more likely to obtain a larger reward. k It reflects the degree of change of the reward of the kth part. A large variance indicates that the learning process of this part is unstable.

[0108] Preferably, the reward sample mean μ is taken into consideration when determining the dynamic weight. k and reward sample variance σ k For larger μ k We give higher priority to the part with larger σ to improve the overall performance of the learning process faster. k We also increase the priority of the part so that the agent can learn how to obtain rewards stably as early as possible and reduce the fluctuation and uncertainty of the learning process. In summary, the priority weight of each policy gradient part is the dynamic weight m k for:

[0109]

[0110] Among them, m k represents the dynamic weight, k∈1,2,...K, K represents the number of sub-reward functions, μ k represents the reward sample mean, σ k represents the reward sample variance.

[0111] Preferably, the dynamic weight m kThe priority learning strategy based on difficulty and stability can help the agent to learn multiple reward parts more efficiently and stably, and accelerate the learning process of the agent in complex tasks. The strategy fully considers the dependency relationship and difficulty difference between different components, so that the learning process of the agent is more intelligent and humanized.

[0112] Preferably, as shown in Figure 3 The multi-head critic network comprises a plurality of head networks with the same structure;

[0113] Each head network comprises a concatenation module, a first hidden layer, a first activation layer, a second hidden layer and a second activation layer.

[0114] The current state data and the action data in the tuple data are input into the concatenation module to obtain concatenated data.

[0115] The concatenated data is sequentially input into the first hidden layer, the first activation layer, the second hidden layer and the second activation layer to obtain an action value.

[0116] Preferably, the action value can be represented as a vector: Wherein i = 1, 2.

[0117] Preferably, each head network learns a corresponding action value based on a corresponding sub-reward function, which is represented as a vector: Wherein i = 1, 2, wherein:

[0118] Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )];

[0119] Wherein, Q t,ki (s t ,a t ) represents the action value of the kth sub-reward function, r tk represents the kth sub-reward function, s t represents the current state data, a t represents the action data, s t+1 represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, and Q t,ki (s t+1 ,at+1 ) represents the Q-value of the next state-action pair, which represents the long-term value of taking the corresponding action, logπ(a t+1 |s t+1 ) represents the logarithm of the probability of selecting action a t+1 in state s t+1 .

[0120] Preferably, the action value Q t,ki (s t ,a t ) of the k-th head depends not only on the current reward value r t , but also on the action value obtained by taking the optimal action a t+1 in the next state s t+1 . This update strategy can make the multi-headed critic converge to the value of the consistent policy, and all sub-reward functions are optimized in the same direction.

[0121] Preferably, the entire action value is defined as the sum of each Q t,ki , i.e.

[0122] Preferably, each head network independently learns a different sub-reward function corresponding to an action value, and through this multi-headed structure, the value of the action in different aspects can be evaluated simultaneously, thereby more comprehensively guiding the learning process of the policy.

[0123] Preferably, the multi-headed critic network includes an online network and a target network, which have the same structure but independent parameters. At the same time, the actor network takes the current state s t as input, performs feature extraction and nonlinear transformation on the state through multiple hidden layers, and outputs the mean and standard deviation of the action distribution, representing the expected value and randomness of taking action in the current state. The reparameterization trick is used to sample actions from the Gaussian distribution to ensure the derivability of the policy gradient.

[0124] Preferably, each critic network includes an online network and a target network: the online critic network is responsible for calculating the Q-value of the current state-action in real time and directly participating in training and updating, while the target critic network is used to calculate the target Q-value to provide a stable learning target.

[0125] Preferably, during training, a batch of tuple data is randomly sampled from the experience replay buffer as training data, and K head networks process the sampled data in parallel, each outputting a corresponding action value.

[0126] Based on the dynamic weight and action value, a hybrid policy gradient is obtained:

[0127] Preferably, the method for obtaining the hybrid policy gradient is:

[0128] Based on the action value, the corresponding policy gradient is obtained:

[0129] Based on the dynamic weight, all the policy gradients are weighted and fused to obtain a hybrid policy gradient

[0130]

[0131]

[0132] wherein, denotes the partial derivative of the parameter φ of the actor network, π φ (a t |s t ) denotes the probability of a policy function determined by the parameter φ to take action a t in state s t , denotes the partial derivative of the action a t , f φ (ε t ; s t ) denotes the action generated by the actor network according to the current state s t and the noise ε t , and ε t is the noise sampled from the standard normal distribution, which is used to increase exploration, denotes the minimum Q value when action a t is selected in the current state s t .

[0133] Preferably, the priority of learning is defined according to the difficulty of obtaining rewards under the current policy, which first encourages the agent to learn those parts that are easy to obtain rewards, and then gradually learns those parts with less rewards.

[0134] Preferably, the dynamic weight m k weights the policy gradient, rather than directly weighting the reward value, and the training of the actor network is separated from the training of the multi-headed critic network. The training of the multi-headed critic network is no longer directly affected by the adjustment of the reward value, but is updated based on the original reward signal, which can ensure the accuracy and stability of the action value estimation. The training of the actor network is adjusted according to the weighted hybrid policy gradient to adapt to the learning priority of different components. This separation makes the training of the actor network and the multi-headed critic network independent of each other, without interference, thereby improving the training stability of the entire system.

[0135] Preferably, the hybrid policy gradient is used to update the parameters of the actor network and guide the behavior selection of the agent. It is worth noting that the hybrid policy gradient is dynamically updated, and each update will be based on the current dynamic weight m kRecalculate. This dynamic adjustment mechanism enables the agent to adapt to changes in the task and its own learning process, constantly optimizing its learning strategy.

[0136] The actor network and the multi-headed critic network are trained based on the hybrid policy gradient and the loss function respectively to obtain the trained actor network and the trained multi-headed critic network:

[0137] Preferably, the loss function is specifically:

[0138]

[0139] wherein, represents the expected value of the sampled state-action pair (s t ,a t ), represents the action value, θ ki represents the parameters of the multi-headed critic network, y t,k represents the target value.

[0140] Preferably, the target value wherein, r(s t ,a t ) represents the immediate reward, γ represents the discount factor, represents the minimum action value when action a t+1 is selected in the next state s t+1 , and α represents the entropy regularization coefficient, log(π φ (a t+1 |s t+1 ) represents the entropy regularization term of the policy, which encourages exploration.

[0141] Preferably, a termination condition is checked to determine whether the maximum number of training rounds has been reached: if the termination condition has not been reached, training continues; if the termination condition has been reached, the final model is saved and training ends.

[0142] The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-headed critic network to obtain the optimized gait data.

[0143] Preferably, the multi-dimensional reward function provides the target signal for learning, the multi-headed critic network ensures that these target signals can be accurately evaluated and learned, and the dynamic policy gradient ensures the rationality and efficiency of the learning process. This design not only solves the information loss problem caused by simple summation in traditional methods, but also improves the learning efficiency through priority dynamic adjustment, enabling the quadruped robot to better complete complex motion control tasks.

[0144] Preferably, based on the optimized gait data, the quadruped robot can achieve smooth straight forward motion and other preset motion modes.

[0145] Example 3

[0146] Example verification

[0147] The action space of the quadruped robot a t It consists of the joint angles of each leg. In this embodiment, each leg has three degrees of freedom, corresponding to the hip joint, thigh joint, and knee joint. Therefore, the complete action space is a 12-dimensional continuous space, with each dimension representing the control instructions for a joint. Using joint angles as actions directly controls the robot's motion.

[0148] For the control task of quadruped robots, an improved SAC algorithm is used, combined with the actor-critic framework to design a neural network structure suitable for quadruped robots.

[0149] This example compares the performance of three algorithms: SAC, Multi-head Review SAC (MH-SAC), and Multi-head Review and Dynamic Policy Gradient SAC (MHD-SAC) in the task of generating gait for a quadruped robot. To ensure fairness in the experiment, the hyperparameter settings of the three algorithms are kept consistent, as shown in Table 1:

[0150] Table 1 Hyperparameters and reward function weights of the three algorithms

[0151]

[0152]

[0153] By analyzing the reward curves, loss curves, and the curves of each reward function in the MH and MHD algorithms during training, we can comprehensively evaluate the applicability and effectiveness of these algorithms for complex control problems.

[0154] like Figure 4 As shown, from the reward curve Figure 4 As can be seen from the figure, within the first 1,500,000 rounds of training, the MHD-SAC algorithm exhibited the fastest convergence rate and the highest reward value, indicating that it achieved a better balance between exploration and exploitation, with superior learning efficiency and policy quality compared to the other two algorithms. The MH-SAC algorithm performed second, while the SAC algorithm had a relatively low convergence rate and reward value. This shows that the introduction of a multi-head structure and dynamic policy gradient can significantly improve the algorithm's performance in continuous control tasks. After 1,500,000 rounds, the reward curve flattens, but the reward values ​​of the MHD-SAC and MH-SAC algorithms remain consistently higher than those of the SAC algorithm, further confirming the superiority of the MHD method.

[0155] like Figure 5 As shown, from the strategy loss curve Figure 5As can be seen, the loss values of the MHD-SAC and MH-SAC algorithms gradually stabilize after 1,500,000 rounds, while the SAC algorithm needs about 2,000,000 rounds to reach a stable state. In addition, the strategy loss curves of the MHD-SAC and MH-SAC algorithms change faster, indicating that they can more effectively utilize experience data and accelerate the learning process of the strategy. This also shows that the multi-head structure and dynamic policy gradient play an important role in improving the learning efficiency and training stability of the algorithm.

[0156] As shown in Figure 6-7 , by observing the reward value curves of each reward part in the MH-SAC and MHD-SAC algorithms, it is found that both algorithms can achieve convergence of rewards within 2,500,000 rounds. This shows that the multi-head structure can effectively handle multi-objective optimization problems, enabling the algorithm to achieve a good balance on each sub-task.

[0157] Embodiment 4

[0158] As shown in Figure 8 , a quadruped robot gait generation system based on an improved SAC algorithm includes a data acquisition module, a gradient acquisition module, a network optimization module, and a result output module.

[0159] The data acquisition module is configured to acquire current state data and action data of the quadruped robot, obtain new state data based on the current state data, set a multi-dimensional reward function, and form tuple data with the current state data, the action data, and the new state data.

[0160] The gradient acquisition module is configured to input the tuple data into an agent composed of an actor network and a multi-head critic network to obtain a dynamic weight and an action value, and obtain a hybrid policy gradient based on the dynamic weight and the action value.

[0161] The network optimization module is configured to train the actor network and the multi-head critic network based on the hybrid policy gradient and a loss function, respectively, to obtain a trained actor network and a trained multi-head critic network.

[0162] The result output module is configured to acquire the current state data of the quadruped robot and input it into the trained actor network and the trained multi-head critic network to obtain optimized gait data.

[0163] Preferably, the functions of each functional module in the embodiment are implemented in a one-to-one correspondence with the above methods, and will not be described again.

[0164] Embodiment 5

[0165] Based on the same inventive concept, the application further provides a computer device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0166] a memory for storing a computer program;

[0167] The processor can execute the program stored in the memory to realize the gait generation method of the quadruped robot based on the improved SAC algorithm in one of embodiments 1 or 2.

[0168] The electronic device can comprise a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The processor can call the logical instructions in the memory to execute the gait generation method of the quadruped robot based on the improved SAC algorithm in one of embodiments 1 or 2.

[0169] In addition, the logical instructions in the memory described above can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

[0170] Via the technical solution, the application discloses a quadruped robot gait generation method and system based on an improved SAC algorithm, a multi-head critic network is constructed in the SAC framework, the multi-head critic network learns a value function for each sub-reward function component, instead of simply weighting and summing the reward functions in the traditional method, this design enables the system to retain the unique information of each control target and avoids the loss of information in the summation process; the application adjusts the optimization priority of different gradients through dynamic weights, improves the learning efficiency based on the priority dynamic adjustment, and enables the quadruped robot to better complete complex motion control tasks; the training of the actor network and the multi-head critic network is separated, the training of the multi-head critic network is updated based on the original reward signal, which can ensure the accuracy and stability of the action value estimation; the training of the actor network is adjusted according to the weighted mixed policy gradient, to adapt to the learning priority of different components. This separation enables the training of the actor network and the multi-head critic network to be independent of each other and not interfere with each other, thereby improving the training stability of the entire system.

[0171] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0172] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A quadruped robot gait generation method based on an improved SAC algorithm, characterized in that: include: Obtain the current state data and motion data of the quadruped robot; obtaining new state data based on the action data; Setting a multi-dimensional reward function and forming tuple data with the current state data, the action data and the new state data; The multi-dimensional reward function includes multiple sub-reward functions; Based on the tuple data, the tuple data is input into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values; The dynamic weight is specifically: Among them, m k represents the dynamic weight, K represents the number of sub-reward functions, μ k represents the reward sample mean, σ k represents the reward sample variance; The multi-head critic network includes multiple head networks with the same structure; The head network includes: a splicing module, a first hidden layer, a first activation layer, a second hidden layer and a second activation layer; The current state data and the action data in the tuple data are input into the splicing module to obtain spliced ​​data; The spliced ​​data is sequentially input into the first hidden layer, the first activation layer, the second hidden layer and the second activation layer to obtain the action value; Each of the head networks learns the corresponding action value based on the corresponding sub-reward function: Q t,ki (s t ,a t )←r tk +γ[Q t,ki (s t+1 ,a t+1 )-αlogπ(a t+1 |s t+1 )] Among them, Q t,ki (s t ,a t ) represents the action value of the k-th sub-reward function, r tk represents the kth sub-reward function, s t Indicates the current state data, a t Represents action data, s t+1 Represents the new state data, a t+1 represents the optimal action data taken under the new state data, γ represents the discount factor, α represents the entropy regularization coefficient, Q t,ki (s t+1 ,a t+1 ) represents the Q value of the next state-action pair, logπ(a t+1 |s t+1 ) means in state s t+1 Next select action a t+1 The logarithm of the probability of Obtaining a hybrid policy gradient based on the dynamic weight and the action value; The hybrid policy gradient acquisition method is: Obtaining a corresponding policy gradient based on the action value; All policy gradients are weightedly fused based on the dynamic weights to obtain the hybrid policy gradient: in, represents the partial derivative of the actor network parameter φ, π φ (a t |s t ) represents a policy function determined by parameter φ in state s t Next take action a t The probability of Indicates action a t Find the partial derivative, f φ (ε t ;s t ) represents the actor network according to the current state s t and noise ε t The generated action, Indicates that in the current state s t Select action a t The minimum Q value when training the actor network and the multi-head critic network based on the hybrid policy gradient and the loss function, respectively, to obtain a trained actor network and a trained multi-head critic network; The current state data of the quadruped robot is obtained and input into the trained actor network and the trained multi-critic network to obtain optimized gait data.

2. The method for generating gait of a quadruped robot based on the improved SAC algorithm according to claim 1, characterized in that: The sub-reward functions are: forward reward function, energy consumption reward function, expected height reward function, angle reward function between the forward direction of the robot body and the forward direction of movement, drift reward function and shaking reward function.

3. The method for generating gait of a quadruped robot based on the improved SAC algorithm according to claim 2, characterized in that: The dynamic weight acquisition method is: The tuple data is input into the agent; The agent normalizes the multi-dimensional reward function in the tuple data to a specified range and calculates a reward sample mean and a reward sample variance; The dynamic weight is obtained based on the reward sample mean and the reward sample variance.

4. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 1, characterized in that: The loss function is specifically: in, Represents the sampled state-action pair (s t ,a t ) expectations, represents the action value, θ ki represents the parameters of the multi-headed critic network, y t,k Indicates the target value.

5. The method for generating gait of a quadruped robot based on an improved SAC algorithm according to claim 1, characterized in that: The current state data includes: joint angle, joint angular velocity, joint torque and body posture.

6. A quadruped robot gait generation system based on an improved SAC algorithm, applied to a quadruped robot gait generation method based on an improved SAC algorithm as claimed in any one of claims 1 to 5, characterized in that: include: Data acquisition module, gradient acquisition module, network optimization module and result output module; The data acquisition module is used to obtain the current state data and motion data of the quadruped robot; and obtain new state data based on the current state data; Setting a multi-dimensional reward function and forming tuple data with the current state data, the action data and the new state data; The gradient acquisition module is used to input the tuple data into an intelligent agent composed of an actor network and a multi-head critic network to obtain dynamic weights and action values; and obtain a hybrid policy gradient based on the dynamic weights and the action values; The network optimization module is used to train the actor network and the multi-critic network based on the hybrid policy gradient and the loss function to obtain a trained actor network and a trained multi-critic network; The result output module is used to obtain the current state data of the quadruped robot and input it into the trained actor network and the trained multi-critic network to obtain optimized gait data.

Images