A six-legged robot training control method based on expert guidance strategy optimization

Abstract

The application discloses a kind of expert guidance strategy optimization's hexapod robot training control method, including expert guidance strategy optimization's training algorithm and end-to-end modular control network.Training algorithm is by introducing expert strategy in the initial stage of reinforcement learning training to carry out action sampling, and using time decay mechanism realizes the smooth transition from imitation learning to autonomous exploration.End-to-end modular control network includes expert strategy module, strategy network module, value network module, privileged information estimation module, terrain information coding module, terrain information estimation module, motor simulation module, these modules can be able to realize efficient training in large-scale parallel simulation environment, realize the mapping from sensor information to joint action.The application effectively solves the problem of low exploration efficiency and convergence difficulty of hexapod robot in high-dimensional action space, improves the naturalness of motion, the generalization ability to complex environment, the balancing ability of robot, and is suitable for various complex terrains.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control, and in particular to a training and control method for a hexapod robot based on expert-guided strategy optimization. Background Technology

[0002] Deep Reinforcement Learning (DRL) has made groundbreaking progress in the control of bipedal and quadrupedal robots, enabling them to perform natural, agile, and highly dynamic movement skills. Representative systems have achieved long-term autonomous traversal in mountainous environments, parkour-style obstacle crossing, and rapid maneuvering navigation. These achievements generally employ end-to-end neural networks, directly outputting the target angles or torques of all joints without requiring manual design of gait timing, significantly improving environmental adaptability and movement diversity. However, when DRL methods are extended to hexapods (18 actuation degrees of freedom), the dimensionality of the action space and system redundancy increase dramatically, leading to an exponential expansion of the exploration space. Randomized strategies are almost impossible to obtain effective rewards in early training; sample efficiency is extremely low, requiring billions of interactions to converge, which is unbearable for real robots; local optima or unnatural, uncoordinated "shaking" gaits are prone to occur, resulting in high transportation costs and significant mechanical wear. Therefore, the direct transfer performance of existing DRL algorithms on hexapods is far inferior to that on quadrupedal or bipedal platforms.

[0003] To alleviate the problem of high-dimensional action space, existing methods can be divided into two technical routes: Hybrid approaches combine traditional control modules (such as central pattern generators, trajectory optimization, or model predictive control) with neural networks, reducing the DRL task to low-dimensional gait parameter or residual moment learning. While this reduces training difficulty, the artificial structure limits the network's full utilization of the whole body's degrees of freedom, and its movement agility and terrain adaptability still lag significantly behind end-to-end quadruped systems.

[0004] Introducing prior knowledge: Expert knowledge is injected into the training process through methods such as behavior cloning, reward shaping, or dataset augmentation. A typical workflow involves pre-training using expert demonstrations followed by fine-tuning with pure RL. However, such methods generally employ multi-stage training, requiring additional design of auxiliary losses, distribution offset corrections, or gradual exit mechanisms, making implementation complex and labor-intensive. Furthermore, the distributional differences between expert policies and RL policies can easily lead to policy drift, even resulting in worse local optima than pure RL.

[0005] Whether it is a hybrid approach or an approach that incorporates prior knowledge, most of them follow the cascaded framework of "perception → planning → control". The localization / terrain perception module and the underlying control network only interact through a sparse interface. They fail to make full use of intermediate features in the perception process (such as terrain height maps and contact force latent vectors) to guide the learning of control strategies, resulting in insufficient generalization and robustness of the system in complex terrain.

[0006] The shortcomings of existing technologies can be summarized as follows: low efficiency, difficulty in early exploration, long training cycle, complex mechanism for introducing stage experts, and high engineering implementation threshold; policy distribution offset leading to policy drift, insufficient coupling between knowledge and control modules, and poor terrain adaptability. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing an expert-guided strategy-optimized training and control method for hexapod robots.

[0008] The objective of this invention is achieved through the following technical solution: a training and control method for a hexapod robot optimized by an expert-guided strategy, comprising: A standard Markov decision process is defined, which divides the robot state into observable state, privileged information, and terrain height vector; A control network is constructed, comprising: inputting observable states as input vectors to a privileged information estimation module, outputting a privileged information estimation vector; inputting a terrain height vector to a terrain information encoding module, outputting an implicit feature vector of terrain height; adding observable states to a historical observation sequence, and inputting them together with the implicit feature vector of terrain height into a long short-term memory module to output an estimated implicit feature vector of terrain height; concatenating the observable states, the estimated privileged information vector, and the estimated implicit feature vector of terrain height and inputting them into a policy network to output a policy network action; combining the output policy network action with expert actions to calculate a hybrid action, and inputting the hybrid action into a motor simulation module to obtain the torque required for each joint; The torque is input into the simulator to obtain a state reward, which is then used to update the control network. The updated policy network is deployed on the physical robot. The robot's observable state, privileged information, and terrain height implicit feature vector are input, and the action is output. The robot then executes the action and returns to the current state, forming a control loop.

[0009] Furthermore, the definition of the standard Markov decision process includes: defining the standard Markov decision process. ,in This represents the state space of a hexapod robot. This represents the motion space of a hexapod robot. This represents the probability that a hexapod robot will transition from one state to another after taking an action. Indicates the reward space, A factor representing the decay of rewards over time.

[0010] Furthermore, the privileged information estimation module input vector, terrain information encoding module, policy network module and motor simulation module in the control network are all built using fully connected neural networks, and the calculation process of the modules is based on the forward propagation algorithm of multilayer perceptron.

[0011] Furthermore, the motor simulation module is pre-trained using supervised learning before application: first, sine waves of different amplitudes and frequencies are applied to the real motor, and then the speed error, angular velocity error, and current torque of the motor are collected; then, the speed error and angular velocity error are used as the network input of the motor simulation module, and the current torque is used as the expected output of the motor simulation module; finally, the module is trained using supervised learning.

[0012] Furthermore, the step of combining the output policy network action with expert actions to calculate the hybrid action includes: ; in As the strategy decay factor, in For expert guidance time, The current time; Actions output by expert strategies For the actions of the policy network.

[0013] Furthermore, the state reward obtained by inputting torque into the simulator includes: the sum of velocity tracking reward, angular velocity tracking reward, angular velocity error penalty, vertical linear velocity error penalty, motion change penalty, joint torque penalty, foot contact force penalty, and collision penalty.

[0014] Furthermore, updating the control network includes: Computational strategy optimization objective function The overall gradient, using the learning rate Multiply by the gradient and then add the result to the parameters of the original policy network. middle; in, To optimize PPO objectives, As the strategy decay factor, To explore the attenuation factor and These are the weight coefficients for the imitation learning objective function and the entropy exploration objective function, respectively. In order to mimic the learning objective function, To explore the entropy objective function; in For expert guidance time, The current time; ,in To explore constants, Total time; ,in Let N be the expert's output action at time t, and N be the total time. The current state at time t ,in The action obtained from the policy network.

[0015] Furthermore, the PPO optimization objective is calculated using the following formula: Importance sampling ratio coefficient The advantage value function is estimated through state and action.

[0016] According to another aspect of the specification, a hexapod robot training control device optimized by an expert-guided strategy is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned hexapod robot training control method optimized by an expert-guided strategy.

[0017] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned expert-guided strategy-optimized hexapod robot training control method.

[0018] The beneficial effects of this invention are: This invention integrates expert actions and entropy exploration within a single reinforcement learning framework by constructing policy decay and exploration decay factors. Without the need for phased design and auxiliary loss, it allows the policy to smoothly transition from imitation to autonomous exploration, eliminating distribution bias and suppressing local optima and jittery gait, significantly improving training efficiency while reducing the difficulty of engineering implementation. This invention constructs an end-to-end modular network that integrates lightweight geometry experts, terrain latent feature extraction, privileged state estimation, bidirectional temporal memory, and motor dynamics simulation. Perception and control are jointly optimized in the same gradient flow, breaking the traditional cascaded sparse interface and improving the flexibility of hexapod robot movement and its adaptability to terrain. Attached Figure Description

[0019] Figure 1 This is an end-to-end network framework diagram provided in an embodiment of the present invention; Figure 2 A training reward comparison chart provided for embodiments of the present invention; Figure 3 This invention provides a simulation of a hexapod robot traversing complex terrain and visualizing its foot trajectory in an embodiment of the invention. Figure 4 Speed ​​tracking error diagram provided for embodiments of the present invention; Figure 5 Figures showing the actual experimental results provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings.

[0022] The present invention provides a training and control system for a hexapod robot with expert-guided strategy optimization, comprising two parts: an expert-guided strategy optimization reinforcement learning training algorithm and an end-to-end modular control network. The reinforcement learning training algorithm optimized by expert-guided strategies includes the following steps: Define a standard Markov decision process ,in This represents the state space of a hexapod robot. This represents the motion space of a hexapod robot. This represents the probability that a hexapod robot will transition from one state to another after taking an action. Indicates the reward space, This refers to a factor that indicates the decay of rewards over time. The discount factor... Used to calculate the decay of future rewards, and is used in the calculation of cumulative rewards in the dominance function.

[0023] definition For the hexapod robot's strategy, let represent the robot's state. Below, the mean of the strategy output is variance is The Gaussian distribution is used to obtain the actions of the policy network by sampling this Gaussian distribution. .

[0024] definition For expert strategies, it means that the robot is in a state. Below, the expert strategy will output an action. .

[0025] Constructing a strategy decay factor ,in For expert guidance time, This is the current time.

[0026] Constructing an exploration decay factor ,in To explore constants, This represents the total time.

[0027] Define blend actions The actual actions performed by the hexapod robot are calculated using formula (1): (1) Construct the policy optimization objective function and calculate it using formula (2): (2) in The optimization objective for PPO (Proximal Policy Optimization) is calculated using formula (3): (3) in In order to mimic the learning objective function, To explore the entropy objective function, and These are the weight coefficients for the imitation learning objective function and the entropy exploration objective function, respectively. In formula (3)... The importance sampling ratio coefficient is calculated using formula (4): (4) In formula (3) The advantage value function is calculated using the following formula: in Indicates the strategy Calculate the mathematical expectation. These represent the initial state and the initial action, respectively.

[0028] The process of constructing the EGPO training algorithm is shown below. need: 1: 2:for t=0 to N do 3: 4: 5: 6: 7: 8: 9: End the for loop Specifically, the control networks trained and deployed using expert-guided strategy optimization methods, such as... Figure 1 As shown. The training and deployment of expert-guided strategy optimization includes the following three parts: 1) building modules; 2) training using the EGPO algorithm; 3) deploying the training results on the robot. The specific implementation steps are as follows: Modules The construction modules include: expert strategy module, privileged information estimation module, terrain information coding module, long short memory coding module, policy network module, value network module, and motor simulation module.

[0029] The implementation steps are as follows: The expert strategy module receives partial state observations of the robot, including the current motor position and the current speed command. Output the desired positions of all joint motors; these desired positions constitute the expert motion. The specific calculation method is as follows: Define the current foot position as... ,in The system represents the robot's coordinate system, with its origin fixed at the robot's geometric center. The x-axis and y-axis are parallel to the robot's forward and lateral movement directions; this defines the robot's coordinate system... From the original time The system has been moved. During this process, the foot remains unchanged in the world coordinate system, and the trajectory of movement in the robot coordinate system can be calculated according to formula (5): (5) in This represents the rotation matrix about the z-axis. Representing the robot's feet at the end The position of the joints; the desired position of all joints can be obtained through inverse kinematics calculations.

[0030] Privilege Information Estimation Module Terrain information coding module The policy network module, value network module, and motor simulation module are all built using fully connected neural networks. The computation process of these modules is based on the forward propagation algorithm of a multilayer perceptron. The specific computation steps are as follows: First, the original input data for each module is normalized and transformed into input feature vectors. Secondly, the input vector It is fed into a hidden layer containing at least two layers. The calculation formula for each layer is as follows: (6) in, This is the learnable weight matrix for this layer. For bias vectors, This is the ELU activation function. The specific expression for the ELU function is: (7) in The parameters are fixed values. The output layer does not use an activation function, but instead... Calculated.

[0031] Privilege Information Estimation Module Input measurable state The estimated value of the unmeasurable state is calculated using the multilayer perceptron forward propagation algorithm described above. Including base speed Contact force between the foot and the ground Components of gravitational acceleration ; Terrain information encoding module The input is a vector consisting of the terrain heights within a rectangular grid area surrounding the robot. The implicit feature vector of the output terrain height vector is calculated using the aforementioned multilayer perceptron forward propagation algorithm. ; The policy network module inputs measurable states. Estimates of unmeasurable states Estimated values ​​of terrain feature vectors Output the expected values ​​of all joint angles. ; The Long Short-Term Memory (LSTM) module maintains a fixed-length sequence of historical observations. When a new state is observed, this module removes the oldest state from the sequence and adds the latest state; it uses a bidirectional long short memory module to encode historical information and then computes the output through a fully connected layer. The dimensions are the same as those in the terrain information module.

[0032] The parameters of all the above modules are jointly updated through backpropagation of the gradient of the policy optimization objective function.

[0033] The motor simulation module aims to make the characteristics of the simulated motor as similar as possible to the real one, thereby reducing the discrepancy between simulation and physical implementation. The module is trained using supervised learning. First, sinusoidal waves of different amplitudes and frequencies are applied to the real motor. Then, the motor's speed error, angular velocity error, and current torque are collected. Next, the speed and angular velocity errors are used as the network input to the motor simulation module, and the current torque is used as the desired output. Finally, supervised learning is used to train the module.

[0034] 2) Training using the EGPO algorithm The first step in training the algorithm is to reset the environment. Specifically, this involves setting the robot's initial position in the simulation environment to the origin of the world coordinate system, setting the robot's initial posture to be parallel to the horizontal ground, and setting the actions the robot will perform. Perform the action in the simulation, and then collect the state. The second step is to start a loop from t=0 to N times.

[0035] The third step is to calculate the state at the current time step t. Actions of the policy network The specific calculation is as follows: Figure 1 : State Divided into observable states Privilege information and terrain height vector ; observable state The privileged information estimation module receives the input vector as input. In the middle, the privileged information estimation vector is output. ; Transform the terrain height vector Input to terrain information encoding module Output the implicit feature vector of terrain height. ; observable state Add to historical observation sequence Then, the historical observation sequence The input is fed into the Long Short-Term Memory (LSTM) module, and the output is the estimated implicit feature vector of terrain height. ; observable state Estimating privileged information vector and estimating implicit feature vectors of terrain height The concatenated input is then fed into the policy network, and the output is the policy network action. .

[0036] The fourth step is to calculate the state at the current time step t. The actions of the experts .

[0037] The fifth step is to calculate the mixed actions, such as... Figure 1Calculate the mixed motion using Formula 1 .

[0038] The sixth step is to calculate the mixed motion. implement After a certain time, the environment will provide feedback on the status reward and whether the process has ended at the next time step. The specific calculations are as follows: Figure 1 Use mixed actions The input is fed into the motor simulation module, which outputs the torque required for each joint. The simulator will transmit torque Apply it to the robot; then calculate the reward and penalty the robot receives. Each of them The calculation method is as follows: Speed ​​Tracking Rewards: ,in Represents the velocity command in the observable state. Represents the robot's current actual speed. Represents the speed tracking sensitivity coefficient; Angular velocity tracking reward: ,in This represents the angular velocity command in the observable state. This represents the robot's current angular velocity around the z-axis. Represents the sensitivity coefficient for angular velocity tracking; Angular velocity error penalty: ,in , These represent the robot's current angular velocities around the x-axis and y-axis, respectively. Vertical linear velocity error penalty: ,in This represents the robot's current velocity along the z-axis.

[0039] Action change penalty: ,in These represent the actions output by the policy network at the current and previous time steps, respectively. Joint torque penalty: ,in It is a component of the torque vector output by each motor simulation module. Foot contact force penalty: ,in The magnitude of the actual contact force at the foot. The maximum permissible contact force.

[0040] Collision penalty: ,in This indicates the magnitude of the external force acting on the robot's other parts besides the feet. This represents a function that takes the value 1 if the condition within the parentheses is true, and takes the value 0 if the condition is false.

[0041] The seventh step is to use the aforementioned reward signals to calculate the cumulative return and further calculate the dominance function. The advantage function is used for gradient updates of the policy optimization objective.

[0042] The eighth step is to calculate the gradients separately. , , Then according to the formula Calculate the overall gradient , The ninth step is to use the learning rate. With gradient Multiply the results and then add the result to the parameters of the original policy network. middle.

[0043] During this loop, the rewards earned by the robot are recorded. Simultaneously, the robot is trained using other methods with the same number of loops, and its rewards are recorded again. Figure 2 Specifically, the reward curve obtained by the algorithm proposed in this invention for training the robot corresponds to the EGPO curve, the expert module obtains the Expert reward curve, the reward obtained by combining imitation learning and reinforcement learning corresponds to the BC+PPO curve, and the reward obtained by training using reinforcement learning corresponds to the PPO curve. The method proposed in this invention ultimately achieves the highest reward.

[0044] 3) Deploy the training results on the robot Specifically, the strategy network and privileged information estimation module will be used. And save the parameters of the long and short memory modules and re-accord according to Figure 1 The connection is established during the deployment phase. Specifically, observable states are obtained from the robot. Then, they are input into the privileged information estimation module output respectively. and long and short memory module output Then, these three vectors are concatenated and input into the policy network to output the action. Then the robot performs the action and returns to the current state. It forms a control loop.

[0045] Before deployment on a physical robot, simulations are performed. Specifically, such as... Figure 3 By controlling the robot to follow speed commands on three different terrains—flat ground, stairs, and slopes—and simultaneously plotting the position of the robot's feet, it can be observed that the robot can walk on these three terrains. Figure 4We can observe the effect of the robot following the two speed command curves cmd:vx and cmd:vy. The robot's current real speed is the curve real:vx and real:vy. We can observe that the robot can control the speed tracking error within 0.02m / s.

[0046] Then the policy network and privileged information estimation module will be used. The long and short-term memory modules are deployed on the physical robot, which is then used to control the robot's movement on stairs, ramps, and gravel paths. For example... Figure 5 The robot can climb steps with a height of 10cm, move on lawns with a slope of 25°, and on mountain gravel roads with a maximum slope of 35° and a maximum drop of 6m.

[0047] Corresponding to the aforementioned embodiment of a hexapod robot training control method optimized by an expert-guided strategy, the present invention also provides an embodiment of a hexapod robot training control device optimized by an expert-guided strategy.

[0048] See Figure 6 The present invention provides a hexapod robot training control device with expert-guided strategy optimization, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a hexapod robot training control method with expert-guided strategy optimization as described in the above embodiment.

[0049] The embodiment of the expert-guided strategy optimization training control device for a six-legged robot provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an expert-guided strategy optimization hexapod robot training control device provided by the present invention. (Except for...) Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0050] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0051] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0052] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a hexapod robot training and control method optimized by an expert-guided strategy as described in the above embodiments.

[0053] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0054] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned expert-guided strategy optimization method for training and controlling a six-legged robot.

[0055] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training and control method for a hexapod robot optimized by an expert-guided strategy, characterized in that, include: A standard Markov decision process is defined, which divides the robot state into observable state, privileged information, and terrain height vector; A control network is constructed, comprising: inputting observable states as input vectors to a privileged information estimation module, outputting a privileged information estimation vector; inputting a terrain height vector to a terrain information encoding module, outputting an implicit feature vector of terrain height; adding observable states to a historical observation sequence, and inputting them together with the implicit feature vector of terrain height into a long short-term memory module to output an estimated implicit feature vector of terrain height; concatenating the observable states, the estimated privileged information vector, and the estimated implicit feature vector of terrain height and inputting them into a policy network to output a policy network action; combining the output policy network action with expert actions to calculate a hybrid action, and inputting the hybrid action into a motor simulation module to obtain the torque required for each joint; The torque is input into the simulator to obtain a state reward, which is then used to update the control network. The updated policy network is deployed on the physical robot. The robot's observable state, privileged information, and terrain height implicit feature vector are input, and the action is output. The robot then executes the action and returns to the current state, forming a control loop.

2. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The definition of the standard Markov decision process includes: defining the standard Markov decision process. ,in This represents the state space of a hexapod robot. This represents the motion space of a hexapod robot. This represents the probability that a hexapod robot will transition from one state to another after taking an action. Indicates the reward space, A factor representing the decay of rewards over time.

3. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The privileged information estimation module input vector, terrain information encoding module, policy network module and motor simulation module in the control network are all built using fully connected neural networks, and the calculation process of the modules is based on the forward propagation algorithm of multilayer perceptron.

4. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The motor simulation module is pre-trained using supervised learning before application: First, sine waves of different amplitudes and frequencies are applied to the real motor, and then the motor's speed error, angular velocity error, and current torque are collected; then, the speed error and angular velocity error are used as the network input of the motor simulation module, and the current torque is used as the expected output of the motor simulation module; finally, supervised learning is used to train the module.

5. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The process of combining the output policy network action with expert action to calculate the hybrid action includes: ； in As the strategy decay factor, in For expert guidance time, The current time; Actions output by expert strategies For the actions of the policy network.

6. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The state reward obtained by inputting torque into the simulator includes: a sum of velocity tracking reward, angular velocity tracking reward, angular velocity error penalty, vertical linear velocity error penalty, motion change penalty, joint torque penalty, foot contact force penalty, and collision penalty. This state reward is used to calculate... The advantage value function is then used for gradient updates of the policy optimization objective.

7. The expert-guided strategy optimization training control method for a hexapod robot according to claim 1, characterized in that, The update of the control network includes: Computational strategy optimization objective function The overall gradient, using the learning rate Multiply by the gradient and then add the result to the parameters of the original policy network. middle; in, To optimize the PPO target, As the strategy decay factor, To explore the attenuation factor and These are the weight coefficients for the imitation learning objective function and the entropy exploration objective function, respectively. In order to mimic the learning objective function, To explore the entropy objective function; in For expert guidance time, The current time; ,in To explore constants, Total time; ,in Let N be the expert's output action at time t, and N be the total time. The current state at time t ,in The action obtained from the policy network.

8. The expert-guided strategy optimization training control method for a hexapod robot according to claim 7, characterized in that, The PPO optimization objective is calculated using the following formula: Importance sampling ratio coefficient The advantage value function is estimated through state and action.

9. A training control device for a hexapod robot optimized by an expert-guided strategy, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a hexapod robot training control method optimized by an expert-guided strategy as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a training and control method for a hexapod robot with expert-guided strategy optimization as described in any one of claims 1-8.

Images