A hydraulic foot type robot jumping control method and system based on reinforcement learning

Abstract

The present application relates to the field of foot type robot control, and particularly relates to a hydraulic foot type robot jump control method and system based on reinforcement learning, comprising: establishing a hydraulic foot type robot bottom layer control framework; determining an action space and an observation space, and constructing a reinforcement learning optimization control model; determining a reward function and a domain randomization parameter, and training the reinforcement learning model in a training environment until convergence; and deploying the trained strategy network model to a robot prototype to optimize control parameters in real time. The embodiment of the present application realizes dynamic optimization of multiple control parameters with coupling relationship by constructing a hydraulic foot type robot limb leg unit continuous jump control method based on reinforcement learning, and improves jump height tracking accuracy, landing flexibility of the foot end and control robustness.

Description

Technical Field

[0001] This invention belongs to the field of legged robot control, specifically a method and system for jumping control of hydraulic legged robots based on deep reinforcement learning. Background Technology

[0002] With the rapid development of robotics technology, hydraulic legged robots, with their high power density and large load capacity, hold significant strategic importance in fields such as field inspection, material transportation, and rescue. In special missions, legged robots often need to perform stable and continuous jumps. During this process, the leg units frequently switch between swinging and supporting phases, generating substantial ground contact forces. Jumping motion not only reflects the maneuverability of hydraulic legged robots under high-dynamic conditions but also directly relates to their obstacle-crossing and rapid movement performance. The strong nonlinear characteristics of the hydraulic system further increase the complexity of jump motion control methods. Therefore, the control performance of jumping motion fully demonstrates the advanced nature of hydraulic legged robot motion control technology.

[0003] Traditional methods for controlling the jumping motion of legged robots mainly improve jumping performance by optimizing the foot trajectory generator or motion tracking controller. For high-dynamic motion tasks such as continuous jumping, the parameters of the trajectory generator and motion tracking controller jointly affect the system performance and have a complex coupling relationship. If they are designed independently, they often lead to suboptimal parameter combinations. Traditional joint optimization methods (such as genetic algorithms) also face the problem of difficulty in achieving multi-objective optimization and poor control robustness.

[0004] Reinforcement learning, benefiting from its powerful learning capabilities and training mechanisms, enables control strategies trained in parallel stochastic environments to adaptively output control parameters based on environmental changes. These strategies can be directly transferred to physical robots, providing solutions for the dynamic optimization of multi-parameter, multi-objective systems in complex coupled systems within the field of robot control. Therefore, this invention addresses the problems of dynamic optimization of coupled control parameters and nonlinear control of the hydraulic system in the jumping motion of hydraulically legged robots. It proposes a reinforcement learning-based jumping control method for hydraulically legged robots, which improves performance in multiple aspects, including jump trajectory tracking and landing compliance, while achieving dynamic optimization of coupled controller parameters. Summary of the Invention

[0005] The purpose of this invention is to provide a reinforcement learning-based jumping control method for hydraulic legged robots, in order to solve problems such as dynamic optimization of coupled control parameters and nonlinear control of hydraulic systems in the motion control of hydraulic legged robots.

[0006] To achieve the above objectives, the specific content of the technical solution of the present invention is as follows: A method for controlling the jumping of a hydraulically legged robot based on reinforcement learning, the method comprising:

[0007] (1) Based on the motion mode of the leg unit of the hydraulic leg robot, generate the foot trajectory and perform motion tracking;

[0008] (2) Determine the action space and observation space, and construct a reinforcement learning optimization control model. The observation space is the actual motion angle and expected motion angle of each joint of the robot's leg unit at each time step, the real-time position of the robot body in the world coordinate system, and the real-time contact force between the foot and the ground. The action space for generating the foot trajectory is the position of the foot before the jump, the time of reaching the highest point, and the speed at the highest point.

[0009] (3) Determine the reward function and domain randomization parameters, and train the reinforcement learning optimization control model to convergence in the training environment;

[0010] (4) Deploy the trained policy network model to the robot and optimize the control parameters in real time.

[0011] Furthermore, step (1) also includes: establishing a kinematic and dynamic model of the leg unit of the hydraulic legged robot;

[0012] Design a foot trajectory generator module for limb and leg units;

[0013] Design a limb and leg unit motion tracker module;

[0014] A dynamic model of the valve-controlled hydraulic system is established, and joint force control is achieved by combining physical constraints.

[0015] Furthermore, the design of the limb unit foot trajectory generator module and motion tracker module includes:

[0016] The foot motion trajectory was designed using a fifth-order polynomial, and the equations were obtained by substituting the initial and final displacements, velocities, and accelerations of the trajectory motion.

[0017] The motion tracker module was designed, and a PD controller with dynamic model compensation was selected as the motion controller for the limb unit. The trajectory of the foot was controlled by calculating the contact force at the foot end.

[0018] Furthermore, a dynamic model of the valve-controlled hydraulic system is established, and joint force control is achieved by combining physical constraints, including:

[0019] In a simulation environment, physical feasibility constraints are imposed on the valve-controlled hydraulic system of the joint hydraulic actuator, limiting the valve flow rate, control current, hydraulic cylinder stroke, and joint friction torque to within the actual range of the hydraulic legged robot.

[0020] Furthermore, determining the motion space in step (2) also includes: for the motion tracker, selecting the stiffness coefficient and damping system as motion output parameters.

[0021] Furthermore, constructing a reinforcement learning neural network model includes:

[0022] The reinforcement learning algorithm uses the Proximal Policy Optimization (PPO) algorithm. The initial neural network model includes a policy network and a value function network. The policy network is used to generate an action probability distribution, and the value function network is used to evaluate the generated actions to obtain state values.

[0023] Furthermore, determining the reward function and domain randomization parameters, and training the reinforcement learning model to convergence in the training environment includes:

[0024] The simulation training environment is set up according to domain randomization;

[0025] Start training with a set number of rounds, and continuously update the strategy network based on rewards;

[0026] Determine if the reward converges;

[0027] If it is determined that the total reward has not converged, the weight coefficient of each reward is redesigned based on the individual reward details.

[0028] Training is completed once the total reward value is determined to converge stably.

[0029] Furthermore, the reward function includes a reward for tracking the target height, a penalty for excessive landing contact force, a penalty for joint angle tracking error, and a reward for completing the task objective.

[0030] Furthermore, the simulation training environment is set up according to domain randomization, including:

[0031] Determine the physical parameters involved in domain randomization and their threshold ranges; based on the threshold ranges, generate and select corresponding physical randomization parameters through a random function; determine the type of environmental noise in the training environment, and determine the environmental noise in the training environment according to the preset noise parameters, the correspondence between the type of environmental noise and the sensor position.

[0032] On the other hand, the present invention provides a reinforcement learning-based hydraulic legged robot control system, the system including a processor configured to execute the reinforcement learning-based hydraulic legged robot control method as described above.

[0033] The beneficial effects of this invention are:

[0034] 1. This invention constructs a low-level motion control framework that incorporates physical feasibility constraints, accurately reflects the performance of real hydraulic drive systems, effectively solves the nonlinear control problem of hydraulic drive systems, and provides a foundation for the simulation training and deployment of reinforcement learning.

[0035] 2. This invention solves the problem of dynamic optimization of coupled control parameters by constructing a reinforcement learning model that simultaneously optimizes the trajectory generator and the motion tracker. Compared with fixed-parameter optimization methods, the method described in this invention can adaptively optimize the output parameters based on real-time state information, exhibiting stronger robustness to environmental noise and thus improving system stability.

[0036] 3. This invention achieves significantly improved multi-objective motion optimization control. It dynamically optimizes continuous jumping tasks for hydraulically legged robots by addressing multiple optimization objectives, such as jump height tracking and landing compliance. Compared to traditional fixed-parameter optimization methods, the proposed method significantly improves performance in jump height error, landing compliance, and motion tracking error.

[0037] 4. This invention constructs a complete method for training and deploying control policies based on reinforcement learning. Its training mechanism can be dynamically adjusted according to the requirements of the target task, and the policy network model obtained after training can be directly deployed to a real prototype. This invention improves the deployment success rate and control robustness through domain randomization, and has good transferability and scalability.

[0038] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 A flowchart illustrating a reinforcement learning-based continuous jumping control method for a hydraulic legged robot leg unit according to an embodiment of the present invention.

[0041] Figure 2 A flowchart illustrating the establishment of the underlying control framework for a hydraulic legged robot according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a hydraulic legged robot leg unit according to an embodiment of the present invention;

[0043] Figure 4 This is a flowchart illustrating the training of a reinforcement learning model for a jumping task in a randomized simulation environment according to an embodiment of the present invention.

[0044] Figure 5 This is a framework diagram illustrating the deployment of a trained reinforcement learning control strategy to a real-world prototype according to one embodiment of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0046] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0047] like Figure 1 The diagram shows a flowchart of a reinforcement learning-based continuous jumping control method for a leg unit of a hydraulically powered legged robot according to an embodiment of the present invention. Figure 1 In this calculation method, the steps may include:

[0048] In step S10, the underlying control framework of the hydraulic legged robot is established;

[0049] In step S11, the action space and observation space are determined, and a reinforcement learning optimization control model is constructed;

[0050] In step S12, the reward function and domain randomization parameters are determined, and the reinforcement learning model is trained to convergence in the training environment;

[0051] In step S13, the trained policy network model is deployed to the robot prototype to optimize control parameters in real time.

[0052] In such Figure 1 In the reinforcement learning-based continuous jumping control method for the leg units of a hydraulically legged robot, step S10 is used to establish the underlying control framework of the hydraulically legged robot. In this embodiment, in one example of the present invention, step S10 includes the following: Figure 2 The steps shown. Figure 2 In this context, step S10 may include:

[0053] In step S20, a kinematic and dynamic model of the leg unit of the hydraulic legged robot is established;

[0054] In step S21, a foot trajectory generator module for the limb and leg unit is designed;

[0055] In step S22, a limb unit motion tracker module is designed;

[0056] In step S23, a dynamic model of the valve-controlled hydraulic system is established, and joint force control is achieved by combining physical constraints.

[0057] In such Figure 2 In the method shown, step S20 is used to establish the kinematic and dynamic model of the leg unit of the hydraulic legged robot. Since the jumping motion in this example only involves the hip and knee joints in the leg unit, and the robot's lateral joints are not considered, the leg unit motion is a motion within a single plane. Specifically, the leg unit structure is as follows: Figure 3 As shown, the structural parameter L 1= =350mm, L 2= =350mm, L 1a= =45mm, L 2a= =45mm, L 1b= =228mm, L 2b= =228mm, X L1 =201~261mm, X L2 =201~261mm.

[0058] The kinematic model of the leg unit of the hydraulic legged robot was established using the DH parameter method;

[0059] (1)

[0060] Where x and y represent the positions of the foot end in the coordinate system of the limb unit base. For hip joint angle; This refers to the knee joint angle; , This refers to the length of the thigh and calf.

[0061] Based on geometric relationships, a formula for calculating the inverse kinematics of a single leg is established, yielding the relationship between the motion angles of the knee and hip joints and structural parameters:

[0062] (2)

[0063] A dynamic model of the limb unit is established using the Lagrangian method to obtain the desired torques of the robot's knee and hip joints:

[0064] (3)

[0065] in, For the joint generalized torque vector, For mass inertia, The vectors of the Coriolis force and the centrifugal force. Let q be the gravitational torque vector, and q be the torque vector in formula (2). and The matrix formed.

[0066] Step S21 is used to design the foot trajectory generator module for the limb and leg unit. In this embodiment, the specific method for designing the foot trajectory generator for the limb and leg unit in step S21 can be of various forms known to those skilled in the art. In one example of the present invention, step S21 may include:

[0067] A fifth-order polynomial is used to design the foot motion trajectory. Since the required motion is a vertical jumping motion, only the foot motion trajectory in the y-axis direction is designed. Substituting the initial and final displacements, velocities, and accelerations of the trajectory motion, a system of six equations is obtained, which can be written in the following matrix form:

[0068] (4)

[0069] in, Represents the initial time; Indicates the end time; Represents the initial displacement; This represents the end of the displacement; Represents the initial velocity; Represents the final speed; Represents the initial acceleration; Represents the end of acceleration. These are the coefficients of a fifth-degree polynomial.

[0070] Step S22 is used to design the limb and leg unit motion tracker module. In this embodiment, the specific method for designing the limb and leg unit motion tracker in step S22 can be of various forms known to those skilled in the art. In one example of the present invention, step S22 may include:

[0071] A PD controller with dynamic model compensation is selected as the motion controller for the limb unit. The foot trajectory is controlled by calculating the foot contact force. The calculation method for the foot contact force is as follows:

[0072] (5)

[0073] in, The actual position of the foot end calculated from the forward kinematics of the representative limb unit; This represents the foot position in the desired trajectory; and The actual velocity and the expected velocity of the foot tip; and is the gain coefficient. Combining formula (3), the output torque required by the two joints of the limb unit to complete the target motion trajectory can be obtained:

[0074] (6)

[0075] Among them, the joint angular velocity in the dynamic model of the limb unit is taken into consideration. With angular acceleration Since it is difficult to obtain data using sensors, the inertial term in the limb unit dynamics model is ignored in formula (6) of this example. And centrifugal force, Coriolis force term Considering only the gravity term The impact.

[0076] Step S23 is used to establish a dynamic model of the valve-controlled hydraulic system and realize joint force control by combining physical constraints. In one example of the present invention, step S23 may include:

[0077] Establish the relationship between output force and hydraulic actuator:

[0078] (7)

[0079] in, The output force representing the joint is the target of control; and This represents the pressure in the two chambers of the hydraulic cylinder; and This represents the actual working area of ​​the two chambers of the hydraulic cylinder. This represents the output displacement of the hydraulic cylinder; This represents the output speed of the hydraulic cylinder; The coefficient representing the combination of modeling damping and viscous friction; Represents Coulomb friction; This represents concentrated modeling errors in dynamics, including uncertain nonlinearities caused by external disturbances, unmodeled friction, and other terms that are difficult to model.

[0080] Combining hydraulic dynamics formulas with servo valve characteristics, the dynamic equation for joint force control is obtained:

[0081] (8)

[0082] in, The bulk modulus of hydraulic oil; The flow gain coefficient representing the servo valve ( Represents flow coefficient, Represents the valve area gradient, Represents the positive gain coefficient, (Represents oil density); and Represents the actual volumes of the rodless and rod-type chambers of a hydraulic cylinder; Coulomb friction can be expressed as... , where amplitude It is an unknown, continuous shape function. It is known.

[0083] In a simulation environment, physical feasibility constraints are imposed on the valve-controlled hydraulic system of the joint hydraulic actuator, limiting the valve flow rate, control current, hydraulic cylinder stroke, and joint friction torque to the actual range of the real prototype.

[0084] Step S11 is used to determine the action space and observation space, and to construct a reinforcement learning optimization control model. The specific method for constructing the reinforcement learning model in step S11 can be of various forms known to those skilled in the art. In one example of this invention, step S11 may include:

[0085] The system acquires observation information from the environment at each step, including the observation space and the motion space. Specifically, the observation space includes the actual and desired motion angles of each joint of the robot's leg units at each time step t, the real-time position of the robot body in the world coordinate system, and the real-time contact force between the foot and the ground. The motion space comprises the key control parameters of the robot trajectory generator and motion tracker. For the trajectory generator, the position of the foot before takeoff is selected. Time to reach the highest point and the speed at the highest point As an output parameter for motion; for a motion tracker, the stiffness coefficient is selected. and damping coefficient As a parameter for motion output, the total motion space And based on engineering experience, the output range of each parameter is limited.

[0086] A reinforcement learning network structure is established, taking observation information as input and robot control parameters as output. To avoid low training efficiency due to excessively large update magnitudes, the deep reinforcement learning model uses the proximal policy optimization (PPO) algorithm. Its objective function is as follows: (9)

[0087] in, For policy network parameters, It is the ratio of the new strategy to the old strategy. is a hyperparameter that limits the update magnitude, where s represents the observed state and a represents the generated action. Both the policy network and the value network consist of two fully connected layers, containing 256 and 128 neurons respectively, with ReLU activation function.

[0088] Step S12 is used to train the reinforcement learning model for the jumping task in a randomized simulation environment. In an example of the present invention, step S12 may include, for example... Figure 4 The steps shown are described in this. Figure 4 In this context, step S12 may include:

[0089] In step S30, the simulation training environment is set according to domain randomization;

[0090] In step S31, training begins with a set number of rounds, and the policy network is continuously updated based on the rewards.

[0091] In step S32, it is determined whether the reward has converged;

[0092] In step S33, if it is determined that the total reward has not converged, the weight coefficient of each reward is redesigned according to the reward situation.

[0093] Training is completed once the total reward value is determined to converge stably.

[0094] Step S30 is used to set the simulation training environment according to domain randomization. In this embodiment, the model established in steps S11 and S10 is run in Matlab / Simulink. ® Implemented in a reinforcement learning environment. To address the delay response characteristics of the hydraulic drive system, a random delay ranging from [5,10] ms is introduced into the desired force output of the training environment; a random force opposite to the direction of motion is applied to the body to simulate uncertainties such as friction in the actual environment; the ground friction coefficient is randomized to a range of [0.4,0.7] before the start of each training round; and Gaussian noise with a standard deviation of 0.05 is added to the observation space to simulate the actual measurement error of the sensor.

[0095] Step S31 is used to begin training with a set number of rounds, continuously updating the policy network based on the rewards. The training objective of the agent is to maximize the cumulative expected reward value, which can be represented as follows:

[0096] (10)

[0097] in, As the current state, For the actions currently being taken, Indicates the index of the time step. This represents the discount factor, used to determine the weight of future rewards. In this example, .

[0098] A reward mechanism is established for the continuous jumping task of the leg unit of the hydraulic legged robot. The reward function is designed with the goals of approaching the target jump height, enhancing landing compliance, and reducing tracking error. Specifically, it includes:

[0099] Target Height: In each jump, the robot is encouraged to reach a preset target height, while falling below or exceeding the target height will result in penalties.

[0100] (11)

[0101] in, Represents the predefined target jump height of the robot. This represents the maximum height of the fuselage during each jump.

[0102] Landing Contact Force: To reduce the large impact force generated when the robot lands, a contact force index is introduced. To encourage robots to improve landing compliance, smaller The value indicates better landing compliance.

[0103] (12)

[0104] in, and These represent the maximum contact force between the foot and the ground during takeoff and landing, respectively.

[0105] Tracking error: The error between the robot's actual trajectory and the desired trajectory.

[0106] (13)

[0107] in, and These represent the angular tracking errors of the hip and knee joints, respectively. and These are the desired rotation angles for the hip and knee joints, respectively.

[0108] Task Reward: If the difference between the robot's single jump height and the target height is less than the target range, the task is considered completed, and a high one-time reward value is given.

[0109] (14)

[0110] in, This represents the error between the actual jump height and the target height. This represents the target error range. In summary, the total reward value expression for the robot's constant-height jump task is:

[0111] (15)

[0112] Among them, the weighting coefficient of each reward The weights can be dynamically adjusted based on the actual goals and training progress. In this example, the weight coefficients are set as follows:

[0113] (16)

[0114] Step S32 is used to determine whether the reward has converged. Step S33: If the total reward has not converged, the weight coefficient of each reward is redesigned according to the individual reward conditions; if the total reward value has converged stably, the training is completed to obtain a control strategy that can be deployed.

[0115] Step S13 is used to deploy the trained policy network model to the robot prototype and optimize the control parameters in real time.

[0116] In this example, the control strategy, trained for 5000 rounds, was deployed to a real-world limb prototype identical to the simulation model. Under a pressure of 15 MPa, it performed three consecutive jumps to a target height of 0.68 m. The deployment process is as follows: Figure 5 As shown. Experimental results measured by sensors: the average error in distance to target height during the three jumps was 0.758%, the maximum error in a single jump was 1.210%, and the contact force index... The average error in the distance to the target height after three jumps, obtained using the traditional fixed-parameter optimization method, was 1.914%, with a maximum error of 2.360% for a single jump, and the contact force index... Therefore, in this example, the proposed reinforcement learning-based continuous jump control method for the leg units of a hydraulic legged robot achieves dynamic optimization of selected control parameters for multiple target performances such as jump height and landing compliance. Compared with the fixed parameter optimization control method, it has significant improvements in jump height error, landing compliance, and motion tracking error.

[0117] On the other hand, the present invention provides a reinforcement learning-based control system for a hydraulically legged robot, the system including a processor configured to perform the methods described above.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for controlling the jumping of a hydraulically legged robot based on reinforcement learning, characterized in that, The method includes: (1) Based on the motion mode of the leg unit of the hydraulic leg robot, generate the foot trajectory and perform motion tracking; (2) Determine the action space and observation space, and construct a reinforcement learning optimization control model. The observation space consists of the actual and expected motion angles of each joint of the robot's leg unit at each time step, the real-time position of the robot body in the world coordinate system, and the real-time contact force between the foot and the ground. The action space for generating the foot trajectory is the key control parameter for the robot trajectory generator and motion tracker. For the trajectory generator, the position of the foot before takeoff is selected. Time to reach the highest point and the speed at the highest point As an output parameter for motion; for a motion tracker, the stiffness coefficient is selected. and damping coefficient As a parameter for motion output, the total motion space And limit the output range of each parameter based on engineering experience; (3) Determine the reward function and domain randomization parameters, and train the reinforcement learning optimization control model to convergence in the training environment; specifically: establish a reward mechanism for the continuous jumping task of the hydraulic legged robot's leg unit. The reward function is designed with the goal of approaching the target jump height, enhancing landing compliance, and reducing tracking error, specifically including: Target Height: In each jump, the robot is encouraged to reach a preset target height, while falling below or exceeding the target height will result in penalties. in, Represents the predefined target jump height of the robot. This represents the maximum height of the fuselage during each jump; Landing Contact Force: To reduce the large impact force generated when the robot lands, a contact force index is introduced. To encourage robots to improve landing compliance, smaller The value indicates better landing compliance; in, and These represent the maximum contact force between the feet and the ground during takeoff and landing, respectively. Tracking error: The error between the robot's actual trajectory and the desired trajectory. in, and These represent the angular tracking errors of the hip and knee joints, respectively. and These are the expected rotation angles of the hip and knee joints, respectively. For hip joint angle; For knee joint angle; in the reward , , These are the weighting coefficients; Task Reward: If the difference between the robot's single jump height and the target height is less than the target range, the task is considered completed, and a high one-time reward value is given. in, This represents the error between the actual jump height and the target height. This is the target error range; in summary, the expression for the total reward value of the robot's fixed-height jump task is: (4) Deploy the trained policy network model to the robot and optimize the control parameters in real time.

2. The method for controlling the jumping of a hydraulically legged robot based on reinforcement learning according to claim 1, characterized in that, Step (1) also includes: establishing a kinematic and dynamic model of the leg unit of the hydraulic legged robot; Design a foot trajectory generator module for limb and leg units; Design a limb and leg unit motion tracker module; A dynamic model of the valve-controlled hydraulic system is established, and joint force control is achieved by combining physical constraints.

3. The method for controlling the jumping of a hydraulically legged robot based on reinforcement learning according to claim 2, characterized in that, The design of the limb unit foot trajectory generator module and motion tracker module includes: The foot motion trajectory was designed using a fifth-order polynomial, and the equations were obtained by substituting the initial and final displacements, velocities, and accelerations of the trajectory motion. The motion tracker module was designed, and a PD controller with dynamic model compensation was selected as the motion controller for the limb unit. The trajectory of the foot was controlled by calculating the contact force at the foot end.

4. The method for jumping control of a hydraulic legged robot based on reinforcement learning according to claim 2, characterized in that, Establishing a dynamic model of a valve-controlled hydraulic system and implementing joint force control by combining physical constraints includes: In a simulation environment, physical feasibility constraints are imposed on the valve-controlled hydraulic system of the joint hydraulic actuator, limiting the valve flow rate, control current, hydraulic cylinder stroke, and joint friction torque to within the actual range of the hydraulic legged robot.

5. The method for controlling the jumping of a hydraulically legged robot based on reinforcement learning according to claim 1, characterized in that, Determining the motion space in step (2) also includes: for the motion tracker, selecting the stiffness coefficient and damping coefficient as motion output parameters.

6. The method for jumping control of a hydraulically legged robot based on reinforcement learning according to claim 1, characterized in that, Building reinforcement learning neural network models includes: The reinforcement learning algorithm uses the Proximal Policy Optimization (PPO) algorithm. The initial neural network model includes a policy network and a value function network. The policy network is used to generate an action probability distribution, and the value function network is used to evaluate the generated actions to obtain state values.

7. The method for jumping control of a hydraulically legged robot based on reinforcement learning according to claim 1, characterized in that, Determining the reward function and domain randomization parameters, and training the reinforcement learning model to convergence in the training environment includes: The simulation training environment is set up according to domain randomization; Start training with a set number of rounds, and continuously update the strategy network based on rewards; Determine if the reward converges; If it is determined that the total reward has not converged, the weight coefficient of each reward is redesigned based on the individual reward details. Training is completed once the total reward value is determined to converge stably.

8. The method for jumping control of a hydraulic legged robot based on reinforcement learning according to claim 1, characterized in that, The reward function includes a reward for tracking target height, a penalty for excessive landing contact force, a penalty for joint angle tracking error, and a reward for completing the task objective.

9. A method for controlling the jumping of a hydraulically legged robot based on reinforcement learning according to claim 7, characterized in that, The simulation training environment is set up according to domain randomization, including: Determine the physical parameters involved in domain randomization and their threshold ranges; based on the threshold ranges, generate and select corresponding physical randomization parameters through a random function; determine the type of environmental noise in the training environment, and determine the environmental noise in the training environment according to the preset noise parameters, the correspondence between the type of environmental noise and the sensor position.

10. A control system for a hydraulically legged robot based on reinforcement learning, characterized in that, The system includes a processor configured to execute the reinforcement learning-based hydraulic legged robot control method as described in any one of claims 1 to 9.

Images