Control method and system of quad-wheeled legged robot based on reinforcement learning and composite model

Abstract

The present disclosure provides a control method and system for a four-wheeled and legged robot based on reinforcement learning and a composite model, which relates to the field of robot automatic control technology, introduces multiple control modes (wheeled, legged, and wheel-legged composite modes) to achieve efficient passage through various terrains. The use of reinforcement learning methods to replace traditional model predictive control greatly simplifies the complexity of the control system and enhances the real-time response capability to dynamic environments. The composite model optimization controller further optimizes speed distribution through quadratic programming to ensure optimal speed and stability performance. Users can manually switch control modes to meet the needs of different mission scenarios, such as patrolling, rescue, and transportation tasks in complex terrains. The present disclosure has shown excellent passing efficiency and energy consumption performance in experiments, especially in terms of terrain transition and smoothness of multi-mode switching.

Description

Technical Field

[0001] The present disclosure relates to the technical field of robot automatic control, and in particular to a control method and system for a four-wheeled legged robot based on reinforcement learning and a composite model. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] The existing control of quad-legged robots mostly relies on accurate kinematic and dynamic models, such as model predictive control (MPC). Although this method has made some progress in control effect, it still has high requirements on motion planning and operator quality. The quad-legged robot combines the motion modes of wheels and feet and has excellent terrain traversal capabilities.

[0004] However, the existing control schemes for four-wheeled and legged robots usually adopt single-mode control, including wheel control, leg control mode and wheel-leg combined control mode. There is a lack of effective mode switching mechanism between the various modes. It is impossible to switch between wheel mode, leg mode and wheel-leg combined mode in real time and flexibly according to the changes in terrain. It is impossible to perform effective switching transition, resulting in the inability to achieve optimal motion performance in complex terrain. For example, when the robot is traveling in a scene from flat ground to rugged terrain, the wheel mode cannot be converted to the leg mode in time to adapt to the terrain changes, resulting in a decrease in stability and passability. Therefore, it is difficult for the existing technology to intelligently dispatch different control modes to cope with changing terrains, and it is impossible to take into account the dual requirements of high-speed movement and complex terrain passage, which limits the application of four-wheeled and legged robots in various scenarios. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a four-wheeled legged robot control method and system based on reinforcement learning and composite model. Through intelligent scheduling of learning strategies and optimization of composite models, efficient and flexible control of quadruped robots and four-wheeled legged robots is achieved, the adaptability and motion stability of robots in complex terrains are enhanced, multi-mode motion control is achieved, and the optimal control strategy can be selected according to different terrains to meet the needs of complex environments.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] The control method of the four-wheeled legged robot based on reinforcement learning and composite model includes:

[0008] Select the robot's multi-mode motion control according to the changes in the external environment and terrain;

[0009] Among them, when the robot is on a rough terrain, the robot is implemented in a foot-type or wheel-foot combined control mode. The foot-type control process includes:

[0010] Get the speed command, the robot's current state space vector, and the action information of the previous step;

[0011] The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed;

[0012] The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate Q-value to guide the update of the control strategy, so that the robot can reach the optimal desired speed and realize foot-type motion control.

[0013] The wheel-leg composite control process includes: using a quadratic programming algorithm to distribute the input target speed command demand to the wheel and the leg, x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel. Reinforcement learning is performed based on the speed distribution value to achieve motion control.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions:

[0015] The control system of the four-wheeled legged robot based on reinforcement learning and composite model includes:

[0016] A control mode selection module is used to select the multi-mode motion control of the robot according to the changes in the external environment terrain;

[0017] The compound control module is used to implement the foot-type or wheel-foot compound control mode for the robot when the robot is on rough terrain. The foot-type control process includes:

[0018] Get the speed command, the robot's current state space vector, and the action information of the previous step;

[0019] The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed;

[0020] The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate Q-value to guide the update of the control strategy, so that the robot can reach the optimal desired speed and realize foot-type motion control.

[0021] The wheel-leg composite control process includes: using a quadratic programming algorithm to distribute the input target speed command demand to the wheel and the leg, x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel. Reinforcement learning is performed based on the speed distribution value to achieve motion control.

[0022] According to some embodiments, the present disclosure adopts the following technical solutions:

[0023] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the control method of a four-wheeled legged robot based on reinforcement learning and a composite model is implemented.

[0024] According to some embodiments, the present disclosure adopts the following technical solutions:

[0025] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the control method of a four-wheeled legged robot based on reinforcement learning and a composite model is implemented.

[0026] According to some embodiments, the present disclosure adopts the following technical solutions:

[0027] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the control method of a four-wheeled legged robot based on reinforcement learning and a composite model.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The disclosed control method of a quadruped robot based on reinforcement learning and a composite model is applicable to quadruped robots and quadruped robots. Multi-mode motion control is achieved by combining reinforcement learning with a composite model to optimize the controller. The present invention not only simplifies the complexity of the control system and enhances the real-time response capability to complex environments, but also enables the robot to have excellent terrain adaptability by introducing multiple control modes (wheeled, legged and composite modes).

[0030] The disclosed control method for a four-wheeled legged robot based on reinforcement learning and a composite model combines a reinforcement learning control strategy with a composite model optimization controller to provide a unified control architecture for quadrupeds and four-wheeled legged robots, and supports manual switching of motion modes. The pre-trained optimal strategy significantly reduces the need for real-time computing, and the composite model optimization controller further optimizes speed and stability, so that the robot has excellent motion performance in all terrains. Especially in the wheel-foot composite mode, the coordinated work of the wheels and the feet greatly improves the motion efficiency and stability.

[0031] The disclosed control method of a four-wheeled legged robot based on reinforcement learning and composite model combines reinforcement learning and composite model optimization control technology, and realizes effective control of the multi-mode motion of the four-wheeled legged robot through a trained strategy network. The robot can use wheel control to achieve high-speed movement on flat ground, and switch to foot control in complex terrain to enhance the ability to pass. At the same time, the composite control mode is used to achieve wheel-foot coordination, significantly improving the adaptability and stability in diverse terrains.

[0032] The control method of the four-wheeled and legged robot based on reinforcement learning and composite model of the present invention realizes efficient passage through various terrains by introducing multiple control modes (wheeled, legged and wheel-legged composite modes). The use of reinforcement learning method to replace the traditional model predictive control (MPC) greatly simplifies the complexity of the control system and enhances the real-time response capability to dynamic environments. The composite model optimization controller further optimizes the speed distribution through quadratic programming (QP) to ensure the best speed and stability performance. Users can manually switch control modes to meet the needs of different mission scenarios, such as patrolling, rescue and transportation tasks in complex terrain. The present invention has shown excellent passing efficiency and energy consumption performance in experiments, especially in terms of terrain transition and smoothness of multi-mode switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0034] Figure 1 A schematic diagram of the structure and coordinate system of a four-wheeled legged robot according to an embodiment of the present disclosure;

[0035] Figure 2 is a multi-mode motion control flow chart of an embodiment of the present disclosure;

[0036] Figure 3 A schematic diagram of a foot-type motion control strategy training process according to an embodiment of the present disclosure;

[0037] Figure 4A schematic diagram of the control of the wheel-foot compound motion combined with reinforcement learning and compound model according to an embodiment of the present disclosure;

[0038] Figure 5 The system deployment and verification flow chart of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] Example 1

[0043] In one embodiment of the present disclosure, a control method for a four-wheeled legged robot based on reinforcement learning and a composite model is provided, comprising:

[0044] Select the robot's multi-mode motion control according to the changes in the external environment and terrain;

[0045] Among them, when the robot is on a rough terrain, the robot is implemented in a foot-type or wheel-foot combined control mode. The foot-type control process includes:

[0046] Get the speed command, the robot's current state space vector, and the action information of the previous step;

[0047] The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed;

[0048] The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate Q-value to guide the update of the control strategy, so that the robot can reach the optimal desired speed and realize foot-type motion control.

[0049] The wheel-leg composite control process includes: using a quadratic programming algorithm to distribute the input target speed command demand to the wheel and the leg, x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel. Reinforcement learning is performed based on the speed distribution value to achieve motion control.

[0050] As an embodiment, the four-wheeled and legged robot control method based on reinforcement learning and composite model disclosed in the present invention provides a manual mode switching function. Through the handle remote control, the operator can manually switch the robot's control mode between wheel type, leg type and wheel-legged composite mode as needed. This manual switching method enables the operator to better control the robot in specific scenarios.

[0051] When the robot is on a flat terrain, select the wheel control mode. Wheel control is to directly control the movement of the wheel motor with speed instructions, which does not require joint angle control, such as Figure 2 As shown, the vehicle can minimize energy consumption and increase speed. When encountering complex terrain (rough terrain), the vehicle can switch to foot-type or wheel-foot combined mode through the handle remote control, control the legs using the strategy generated by the Actor network, and use QP to optimize the speed assigned to the wheel controller to control the wheels for efficient passage.

[0052] As an embodiment, the control method of the four-wheeled legged robot based on reinforcement learning and composite model disclosed in the present invention switches to the foot-type or wheel-foot composite mode when encountering complex terrain. The specific process of foot-type reinforcement learning motion control is:

[0053] Step 1: Get the speed command, the robot's current state space vector, and the action information of the previous step;

[0054] The current state space vector of the robot includes the linear velocity of the base, the angular velocity of the base, the gravity vector, the current joint position and the current joint velocity, and the action information of the previous step is the action command of the previous step.

[0055] The speed command is command = ,in , and They represent the robot along the x-axis direction (coordinate system such as Figure 1 The linear velocity along the y-axis and the angular velocity around the z-axis (yaw velocity) are shown.

[0056] During training, the speed command is a desired range, e.g. , the robot will learn and explore from slow to fast within this speed range, where the negative sign indicates movement in the opposite direction of the x-axis. In simulation deployment and physical deployment, specific movement speed instructions are sent to the robot through external input devices (such as keyboards and remote controls). For example, by inputting command = To control the robot to move along the x-axis at a speed of 0.5m / s.

[0057] The speed command is passed to the control system. In the present invention, a reinforcement learning strategy is used to control the robot according to the current state space vector, which is calculated by the physical engine, including the base linear speed. , base angular velocity , gravity vector g, current joint position , current joint velocity , the last actions command and speed command of the previous step Generate action commands Actions, such as Figure 3 shown.

[0058] Step 2: Input the state space vector, speed command and the action information of the previous step into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and the action information of the previous step, and outputs the expected joint position and the expected joint speed;

[0059] Actor MLP is used to receive the state space vector, speed command and action information of the previous step, and generate the corresponding action output. The goal is to find the optimal (the action that can maximize the reward) ) Control signal: desired position of the joint and the desired velocity of the joint .

[0060] The strategy of an actor network can be expressed as a function ,Actor network maximizes the objective function:

[0061]

[0062] To find the optimal action. Where s is the input state space vector, Parameters for the Actor network.

[0063] Then the policy gradient method is used to calculate the gradient and update the network weights:

[0064]

[0065] Next, the weights are updated using the optimization method:

[0066]

[0067] in is the learning rate, which determines the update step size.

[0068] Step 3: Input the desired joint position and desired joint speed into the joint controller to obtain the current action of the robot. Use the Critic network to evaluate the current action and generate a reward value to pass to the Critic to generate a Q-value to guide the update of the control strategy so that the robot reaches the optimal desired speed.

[0069] Specifically, the Critic MLP is used to evaluate the quality of actions generated by the Actor network. The Critic network inputs state and action information, and generates an evaluation result Q-value based on the reward value from the environment for policy gradient update to update the weights of the Actor network, so that the Actor network gradually evolves towards generating a better control strategy.

[0070] The Critic network calculates the target Q-value through the Bellman equation:

[0071]

[0072] Among them are instant rewards, For this discount factor, Represents the weight of the target network, which is used to stabilize the training of the Critic network.

[0073] The critic network updates the weights by minimizing the mean square error between the predicted Q-value and the target Q-value. Its loss function is defined as:

[0074]

[0075] Calculate the loss function of the network parameters through back propagation The gradient of , and update it using the gradient descent method:

[0076]

[0077] in is the learning rate, which determines the update speed of the Critic network.

[0078] When the optimal desired speed is obtained, the desired speed is input into the joint controller (PD control). The underlying joint controller uses a PD (proportional-derivative) controller to execute the action generated by the strategy. The P controller is used to reduce the deviation between the joint position and the target position, and the D controller reduces the jitter caused by speed changes to ensure smoothness.

[0079] Control formula:

[0080]

[0081] in, is the joint torque, and are proportional gain and differential gain respectively, is the desired joint position, is the current joint position, is the expected joint velocity, is the current joint velocity.

[0082] Furthermore, motion execution and feedback control are performed, and the joint controller drives the joints through the calculated torque so that the robot moves in the desired manner.

[0083] In the simulation environment, the torque output of the controller is passed to the physics engine (NVIDIA Isaac Gym), which calculates the robot's state at the next moment, including position and velocity, based on these torques. At the same time, the generated reward value is passed to the Critic to generate Q-value to guide the update of the control strategy, so as to further optimize the action selection and ensure the optimal control performance at each time step.

[0084] The new state calculated by the simulation engine is fed back to the control system to form a closed-loop control. The controller continuously adjusts the torque of the joint according to the current state to achieve the desired movement.

[0085] As an embodiment, the control process of realizing the left turn of the robot is as follows:

[0086] 1) Input left turn speed command (rad / s);

[0087] 2) The target angular velocity is passed to the trained policy model. The PPO policy generates action commands based on the current state to control the left leg to push outward and the right leg to push inward to generate an angular velocity of rotation around the z-axis.

[0088] 3) These motion commands are passed to the PD controller, which calculates the torque applied to each joint based on the current joint position, current joint velocity, target joint position, and target joint velocity.

[0089] 4) The physics engine calculates the new state of the robot based on the torque output, and the robot starts to turn left.

[0090] 5) The torque of each joint is continuously adjusted through feedback control to ensure that the robot turns left stably.

[0091] As an embodiment, the control method of the four-wheeled legged robot based on reinforcement learning and composite model disclosed in the present invention, the wheel-foot composite control mode is based on the foot-type motion control based on reinforcement learning combined with the composite model optimization controller to achieve the optimal control of the wheel-foot composite mode, such as Figure 4 In the transition between flat and rough terrain, a composite model optimization controller is used to adjust and distribute the velocity to maintain the robot’s balance and efficient motion.

[0092] Application of Quadratic Programming (QP) Algorithm: The composite model optimization controller uses the quadratic programming (QP) algorithm to distribute the speed requirements of the robot during the movement to the wheels and legs, and enables the robot to adapt to different terrain conditions through real-time optimization. For example, in flat terrain, more power can be allocated to the wheels to achieve high-speed movement; while in complex terrain, the movement ratio of the legs will increase to improve stability and terrain passing ability. First, the QP optimization algorithm is used to achieve the speed distribution of the wheel-foot composite optimization control to maintain the balance and efficient movement of the robot. Then, according to the allocated speed value, the Actor MLP is used to generate the corresponding action output according to the state space vector, the allocated speed value and the action information of the previous step, and the Critic network is used to find the optimal control signal-the expected position of the joint and the expected speed of the joint to maintain the balance and efficient movement of the robot.

[0093] Mainly, a quadratic programming algorithm is used to allocate the optimal expected speed requirements to the wheels and legs. x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel.

[0094] The specific QP optimization algorithm is as follows:

[0095] The general form of quadratic programming is:

[0096]

[0097]

[0098]

[0099] in, x is the variable to be optimized, G is the Hessian matrix, h is the coefficient vector of the objective function, a is the coefficient matrix of the equality constraints, b is the equality constraint vector, c is the coefficient matrix of the inequality constraints, d is the inequality constraint vector,M for m A set of constraints, N for n A set of inequality constraints.

[0100] The present invention uses secondary optimization to adjust the linear velocity in the x-axis direction Optimized for wheel speed and leg speed , the decision vector is set to , the objective function is set to,

[0101]

[0102] in, is the weight factor that adjusts the importance of the two parts in the objective function. The value of can change the contribution of leg speed and wheel speed to the final objective function. If is larger, more emphasis is placed on minimizing the difference between the wheel speed and the target speed. If it is smaller, the optimization of leg speed dominates.

[0103] Substituting the decision vector and objective function into the general form of quadratic optimization, we can get the Hessian matrix G and the coefficient vector of the objective function h :

[0104]

[0105] According to the constraints The coefficient matrix a and constraint vector b of the equality constraint can be obtained:

[0106]

[0107] According to the wheel speed inequality constraint:

[0108]

[0109] The coefficient matrix of the inequality constraints can be obtained c and constraint vector d :

[0110]

[0111] At this point, the QP quadratic optimization problem is solved.

[0112] Furthermore, the speed and posture are optimized and the composite model optimization controller receives the target speed input from the user. , the target speed is decomposed through the QP algorithm to determine the speed distribution of each joint and wheel.

[0113] Wheel speed It is transmitted to the wheel controller, driving the wheel motor to move.

[0114] Leg speed , combined with the user's input, the angular velocity around the z-axis and the linear velocity along the y-axis , input into the policy model obtained by reinforcement learning training, the output action is transmitted to the underlying PD joint controller, and the target torque of each joint is calculated to control the joint movement of the leg.

[0115] Finally, through the coordinated control of wheels and legs, the quadruped robot can move stably and efficiently in different terrain environments. The core role of the QP optimization algorithm is to ensure the allocation of control resources between wheels and legs to achieve a balance between stability and efficiency, thereby achieving the optimal motion control effect.

[0116] This process ensures a balance between the robot's stability and efficiency in different terrain conditions.

[0117] The composite model optimization controller significantly improves the robot's ability to transition between different terrains. On flat ground, the robot achieves high-speed movement close to that of a wheeled robot, while on rugged terrain, the optimization of the leg posture ensures smooth passability.

[0118] Simulation experiment

[0119] like Figure 5 As shown in the figure, the system deployment and verification, the trained Actor-Critic strategy network is deployed to the control system of the quadruped (lock the wheels of the quadruped robot) and the quadruped robot. After the robot is initialized with the remote control, the desired speed and desired angular velocity are input. The robot detects its own state and selects different motion modes. The strategy network selects the action to be performed in the next step according to the speed command and the current state and the action of the previous step, calculates the joint torque, and then transmits it to each joint driver to generate torque to control the robot's motion that meets energy efficiency, action continuity, and smoothness of working condition switching. The multi-mode switching capability and adaptability to complex terrain of the present invention were verified by testing in multiple different physical environments (such as flat ground and rugged gravel roads).

[0120] Deployment preparation:

[0121] Configure the software system, install and configure the Robot Operating System (ROS).

[0122] Configure the reinforcement learning strategy running environment.

[0123] Load the network parameters and load the trained network parameters of the reinforcement learning model into the constructed network model.

[0124] Webots simulation verification:

[0125] Simulation platform configuration, configure the four-wheeled robot model in the Webots simulation platform, including the parameters of sensors and actuators. Load the reinforcement learning network and the composite model optimization controller.

[0126] Build different terrain conditions in Webots, including flat land, slopes, gravel roads, etc., to simulate the complex terrain in the real environment.

[0127] Simulation verification process:

[0128] Preliminary test: Conduct preliminary tests in a simple flat environment to verify the robot's basic motion functions, such as moving forward, backward, and turning.

[0129] Complex terrain test: Gradually increase the complexity of the terrain to verify the robot's stability and control effect during multi-mode switching.

[0130] Multi-mode switching: Test the switching between wheeled, footed and combined modes to ensure that the robot can smoothly switch motion modes under different terrains.

[0131] Closed-loop control performance verification:

[0132] Verify closed-loop control performance and check the robot’s sensor data feedback in simulation to ensure that the PD controller can maintain smooth motion by adjusting joint torques in real time.

[0133] Physical deployment and testing:

[0134] Deploy to the actual robot:

[0135] The control system verified in Webots is deployed to the actual four-wheeled legged robot prototype.

[0136] Check the hardware connection to ensure normal communication between sensors, actuators, and control platforms.

[0137] Performance Test:

[0138] Conduct static tests in a flat environment to verify whether the robot can perform basic actions such as moving forward, turning, and stopping.

[0139] Dynamic tests are carried out in different physical environments (such as flat ground, sloped ground, and gravel roads) to observe the stability and control effect of the robot during multi-mode switching.

[0140] Performance optimization:

[0141] According to the test results, the PD controller gain parameters and control strategy are adjusted to optimize the robot's motion performance to ensure its stability and efficiency.

[0142] Experimental results show that the reinforcement learning method has the following advantages over traditional MPC: First, reinforcement learning reduces the reliance on real-time calculations through pre-trained strategies, thereby significantly reducing computational complexity; second, the reinforcement learning model has higher environmental adaptability and can adjust gaits in real time according to different terrain conditions; in addition, the reinforcement learning control method has a faster response speed and can better cope with dynamic changes in complex terrain. Combined with the smooth multi-mode switching of the manual mode, the overall motion performance of the robot is significantly improved.

[0143] Example 2

[0144] In one embodiment of the present disclosure, a four-wheeled legged robot control system based on reinforcement learning and a composite model is provided, comprising:

[0145] A control mode selection module is used to select the multi-mode motion control of the robot according to the changes in the external environment terrain;

[0146] The composite control module is used to implement the wheel-foot composite control mode for the robot when the robot is on rough terrain, including:

[0147] Get the speed command, the robot's current state space vector, and the action information of the previous step;

[0148] The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed;

[0149] The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate the Q-value to guide the update of the control strategy, so that the robot reaches the optimal desired speed.

[0150] The quadratic programming algorithm is used to allocate the optimal expected speed requirements to the wheels and legs. x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel.

[0151] Example 3

[0152] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the control method of a four-wheeled legged robot based on reinforcement learning and a composite model is implemented.

[0153] Example 4

[0154] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the control method of the four-wheeled legged robot based on reinforcement learning and a composite model is implemented.

[0155] Example 5

[0156] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the control method of the four-wheeled legged robot based on reinforcement learning and the composite model.

[0157] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A control method for a four-wheeled legged robot based on reinforcement learning and a composite model, characterized in that: include: Select the robot's multi-mode motion control according to the changes in the external environment and terrain; Among them, when the robot is on a rough terrain, the robot is implemented in a foot-type or wheel-foot combined control mode. The foot-type control process includes: Get the speed command, the robot's current state space vector, and the action information of the previous step; The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed; The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate Q-value to guide the update of the control strategy, so that the robot can reach the optimal desired speed and realize foot-type motion control. Critic MLP is used to evaluate the quality of actions generated by the Actor network. The Critic network inputs state and action information, and generates an evaluation result Q-value based on the reward value from the environment for policy gradient update to update the weight of the Actor network, so that the Actor network gradually evolves towards generating a better control strategy. The wheel-leg composite control process includes: using a quadratic programming algorithm to distribute the input target speed command demand to the wheel and the leg, x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel. Reinforcement learning is performed according to the speed distribution value to achieve motion control. The quadratic programming algorithm is used to distribute the speed requirements of the robot during the movement to the wheels and legs. Through real-time optimization, the robot can adapt to different terrain conditions. x Linear speed in the axial direction Optimized for wheel speed and leg speed , the decision vector is set to , the objective function is set to, in, It is the weight factor that adjusts the importance of the two parts in the objective function; Substituting the decision vector and objective function into the general form of quadratic optimization, we get the Hessian matrix G and the coefficient vector of the objective function h : According to the constraints Get the coefficient matrix of the equality constraint a and constraint vector b : According to the wheel speed inequality constraint: The coefficient matrix of the inequality constraints is c and constraint vector d : 。 2. The control method of a four-wheeled legged robot based on reinforcement learning and composite model as claimed in claim 1, characterized in that: According to the changes in the terrain of the external environment, the robot is subjected to multi-mode motion control, including: when the robot is on flat terrain, the wheel control mode is selected; when the robot is on rough terrain, it is switched to the wheel-foot composite mode.

3. The control method of a four-wheeled legged robot based on reinforcement learning and composite model as claimed in claim 1, characterized in that: The current state space vector of the robot includes the linear velocity of the base, the angular velocity of the base, the gravity vector, the current joint position and the current joint velocity, and the action information of the previous step is the action command of the previous step.

4. The control method of a four-wheeled legged robot based on reinforcement learning and composite model as claimed in claim 1, characterized in that: The desired joint position and desired joint velocity are input into the joint controller to obtain the current action of the robot. The control formula is: in, is the joint torque, and are proportional gain and differential gain respectively, is the desired joint position, is the current joint position, is the expected joint velocity, is the current joint velocity; The joint controller drives the joints with the calculated torques so that the robot moves in the desired way.

5. A four-wheeled robot control system based on reinforcement learning and composite model, characterized in that: include: A control mode selection module is used to select the multi-mode motion control of the robot according to the changes in the external environment terrain; The compound control module is used to implement the foot-type or wheel-foot compound control mode for the robot when the robot is on rough terrain. The foot-type control process includes: Get the speed command, the robot's current state space vector, and the action information of the previous step; The state space vector, speed command and action information of the previous step are input into the Actor network, and the Actor network performs action optimization according to the input state space vector, speed command and action information of the previous step, and outputs the expected joint position and expected joint speed; The desired joint position and desired joint speed are input into the joint controller to obtain the current action of the robot. The current action is evaluated using the Critic network, and the reward value is generated and passed to the Critic to generate Q-value to guide the update of the control strategy, so that the robot can reach the optimal desired speed and realize foot-type motion control. The wheel-leg composite control process includes: using a quadratic programming algorithm to distribute the input target speed command demand to the wheel and the leg, x The linear speed in the axial direction is optimized to the wheel speed and the leg speed. The decision vector is set and the objective function is constructed. The objective function is solved according to the constraints to determine the optimal speed distribution of each leg joint and the wheel. Reinforcement learning is performed according to the speed distribution value to achieve motion control. The quadratic programming algorithm is used to distribute the speed requirements of the robot during the movement to the wheels and legs. Through real-time optimization, the robot can adapt to different terrain conditions. x Linear speed in the axial direction Optimized for wheel speed and leg speed , the decision vector is set to , the objective function is set to, in, It is the weight factor that adjusts the importance of the two parts in the objective function; Substituting the decision vector and objective function into the general form of quadratic optimization, we get the Hessian matrix G and the coefficient vector of the objective function h : According to the constraints Get the coefficient matrix of the equality constraint a and constraint vector b : According to the wheel speed inequality constraint: The coefficient matrix of the inequality constraints is c and constraint vector d : 。 6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the control method of a quadricycle-legged robot based on reinforcement learning and a composite model as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the quadricycle-legged robot control method based on reinforcement learning and composite model as described in any one of claims 1 to 4 is implemented.

8. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the control method of a four-wheeled legged robot based on reinforcement learning and a composite model as described in any one of claims 1 to 4.

Images