Wheel-legged robot motion control method, system and device for unstructured environment, storage medium and computer equipment

Abstract

The invention discloses an unstructured environment-oriented wheel-legged robot motion control method, system and device, a storage medium and computer equipment, and the control method comprises the steps: constructing a wheel-legged robot simulation model, building a learning training scene of a wheel-legged robot, and carrying out the training through a reinforcement learning algorithm, so as to obtain a nonlinear controller model; converting the nonlinear controller model into a nonlinear controller which can be deployed and operated on an actual physical prototype platform; acquiring current motion state data of the wheel-legged robot, and inputting the current motion state data into a nonlinear controller to obtain an expected joint position and an expected wheel speed of the wheel-legged robot; and the expected joint position and the expected wheel speed are input into a low-level controller, and a joint torque used for controlling the wheel-legged robot to execute a control task is output. According to the method, the development period of the wheel-legged robot controller is greatly shortened, and the robustness, the dynamics and the flexibility of the wheel-legged robot moving in the unstructured environment are also improved.

Description

Technical Field

[0001] The present invention relates to robot technology, and in particular to a wheel-legged robot motion control method, system, device, storage medium and computer equipment for unstructured environments. Background Art

[0002] Wheel-legged robots have great application prospects because they combine the advantages of wheeled robots, such as fast movement speed and high energy efficiency on flat terrain, and footed robots, which have strong adaptability to complex terrain. However, since wheel-legged robots are inherently unstable under-actuated systems, there are problems with balance and control of highly dynamic motion in unstructured rugged terrain. In related technologies, a linear quadratic regulator (LQR) or a model predictive control (MPC) controller is constructed by simplifying the dynamic model to achieve balance control, and a proportional-integral-derivative controller (PID) is combined to achieve tasks such as changing leg length and turning. There is also related technology that controls the process of wheel-legged robots performing various tasks by constructing a full-body dynamics controller.

[0003] However, the control scheme combining LQR or MPC and PID in the related art simplifies the dynamic model, so the wheel-legged robot cannot deviate too much from its linearized equilibrium point, and can only achieve a few specific movements in a simple environment. The solution of building a whole-body dynamic controller in the related art has a certain dynamic motion capability in an unstructured environment, but this solution requires the establishment of a more accurate dynamic model and a very complex control framework, and requires very time-consuming and labor-intensive multi-task design and parameter setting.

[0004] How to shorten the design cycle of wheel-legged robot controllers and enable wheel-legged robots to achieve agile and dynamic movement in unstructured environments are problems that need to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above prior art and provide a wheel-legged robot motion control method for unstructured environments. This wheel-legged robot motion control method for unstructured environments can significantly shorten the design cycle of the wheel-legged robot controller and improve its motion ability in unstructured environments.

[0006] The second object of the present invention is to provide a wheel-legged robot motion control system.

[0007] A third object of the present invention is to provide a wheel-legged robot device.

[0008] A fourth object of the present invention is to provide a storage medium.

[0009] A fifth object of the present invention is to provide a computer device.

[0010] The first object of the present invention is achieved by the following technical solution: The motion control method of the wheel-legged robot for unstructured environment comprises the following steps:

[0011] S100, constructing a wheel-legged robot simulation model, and building a learning and training scenario for the wheel-legged robot on a virtual simulation platform, and using a reinforcement learning algorithm to train a nonlinear controller model in a simulation environment;

[0012] S200, converting the nonlinear controller model into a nonlinear controller that can be deployed and run on an actual physical prototype platform;

[0013] S300, obtaining current motion state data of the wheel-legged robot;

[0014] S400, inputting the current motion state data of the wheel-legged robot into the nonlinear controller to obtain the desired joint position and the desired wheel speed of the wheel-legged robot;

[0015] S500: Input the desired joint positions and the desired wheel speeds into the low-level controller, thereby obtaining the joint torques for controlling the wheel-legged robot.

[0016] Preferably, the learning training scene in step S100 is an unstructured environment, and the unstructured environment includes flat ground, rough flat ground, flat slope, rough slope, flat single-sided bridge and rough single-sided bridge.

[0017] Preferably, in step S100, obtaining a nonlinear controller model using a reinforcement learning algorithm in a simulation environment includes the following steps:

[0018] S110, Model the motion control problem as a Marxian decision process;

[0019] S120, defining a state space;

[0020] S130, defining an action space;

[0021] S140, defining a reward function;

[0022] S150, set up automated course learning;

[0023] S160, set domain randomization and add random noise;

[0024] S170, setting training parameters, thereby completing the training to obtain a nonlinear controller model.

[0025] Preferably, the reward function includes a command tracking reward function, a body stability reward function, an action smoothness reward function, a constraint reward function and an energy saving reward function.

[0026] Preferably, step S200 includes the following steps:

[0027] S210, converting the file format of the nonlinear controller model into an intermediate format that is convenient for further processing;

[0028] S220, generating executable code that does not depend on a specific library from the model in the converted format;

[0029] S230, compile the executed code into a nonlinear controller suitable for a real-time control system of a wheel-legged robot.

[0030] Preferably, the low-level controller includes any one or a combination of a PD controller and a P controller.

[0031] The second object of the present invention is achieved by the following technical solution: a wheel-legged robot motion control system for unstructured environments, comprising:

[0032] A data acquisition module is used to obtain the current motion state data of the wheel-legged robot;

[0033] A data processing module processes the current motion state data to obtain the joint torque for controlling the wheel-leg robot;

[0034] An execution module is used to execute the wheel-legged robot motion control method for unstructured environments described in the first purpose.

[0035] The third purpose of the present invention is achieved through the following technical scheme: a wheel-legged robot device for unstructured environments, comprising a wheel-legged robot body, a main control module, a communication module and an auxiliary module, the body being provided with a sensor for collecting data, the sensor being connected to the communication module, the main control module and the auxiliary module being both connected to the communication module, thereby achieving the wheel-legged robot motion control method for unstructured environments described in the first purpose.

[0036] The fourth purpose of the present invention is achieved through the following technical solution: a storage medium stores a program, and when the program is executed by a processor, the motion control method of a wheel-legged robot for an unstructured environment described in the first purpose is achieved.

[0037] The fifth purpose of the present invention is achieved through the following technical solution: a computer device, comprising a processor and a memory for storing processor executable programs, characterized in that when the processor executes the program stored in the memory, the wheel-legged robot motion control method for unstructured environments described in the first purpose is implemented.

[0038] The present invention has the following advantages over the prior art:

[0039] The present invention obtains a nonlinear controller model of the robot by performing large-scale parallel training on virtual simulation software, and then converts the nonlinear controller model into a nonlinear controller that can be deployed on an actual prototype and deploys it to the actual prototype for control. This method greatly shortens the development cycle of the wheel-legged robot controller. In addition, this method builds many complex unstructured environments during the controller training phase, and optimizes the controller end-to-end by using data generated by interaction with the environment. The obtained full-body force controller has robustness, dynamics and flexibility that are difficult to achieve with controllers designed based on traditional methods, and is therefore more adaptable to various complex unstructured environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the mechanical structure of the wheel-legged robot of the present invention.

[0041] Figure 2 It is a flow chart of the control method of the wheel-legged robot of the present invention.

[0042] Figure 3 It is a schematic diagram of a learning and training scenario of a wheel-legged robot built on a simulation platform of the present invention.

[0043] Figure 4 It is a schematic diagram of a training framework for training a nonlinear controller model using a reinforcement learning algorithm in a simulation environment according to the present invention.

[0044] Figure 5 It is a flow chart of the present invention for training a nonlinear controller model of a wheel-legged robot using a reinforcement learning algorithm in a simulation environment.

[0045] Figure 6 It is a schematic diagram of the state space defined when the reinforcement learning algorithm is used to train the nonlinear controller model of the wheel-legged robot in a simulation environment of the present invention.

[0046] Figure 7 It is the reward function definition table of the present invention.

[0047] Figure 8 This is a table explaining the meaning of the symbols involved in the reward function definition of the present invention.

[0048] Fig. 9It is a structural block diagram of the wheel-legged robot control system of the present invention.

[0049] Fig.10 It is a schematic diagram of the wheel-legged robot device of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] like Figure 1 As shown, the wheel-legged robot used in this embodiment includes a wheel-legged robot body 1 and two tandem wheel-legged structures arranged symmetrically about the sagittal plane. The tandem wheel-legged structure consists of a thigh 2, a shank 3 and a driving wheel 4. The thigh 2 is connected to the body through a hip joint, the thigh 2 is connected to the shank 3 through a knee joint, and the shank 3 is connected to the driving wheel 4 through a driving wheel joint. Each wheel-legged structure contains three independent rotational degrees of freedom of the hip joint, the knee joint and the driving wheel. The knee and hip joint motors are coaxially arranged on the body, and the knee joint motor is connected through a Figure 1 The parallel four-bar mechanism shown transmits power to the knee joint, and the driving wheel is also equipped with a rotary motor. In addition, an inertial measurement unit (IMU) is installed on the main body of the wheel-legged robot to obtain the posture information of the main body of the wheel-legged robot, and each rotary motor is equipped with a position encoder to obtain information such as the motor position and speed.

[0052] like Figure 2 As shown, the motion control method of the wheel-legged robot for unstructured environment includes the following steps:

[0053] S100, constructing a wheel-legged robot simulation model, and building a learning and training scenario for the wheel-legged robot on a virtual simulation platform, and using a reinforcement learning algorithm to train a nonlinear controller model in a simulation environment;

[0054] Among them, the wheel-legged robot simulation model is imported into the simulation platform as an intelligent agent trained by the reinforcement learning algorithm. The simulation model adopts the URDF format and has mass, inertia, joint limit, joint motor torque, speed and other constraints similar to the actual prototype. The simulation platform used in this embodiment is NVIDIA Isaac Gym (proprietary name of the simulation platform). This NVIDIA Isaac Gym platform has a GPU acceleration architecture and branch domain randomization function, which can significantly speed up the training of the intelligent agent through parallelization, and learn robust strategies by introducing environmental changes during the training process, which is convenient for transfer to the real world.

[0055] like Figure 3As shown, the learning training scene in step S100 is an unstructured environment, which includes flat flat land, rough flat land, flat slope, rough slope, flat single-sided bridge and rough single-sided bridge. Rough terrain is relative to flat terrain, and the surface of rough terrain is a rugged ground. Exemplarily, the height range of rough terrain is [-0.01, 0.01] m, and the minimum height interval is 0.005 m. It is generated by a series of equidistant two-dimensional grids with grid vertex heights within the range. In addition, in order to carry out curriculum learning in terrain, different difficulties are set for the constructed terrain. As the difficulty of the terrain increases, the slope of the slope or the height of the single-sided bridge will increase.

[0056] The training framework of this embodiment is as follows Figure 4 As shown in the figure, the dashed arrows and dashed boxes only exist in training, and the solid arrows and solid boxes exist in both training and deployment. This embodiment uses the proximal policy optimization (PPO) reinforcement learning algorithm with an asymmetric actor-critic structure. The open source PPO framework released by the Robotics Systems Lab of the Swiss Federal Institute of Technology is used in the specific implementation process. Figure 5 As shown, in step S100, using the reinforcement learning algorithm in the simulation environment to obtain a nonlinear controller model includes the following steps:

[0057] S110. Model the motion control problem as a Marxist decision process. This Marxist decision process is a mathematical framework for describing discrete-time control processes in which the evolution of states and outcomes is partially random. The MDP consists of a state space S, an action space A, and a reward function R(s t ,s t+1 ) and transition probability P(s t+1 |s t ,a t ) is defined by a four-tuple (s, a, p, r). The agent is based on the current state s t From the strategy π(a t |s t ) to interact with the environment and get an immediate reward r t , the goal of RL is to find an optimal policy π * Maximize the sum of discounted rewards in infinite time domain, that is,

[0058]

[0059] Among them, γ∈(0,1) is the discount factor, which is used to adjust the weights of immediate rewards and long-term rewards, and ρπ is the trajectory distribution under strategy π.

[0060] S120. Define state space. This embodiment adopts an asymmetric actor-critic structure to divide the state space into observations and privileged information. Observations include proprioceptive observations and given instructions. Proprioceptive observations refer to state information provided by the robot's own sensors. Therefore, these state quantities can be obtained through sensors such as IMU and motor position encoders during actual deployment. Privileged information refers to some information that can only be obtained during the simulation training phase, such as terrain height information and friction coefficient. In simulation training, both the actor and critic networks can access observations, but only the critic network can access observations and privileged information at the same time. This embodiment adopts the following structure: Figure 6 The state space defined as shown.

[0061] S130. Define the action space. In this embodiment, the dimension of the action space is equal to the degree of freedom of the wheel-legged robot. For the knee and hip joints, the output action is regarded as the desired joint position and is tracked by the PD controller. For the driving wheel joint, the output action is regarded as the desired wheel speed and is tracked by the P controller.

[0062] S140, define a reward function; the reward function includes a command tracking reward function, a fuselage stability reward function, an action smoothness reward function, a constraint condition reward function and an energy saving reward function. The specific reward function definition is as follows: Figure 7 As shown, the meaning of the symbols involved in the reward function is as follows Figure 8 shown.

[0063] Among them, the instruction tracking reward function is to make the wheel-legged robot follow the instructions given by humans, including: tracking the fuselage linear velocity reward function, tracking the fuselage angular velocity reward function, and tracking the fuselage height reward function.

[0064] The body stability reward function is to ensure that the wheel-legged robot can always keep its body level and stable during movement, including: body attitude angle reward function, body attitude angular velocity reward function, body linear velocity reward function in the direction of gravity, and projected gravity reward function.

[0065] The action smoothness reward function is to make the actions taken by the wheel-legged robot smoother and reduce vibration and mutation, including: joint velocity reward function, joint acceleration reward function, action change rate reward function, and action smoothness reward function.

[0066] The constraint reward function is mainly used to introduce some constraints that exist in actual physical prototypes during the training process, such as upper and lower limits of joints. Therefore, it mainly includes: joint limit reward function and collision reward function.

[0067] The energy-saving reward function is to make the wheel-legged robot adopt a more energy-saving movement mode and avoid damaging the hardware due to the output torque exceeding the limit, including: torque reward function and energy reward function.

[0068] S150, setting up automated course learning; as a preferred implementation of the embodiment of the present application, game-inspired automated course learning is adopted for various unstructured terrains. Exemplarily, during training, all robots are randomly assigned to a terrain type and a level representing the difficulty of the terrain. For slope terrain, the slope increases from 0 degrees to 15 degrees, and for single-side bridge terrain, the height increases from 0 cm to 8 cm. If the robot successfully crosses the terrain, it will be reset to a more difficult terrain next time; if within a certain period of time, the distance moved is less than half of the distance required for its target speed, its terrain level will be reduced. When the robot successfully passes the highest level of terrain, a level of terrain is randomly selected for training in the next round of training.

[0069] S160, setting domain randomization and adding random noise; in order to enhance the robustness and generalization of the trained controller, thereby facilitating the transfer from simulation to reality (sim-to-real transfer), domain randomization technology and random noise addition technology are used in the training process of the embodiment of the present application. Exemplarily, the domain randomization technology randomizes some kinematic and dynamic parameters during training, including: low-level PD controller coefficients, friction coefficients, collision recovery coefficients, robot component mass, robot component inertia, robot center of mass position, torque command delay, robot default joint position. Random noise mainly includes: observation value noise, torque noise, and random thrust disturbance.

[0070] S170, setting training parameters, thereby completing the training to obtain a nonlinear controller model. The training parameters include the maximum number of iterations, the number of robots trained in parallel, etc. In this embodiment, the controller of the wheel-legged robot obtained through training is essentially a neural network model, whose input is the proprioceptive observation value accessible to the actor network during training, and the output dimension is the same as the defined action state space dimension. Exemplarily, the nonlinear controller model is saved in .pt format.

[0071] S200, converting the nonlinear controller model into a nonlinear controller that can be deployed and run on an actual physical prototype platform; step S200 includes the following steps:

[0072] S210, converting the file format of the nonlinear controller model into an intermediate format that is convenient for further processing; in this embodiment, the nonlinear controller model in .pt format is converted into .mat intermediate format;

[0073] S220. Generate executable code from the converted format model that is not dependent on a specific library. Specifically, this embodiment uses the MATLAB toolbox to convert the .mat format model into a C language code that is not dependent on any library. The C language code is a specific program implementation of the wheel-legged robot nonlinear controller during actual deployment.

[0074] S230, compile the executed code into a nonlinear controller suitable for the real-time control system of the wheel-legged robot. In this embodiment, the C code is cross-compiled into a nonlinear controller that can be run in a real-time control system based on the ARM architecture.

[0075] S300, obtaining the current motion state data of the wheel-legged robot; the current motion state data is used to represent the motion characteristics of the wheel-legged robot. The nonlinear controller input of the wheel-legged robot is the proprioceptive observation value and the given instruction. In this embodiment, the state quantity or given instruction required by the wheel-legged robot can be obtained in real time from sensors such as IMU, position encoder, or other means.

[0076] S400, input the current motion state data of the wheel-legged robot into the nonlinear controller to obtain the expected joint position and expected wheel speed of the wheel-legged robot; the expected joint position and expected wheel speed are used as inputs of the low-level controller.

[0077] S500, input the expected joint position and the expected wheel speed into the low-level controller, so as to obtain the joint torque for controlling the wheel-legged robot. Specifically, the expected joint position in this embodiment includes the expected position of the knee joint and the expected position of the hip joint, and the low-level controller includes a PD controller and a P controller. The expected positions of the knee and hip joints are respectively input into their respective PD controllers, and the expected joint speed in the PD controller is set to 0, so as to calculate the required control torque of the knee and hip joints; similarly, the expected wheel speed is input into the P controller, and the required control torque of the drive wheel is obtained through calculation.

[0078] The second object of the present invention is achieved by the following technical solutions: Fig. 9 As shown, the motion control system of the wheel-legged robot for unstructured environment includes:

[0079] The data acquisition module is used to obtain the current motion state data of the wheel-legged robot; the current motion state data specifically includes the attitude angle and attitude angular velocity information returned by the IMU, the motor position and speed information obtained by the 6 motor position encoders, the given instruction information, and the action value output by the last model.

[0080] A data processing module processes the current motion state data to obtain the joint torque for controlling the wheel-leg robot;

[0081] An execution module is used to execute the wheel-legged robot motion control method for unstructured environments described in the first purpose.

[0082] The third object of the present invention is achieved by the following technical solutions: Fig.10 As shown, a wheel-legged robot device for an unstructured environment includes a wheel-legged robot body, a main control module, a communication module and an auxiliary module, wherein the body is provided with a sensor for collecting data, the sensor is connected to the communication module, and the main control module and the auxiliary module are both connected to the communication module, so as to realize the wheel-legged robot motion control method for an unstructured environment described in the first purpose. Specifically, the sensor includes an IMU and a motor position encoder, wherein the IMU is used to obtain the posture information of the wheel-legged robot body, and the motor position encoder is used to obtain information such as the motor position and speed. The main control module is a main control PCB board including a processor and a memory, and the main control PCB board includes an STM32F405RGT6 chip. The communication module is used for information interaction between modules, and undertakes tasks including reading sensor signals and sending control signals. In an embodiment of the present application, the IMU communicates with the main control PCB board via a serial port line, and each motor communicates with the main control PCB board via a CAN bus protocol. The auxiliary module refers to a module that undertakes other tasks. In the embodiment of the present application, the auxiliary module includes a power supply module, a heat dissipation module, etc., wherein the power supply module uses a lithium battery with a 24V DC voltage output, and the heat dissipation module uses two cooling fans with a size of 40mm×40mm×10mm and a 24V DC voltage input.

[0083] The fourth purpose of the present invention is achieved through the following technical solution: a storage medium stores a program, and when the program is executed by a processor, the motion control method of a wheel-legged robot for an unstructured environment described in the first purpose is achieved.

[0084] The fifth purpose of the present invention is achieved through the following technical solution: a computer device, comprising a processor and a memory for storing processor executable programs, characterized in that when the processor executes the program stored in the memory, the wheel-legged robot motion control method for unstructured environments described in the first purpose is implemented.

[0085] The above specific implementation modes are preferred embodiments of the present invention and cannot be used to limit the present invention. Any other changes or other equivalent replacement methods that do not deviate from the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. A motion control method for a wheel-legged robot in an unstructured environment, characterized in that: The following steps are involved: S100, constructing a wheel-legged robot simulation model, and building a learning and training scenario for the wheel-legged robot on a virtual simulation platform, and using a reinforcement learning algorithm to train a nonlinear controller model in a simulation environment; S200, converting the nonlinear controller model into a nonlinear controller that can be deployed and run on an actual physical prototype platform; S300, obtaining current motion state data of the wheel-legged robot; S400, inputting the current motion state data of the wheel-legged robot into the nonlinear controller to obtain the desired joint position and the desired wheel speed of the wheel-legged robot; S500: Input the desired joint positions and the desired wheel speeds into the low-level controller, thereby obtaining the joint torques for controlling the wheel-legged robot.

2. The motion control method of a wheel-legged robot for an unstructured environment according to claim 1, characterized in that: The learning training scene in step S100 is an unstructured environment, which includes flat ground, rough flat ground, flat slope, rough slope, flat single-sided bridge and rough single-sided bridge.

3. The motion control method of a wheel-legged robot for an unstructured environment according to claim 1, characterized in that: In step S100, obtaining a nonlinear controller model using a reinforcement learning algorithm in a simulation environment includes the following steps: S110, Model the motion control problem as a Marxian decision process; S120, defining a state space; S130, defining an action space; S140, defining a reward function; S150, set up automated course learning; S160, set domain randomization and add random noise; S170, setting training parameters, thereby completing the training to obtain a nonlinear controller model.

4. The motion control method of a wheel-legged robot for an unstructured environment according to claim 3, characterized in that: The reward function includes a command tracking reward function, a body stability reward function, an action smoothness reward function, a constraint reward function and an energy saving reward function.

5. The motion control method of a wheel-legged robot for an unstructured environment according to claim 1, characterized in that: Step S200 includes the following steps: S210, converting the file format of the nonlinear controller model into an intermediate format that is convenient for further processing; S220, generating executable code that does not depend on a specific library from the model in the converted format; S230, compile the executed code into a nonlinear controller suitable for a real-time control system of a wheel-legged robot.

6. The motion control method of a wheel-legged robot for an unstructured environment according to claim 1, characterized in that: The low-level controller includes any one or a combination of a PD controller and a P controller.

7. A wheel-legged robot motion control system for unstructured environments, characterized in that: include: A data acquisition module is used to obtain the current motion state data of the wheel-legged robot; A data processing module processes the current motion state data to obtain the joint torque for controlling the wheel-legged robot; An execution module is used to execute the wheel-legged robot motion control method for unstructured environments as described in any one of claims 1 to 6.

8. A wheel-legged robot device for unstructured environments, characterized in that: The invention comprises a wheel-legged robot body, a main control module, a communication module and an auxiliary module, wherein the body is provided with a sensor for collecting data, the sensor is connected to the communication module, and the main control module and the auxiliary module are both connected to the communication module, so as to realize the wheel-legged robot motion control method for an unstructured environment as described in any one of claims 1 to 6.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, the motion control method of a wheel-legged robot for an unstructured environment according to any one of claims 1 to 6 is implemented.

10. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that When the processor executes the program stored in the memory, the motion control method of the wheel-legged robot for an unstructured environment according to any one of claims 1 to 6 is implemented.

Images