Wheel-legged robot motion control method, system and device facing unstructured environment, storage medium and computer device

Abstract

The application discloses a wheel-legged robot motion control method, system and device facing an unstructured environment, a storage medium and computer equipment, and relates to the technical field of robot control. The control method comprises the following steps: constructing a simulation model of a wheel-legged robot, building a learning training scene of the wheel-legged robot, and training a nonlinear controller model by using a reinforcement learning algorithm; converting the nonlinear controller model into a nonlinear controller which can be deployed and run on an actual physical prototype platform; acquiring current motion state data of the wheel-legged robot and inputting the current motion state data into the nonlinear controller to obtain expected joint positions and expected wheel speeds of the wheel-legged robot; and inputting the expected joint positions and the expected wheel speeds into a low-level controller to output joint torques for controlling the wheel-legged robot to perform a control task. The application not only greatly shortens the development cycle of the wheel-legged robot controller, but also improves the robustness, dynamics and compliance of the wheel-legged robot in an unstructured environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robotics, in particular to a wheel-legged robot motion control method, system, device, storage medium and computer equipment for unstructured environment. BACKGROUND

[0002] Wheel-legged robots have a strong application prospect because they combine the advantages of fast speed and high energy efficiency of wheeled robots in flat terrain and strong adaptability of legged robots in complex terrain. However, since the wheel-legged robot is an unstable under-actuated system, there is a control problem of balancing and achieving highly dynamic motion in unstructured rugged terrain. In the related art, a linear quadratic regulator (LQR) or model predictive control (MPC) controller is constructed by simplifying the dynamic model to achieve balance control, and a proportional-integral-derivative controller (PID) is combined to achieve tasks such as variable leg length and turning. There is also a related technology that constructs a full-body dynamics controller to control the process of the wheel-legged robot performing various tasks.

[0003] However, the control scheme combining LQR or MPC and PID in the related art cannot deviate from the linearized balance point too much because the dynamic model is simplified, so the wheel-legged robot can only achieve a few specific motions in a simple environment. The scheme of constructing a full-body dynamics controller in the related art has certain dynamic motion capability in unstructured environments, but this scheme needs to establish a relatively accurate dynamic model and a very complex control framework, and requires a very time-consuming and laborious multi-task design and parameter tuning.

[0004] How to shorten the design cycle of the wheel-legged robot controller and enable the wheel-legged robot to achieve agile and dynamic motion in unstructured environments is a problem to be solved. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies of the above prior art, and to provide a wheel-legged robot motion control method for unstructured environment. The wheel-legged robot motion control method for unstructured environment can greatly shorten the design cycle of the wheel-legged robot controller and improve its motion capability in unstructured environments.

[0006] The second purpose of the present application is to provide a wheel-legged robot motion control system.

[0007] The third purpose of the present application is to provide a wheel-legged robot device.

[0008] A fourth object of the present application is to provide a storage medium.

[0009] A fifth object of the present application is to provide a computer device.

[0010] The first object of the present application is achieved by the following technical solution: a wheel-legged robot motion control method for unstructured environment, comprising the following steps:

[0011] S100, a simulation model of the wheel-legged robot is constructed, a learning training scene of the wheel-legged robot is built on a virtual simulation platform, and a nonlinear controller model is trained using a reinforcement learning algorithm in a simulation environment;

[0012] S200, the nonlinear controller model is converted into a nonlinear controller that can be deployed and run on an actual physical prototype platform;

[0013] S300, current motion state data of the wheel-legged robot is acquired;

[0014] S400, the current motion state data of the wheel-legged robot is input into the nonlinear controller to obtain expected joint positions and expected wheel speeds of the wheel-legged robot;

[0015] S500, the expected joint positions and the expected wheel speeds are input into a low-level controller to obtain joint torques for controlling the wheel-legged robot.

[0016] Preferably, the learning training scene in step S100 is an unstructured environment, and the unstructured environment includes flat ground, rough ground, flat slope, rough slope, flat single-edge bridge, and rough single-edge bridge.

[0017] Preferably, the step of obtaining the nonlinear controller model using the reinforcement learning algorithm in the simulation environment in step S100 comprises the following steps:

[0018] S110, modeling the motion control problem as a Markov decision process;

[0019] S120, defining a state space;

[0020] S130, defining an action space;

[0021] S140, defining a reward function;

[0022] S150, setting automated curriculum learning;

[0023] S160, setting domain randomization and adding random noise;

[0024] S170, setting training parameters to complete training to obtain the nonlinear controller model.

[0025] Preferably, the reward function comprises an instruction tracking reward function, a body stability reward function, a motion smoothness reward function, a constraint condition reward function and an energy saving reward function.

[0026] Preferably, the step S200 comprises the following steps:

[0027] S210, converting the file format of the nonlinear controller model into an intermediate format for further processing;

[0028] S220, generating executable code independent of specific libraries from the converted format model;

[0029] S230, compiling the executed code into a nonlinear controller suitable for the real-time control system of the wheel-legged robot.

[0030] Preferably, the low-level controller comprises any one or combination of a PD controller and a P controller.

[0031] The second object of the present application is achieved by the following technical solution: a wheel-legged robot motion control system for unstructured environments, comprising:

[0032] A data acquisition module for acquiring current motion state data of the wheel-legged robot;

[0033] A data processing module for processing the current motion state data to obtain joint torque for controlling the wheel-legged robot;

[0034] An execution module for executing the wheel-legged robot motion control method for unstructured environments of the first object.

[0035] The third object of the present application is achieved by the following technical solution: a wheel-legged robot device for unstructured environments, comprising a wheel-legged robot main body, a main control module, a communication module and an auxiliary module, the main body is provided with a sensor for collecting data, the sensor is connected with the communication module, the main control module and the auxiliary module are both connected with the communication module, and the wheel-legged robot motion control method for unstructured environments of the first object is realized.

[0036] The fourth object of the present application is achieved by the following technical solution: a storage medium storing a program, when the program is executed by a processor, the wheel-legged robot motion control method for unstructured environments of the first object is realized.

[0037] The fifth object of the application is achieved by the following technical solution: a computer device comprising a processor and a memory for storing programs executable by the processor, characterized in that the processor, when executing the programs stored in the memory, implements the motion control method of the wheel-legged robot facing unstructured environments according to the first object.

[0038] The present application has the following advantages over the prior art:

[0039] The present application obtains the nonlinear controller model of the robot by large-scale parallel training on virtual simulation software, and then converts the nonlinear controller model into a nonlinear controller that can be deployed on an actual prototype and deploys it to the actual prototype for control. This method greatly shortens the development cycle of the wheel-legged robot controller. In addition, the method builds many complex unstructured environments during the training phase of the controller, and optimizes the controller end-to-end by using the data generated by the interaction with the environment, so that the full-body force controller has robustness, dynamics and compliance that cannot be achieved by the controller designed based on traditional methods, and thus is more suitable for various complex unstructured environments. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a mechanical structure schematic diagram of the wheel-legged robot of the present application.

[0041] Figure 2 is a flowchart of the control method of the wheel-legged robot of the present application.

[0042] Figure 3 is a learning and training scene schematic diagram of the wheel-legged robot built on a simulation platform of the present application.

[0043] Figure 4 is a training framework schematic diagram of the nonlinear controller model of the wheel-legged robot trained using the reinforcement learning algorithm in the simulation environment of the present application.

[0044] Figure 5 is a flowchart of the nonlinear controller model of the wheel-legged robot trained using the reinforcement learning algorithm in the simulation environment of the present application.

[0045] Figure 6 is a state space schematic diagram defined when the nonlinear controller model of the wheel-legged robot is trained using the reinforcement learning algorithm in the simulation environment of the present application.

[0046] Figure 7 is a reward function definition table of the present application.

[0047] Figure 8 is a table explaining the meaning of symbols involved in the definition of the reward function of the present application.

[0048] Figure 9is a structural block diagram of a wheel-legged robot control system of the present application.

[0049] Figure 10 is a schematic diagram of a wheel-legged robot device of the present application. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the accompanying drawings and examples.

[0051] As Figure 1 shown, the wheel-legged robot adopted in the present embodiment includes a wheel-legged robot body 1 and two series wheel-legged structures symmetrically arranged about the sagittal plane. The series wheel-legged structure is composed of a thigh 2, a shank 3 and a driving wheel 4. The thigh 2 is connected with the body through a hip joint, the thigh 2 is connected with the shank 3 through a knee joint, and the shank 3 is connected with the driving wheel 4 through a driving wheel joint. Each wheel-legged structure contains three independent rotation degrees of freedom of the hip joint, the knee joint and the driving wheel 3, the knee joint motor and the hip joint motor are coaxially arranged on the body, the knee joint motor is connected with the body through a parallel four-bar mechanism shown in Figure 1 , and the driving wheel is also arranged with a rotary motor. In addition, an inertial measurement unit (IMU) is mounted at the wheel-legged robot body for obtaining the attitude information of the wheel-legged robot body, and each rotary motor is attached with a position encoder for obtaining the motor position and rotation speed information.

[0052] As Figure 2 shown, the wheel-legged robot motion control method for unstructured environment includes the following steps:

[0053] S100, a simulation model of the wheel-legged robot is constructed, a learning and training scene of the wheel-legged robot is built on a virtual simulation platform, and a nonlinear controller model is trained in the simulation environment using a reinforcement learning algorithm;

[0054] Among them, the simulation model of the wheel-legged robot is imported into the simulation platform as an agent for training the reinforcement learning algorithm, the simulation model adopts the URDF format, has similar mass, inertia and joint limit, joint motor torque, rotation speed and other constraints to the actual prototype. The simulation platform adopted in the present embodiment is Nvidia Isaac Gym (the proprietary name of the simulation platform), which has GPU acceleration architecture and domain randomization function, can significantly speed up the training of the agent through parallelization, and learn a robust strategy by introducing environmental changes in the training process, and then facilitate the transfer to the real world.

[0055] As Figure 3As shown, the learning training scene in step S100 is an unstructured environment, which includes flat ground, rough ground, flat slope, rough slope, flat single-edge bridge and rough single-edge bridge. Rough terrain is relative to flat terrain, and the rough terrain surface is rugged ground. For example, the height range of rough terrain is [-0.01, 0.01]m, and the minimum height interval is 0.005m, which is generated by a series of equidistant two-dimensional grids with grid vertex heights in the range. In addition, in order to carry out terrain curriculum learning, different difficulties are set for the terrain, and as the terrain difficulty increases, the slope of the slope or the height of the single-edge bridge also increases.

[0056] The training framework of the embodiment is as shown in Figure 4 , wherein the contents indicated by the dashed arrows and dashed boxes only exist in training, and the contents indicated by the solid arrows and solid boxes exist in both training and deployment stages. The embodiment adopts a reinforcement learning algorithm of asymmetric actor-critic structure, proximal policy optimization (PPO), and in the specific implementation process, an open source PPO framework from Robotics Systems Lab of Swiss Federal Institute of Technology is adopted. As shown in Figure 5 , the following steps are included in step S100 of obtaining a nonlinear controller model using a reinforcement learning algorithm in a simulation environment:

[0057] S110, model the motion control problem as a Markov decision process; the Markov decision process is a mathematical framework for describing discrete-time control processes, in which the evolution of states and results is partially random; MDP is defined by a four-tuple (s, a, p, r) containing state space S, action space A, reward function R(s t ,s t+1 ) and transition probability P(s t+1 |s t ,a t ). The agent selects an action from the policy π(a t |s t ) according to the current state s t , interacts with the environment, and gets an immediate reward r t , and the goal of RL is to find an optimal policy π * that maximizes the total discounted reward in the infinite time domain, i.e.

[0058]

[0059] , wherein γ∈(0, 1) is a discount factor for adjusting the weight of immediate reward and long-term reward, and ρπ To define the trajectory distribution under the policy p.

[0060] S120, define the state space; the embodiment adopts an asymmetric actor-critic structure, and divides the state space into observation values and privileged information. The observation values include proprioceptive observation values and given instructions. The proprioceptive observation values refer to state information provided by a robot itself sensor, and therefore, the state quantities can be obtained through an IMU, a motor position encoder and the like sensor in actual deployment. The privileged information refers to information that can only be obtained in a simulation training stage, such as terrain height information, a friction coefficient and the like. In the simulation training, both an actor network and a critic network can access the observation values, but only the critic network can access both the observation values and the privileged information. The embodiment adopts a state space defined as shown in Figure 6

[0061] S130, define the action space; in the embodiment, the dimension of the action space is equal to the degree of freedom of the wheel-legged robot. For the knee and hip joints, the output action is regarded as a desired joint position, which is tracked by a PD controller. For the driving wheel joint, the output action is regarded as a desired wheel speed, which is tracked by a P controller.

[0062] S140, define the reward function; the reward function includes an instruction tracking reward function, a body stability reward function, an action smoothness reward function, a constraint condition reward function and an energy saving reward function. The specific reward function definition is as shown in Figure 7 Figure 8

[0063] The instruction tracking reward function is to make the wheel-legged robot track the human given instruction, including a body linear velocity tracking reward function, a body angular velocity tracking reward function and a body height tracking reward function.

[0064] The body stability reward function is to make the wheel-legged robot always keep the body horizontal and stable during movement, including a body attitude angle reward function, a body attitude angular velocity reward function, a body linear velocity reward function in the direction of gravity and a projected gravity reward function.

[0065] The action smoothness reward function is to make the action of the wheel-legged robot more smooth, reduce vibration and mutation, including a joint velocity reward function, a joint acceleration reward function, an action change rate reward function and an action smoothness reward function.

[0066] The constraint condition reward function is mainly to introduce some constraint conditions existing in the actual physical prototype, such as joint upper and lower limit positions, during the training process, and therefore mainly includes a joint limit position reward function and a collision reward function.​​​

[0067] The energy-saving reward function is to make the wheel-legged robot adopt a more energy-saving motion mode and avoid damaging the hardware due to the output torque exceeding the limit. The energy-saving reward function includes a torque reward function and an energy reward function.

[0068] S150, set an automatic course learning; as a preferred embodiment of the present application, a game-inspired automatic course learning is adopted for various different unstructured terrains. For example, during training, all robots are randomly assigned to a terrain type and a level representing the difficulty of the terrain. For the slope terrain, the slope increases from 0 degrees to 15 degrees, and for the single-side bridge terrain, the height increases from 0 cm to 8 cm. If the robot successfully crosses the terrain, it is reset to a more difficult terrain next time; if the distance moved is less than half the distance required by the target speed within a certain time, the terrain level will be reduced. When the robot successfully passes through the highest level terrain, a level of terrain is randomly selected for training in the next round of training.

[0069] S160, set domain randomization and add random noise; in order to enhance the robustness and generalization of the controller obtained by training, thereby facilitating the sim-to-real transfer, the domain randomization technology and the random noise adding technology are adopted in the training process of the present application. For example, the domain randomization technology randomly changes some kinematics and dynamics parameters during training, including: low-level PD controller coefficients, friction coefficients, collision restitution coefficients, robot component masses, robot component inertias, robot center of mass positions, torque command delay, and robot default joint positions. Random noise mainly includes: observation noise, torque noise, and random thrust disturbance.

[0070] S170, set training parameters to complete training to obtain a nonlinear controller model. The training parameters include the maximum number of iterations, the number of robots for parallel training, etc. The controller of the wheel-legged robot obtained by training in the present embodiment is essentially a neural network model, the input of which is the proprioceptive observation value accessible to the actor network during training, and the output dimension is the same as the defined action state space dimension. For example, the nonlinear controller model is saved in.pt format.

[0071] S200, convert the nonlinear controller model into a nonlinear controller that can be deployed and run on an actual physical prototype platform; step S200 includes the following steps:

[0072] S210, convert the file format of the nonlinear controller model to an intermediate format for further processing; in the present embodiment, the nonlinear controller model in.pt format is converted to.mat intermediate format;

[0073] S220, the model in the conversion format is generated executable code not dependent on a specific library; specifically, the embodiment uses a MATLAB toolbox to convert the model in the.mat format to C language code not dependent on any library, which is a specific program implementation of the nonlinear controller of the wheel-legged robot in actual deployment.

[0074] S230, the code for execution is compiled into a nonlinear controller suitable for a real-time control system of a wheel-legged robot. The embodiment cross-compiles the C code into a nonlinear controller that can run in a real-time control system based on an ARM architecture.

[0075] S300, current motion state data of the wheel-legged robot is acquired; the current motion state data is used to represent the motion characteristics of the wheel-legged robot. The input of the nonlinear controller of the wheel-legged robot is the proprioceptive observation value and the given instruction. In the embodiment, the required state quantity or the given instruction of the wheel-legged robot can be acquired in real time from sensors such as IMU and position encoder or other ways.

[0076] S400, the current motion state data of the wheel-legged robot is input to the nonlinear controller to obtain the expected joint position and the expected wheel speed of the wheel-legged robot; the expected joint position and the expected wheel speed are used as the input of the low-level controller.

[0077] S500, the expected joint position and the expected wheel speed are input to the low-level controller to obtain the joint torque for controlling the wheel-legged robot. Specifically, in the embodiment, the expected joint position includes the expected position of the knee joint and the expected position of the hip joint, and the low-level controller includes a PD controller and a P controller. The expected positions of the knee joint and the hip joint are input to the respective PD controllers, and the expected joint speed in the PD controller is set to 0, so that the required control torque of the knee joint and the hip joint is calculated; similarly, the expected wheel speed is input to the P controller, and the required control torque of the driving wheel is obtained through operation.

[0078] The second object of the application is achieved by the following technical scheme: as shown in the figure, the wheel-legged robot motion control system facing an unstructured environment comprises: Figure 9

[0079] The data acquisition module is used to acquire the current motion state data of the wheel-legged robot; the current motion state data specifically includes the attitude angle and attitude angular velocity information returned by the IMU, the motor position and speed information obtained by six motor position encoders, the given instruction information, and the action value output by the last model.

[0080] The data processing module processes the current motion state data to obtain the joint torque for controlling the wheel-legged robot. ​

[0081] The execution module is used for executing the wheel-legged robot motion control method facing an unstructured environment.

[0082] The third object of the application is achieved by the technical scheme as shown in the following. Figure 10 The wheel-legged robot device facing an unstructured environment includes a wheel-legged robot body, a main control module, a communication module and an auxiliary module. The body is provided with a sensor for collecting data. The sensor is connected with the communication module. The main control module and the auxiliary module are both connected with the communication module, and the wheel-legged robot motion control method facing an unstructured environment is realized. Specifically, the sensor includes an IMU and a motor position encoder. The IMU is used for obtaining attitude information of the wheel-legged robot body, and the motor position encoder is used for obtaining motor position and speed information. The main control module is a main control PCB board including a processor and a memory. The main control PCB board includes an STM32F405RGT6 chip. The communication module is used for information interaction between modules, and is responsible for tasks such as reading of sensor signals and issuing of control signals. In the embodiment of the application, the IMU communicates with the main control PCB board through a serial port line, and each motor communicates with the main control PCB board through a CAN bus protocol. The auxiliary module refers to a module responsible for other tasks. In the embodiment of the application, the auxiliary module includes a power supply module and a heat dissipation module. The power supply module adopts a lithium battery with 24V DC voltage output, and the heat dissipation module adopts two 24V DC voltage input heat dissipation fans with a size of 40mm*40mm*10mm.

[0083] The fourth object of the application is achieved by the technical scheme as shown in the following. A storage medium stores a program. When the program is executed by a processor, the wheel-legged robot motion control method facing an unstructured environment is realized.

[0084] The fifth object of the application is achieved by the technical scheme as shown in the following. A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the wheel-legged robot motion control method facing an unstructured environment is realized.

[0085] The above specific embodiments are preferred embodiments of the application, and cannot limit the application. Any changes or other equivalent replacement manners without departing from the technical scheme of the application are included in the protection scope of the application.

Claims

1. A method for motion control of a wheel-legged robot facing an unstructured environment, characterized in that, The method comprises the following steps: S100, a simulation model of the wheel-legged robot is constructed, a learning training scene of the wheel-legged robot is built on a virtual simulation platform, and a nonlinear controller model is trained in the simulation environment using a reinforcement learning algorithm; S200, the nonlinear controller model is converted into a nonlinear controller that can be deployed and run on an actual physical prototype platform; S300, current motion state data of the wheel-legged robot is acquired; S400, the current motion state data of the wheel-legged robot is input into the nonlinear controller to obtain expected joint positions and expected wheel speeds of the wheel-legged robot; S500, the expected joint positions and the expected wheel speeds are input into a low-level controller to obtain joint torques for controlling the wheel-legged robot.

2. The motion control method of the wheel-legged robot facing unstructured environment according to claim 1, wherein, The learning training scene in step S100 is an unstructured environment, and the unstructured environment includes flat ground, rough ground, flat slopes, rough slopes, flat single-edge bridges, and rough single-edge bridges.

3. The motion control method of the wheel-legged robot facing unstructured environment according to claim 1, wherein, The nonlinear controller model is obtained in the simulation environment using the reinforcement learning algorithm in step S100, and the method comprises the following steps: S110, a motion control problem is modeled as a Markov decision process; S120, a state space is defined; S130, an action space is defined; S140, a reward function is defined; S150, an automated curriculum learning is set; S160, domain randomization is set, and random noise is added; S170, training parameters are set, and thus the nonlinear controller model is obtained through training.

4. The motion control method of the wheel-legged robot facing unstructured environment according to claim 3, characterized in that, The reward function comprises an instruction tracking reward function, a body stability reward function, an action smoothness reward function, a constraint condition reward function, and an energy saving reward function.

5. The motion control method of the wheel-legged robot facing unstructured environment according to claim 1, wherein, Step S200 comprises the following steps: S210, a file format of the nonlinear controller model is converted into an intermediate format for further processing; S220, the model in the converted format is generated into executable code that does not depend on a specific library; S230, the executed code is compiled into a nonlinear controller suitable for a real-time control system of the wheel-legged robot.

6. The motion control method of the wheel-legged robot facing unstructured environment according to claim 1, wherein, The low-level controller comprises any one or a combination of a PD controller and a P controller.

7. A motion control system for a wheel-legged robot facing an unstructured environment, characterized in that, The method comprises the following steps: a data acquisition module for acquiring current motion state data of the wheel-legged robot; a data processing module for processing the current motion state data to obtain joint torques for controlling the wheel-legged robot; an execution module for executing the wheel-legged robot motion control method for unstructured environments according to any one of claims 1-6.

8. A wheel-legged robot apparatus oriented to an unstructured environment, characterized by, The wheel-legged robot comprises a main body, a main control module, a communication module, and an auxiliary module, the main body is provided with a sensor for collecting data, the sensor is connected with the communication module, the main control module and the auxiliary module are both connected with the communication module, and the wheel-legged robot motion control method for unstructured environments according to any one of claims 1-6 is realized.

9. A storage medium storing a program, characterized by comprising: The program is executed by the processor to realize the wheel-legged robot motion control method for unstructured environments according to any one of claims 1-6.

10. Computer device comprising a processor and a memory for storing a processor executable program, characterized in that, The processor executes the program stored in the memory to realize the wheel-legged robot motion control method for unstructured environments according to any one of claims 1-6.

Images