Training method and device for quadruped robot controller based on reinforcement learning

Abstract

The present application provides a training method, device and electronic device for a quadruped robot controller based on reinforcement learning, the method comprising: training a first controller according to first sensor data, first elevation map data and a first target control signal to obtain a trained first target controller; controlling the quadruped robot to execute a second target control signal in a simulation environment to determine second sensor data, second elevation map data and RGB image data; constructing training data according to target position information of each joint, RGB image data, second sensor data and second target control signal; training the second controller to be trained according to the training data to obtain a trained second target controller, thereby, by using RGB image data as visual perception information, it can be more adaptable to complex lighting and obstacle environments, and can control the quadruped robot more accurately and efficiently.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a training method, device and electronic equipment for a quadruped robot controller based on reinforcement learning. Background Art

[0002] Quadruped robots have great potential applications in various tasks such as various inspections, rescue and post-disaster exploration. In order to cope with the complex terrain environment in the real environment, it is usually necessary to integrate visual perception information into the control algorithm of the quadruped robot. In related technologies, the commonly used method is to use depth cameras or lidar to obtain visual perception information, but the above methods often cannot accurately obtain visual perception information, resulting in the inability to more accurately and flexibly control the quadruped robot. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] According to a first aspect of the present application, a training method for a quadruped robot controller based on reinforcement learning is provided, comprising: controlling the quadruped robot to execute a first target control signal in a simulation environment, and determining first sensor data and first elevation map data; training the first controller according to the first sensor data, the first elevation map data and the first target control signal to obtain a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network; controlling the quadruped robot to execute a second target control signal in a simulation environment, and determining second sensor data, second elevation map data and RGB image data; inputting the second sensor data, the second elevation map data and the second target control signal into the first target controller to obtain target position information of each joint of the quadruped robot; constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal; training the second controller to be trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a second target external perception network and a second target controller network.

[0005] According to a second aspect of the present application, a training device for a quadruped robot controller based on reinforcement learning is provided, comprising: a first determination module, used to control the quadruped robot to execute a first target control signal in a simulation environment, and determine first sensor data and first elevation map data; a first training module, used to train the first controller according to the first sensor data, the first elevation map data and the first target control signal, to obtain a trained first target controller, wherein the first target controller comprises a first target internal perception network, a first target external perception network and a first target controller network; a second determination module, used to control the quadruped robot to execute a second target control signal in a simulation environment, and determine the second sensor data , second elevation map data and RGB image data; a first acquisition module, used to input the second sensor data, the second elevation map data and the second target control signal into the first target controller, and obtain the target position information of each joint of the quadruped robot; a second acquisition module, used to construct training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal; a second training module, used to train the second controller to be trained according to the training data, and obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a second target external perception network and a second target controller network.

[0006] The third aspect of the present application proposes an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements the training method of the quadruped robot controller based on reinforcement learning as described in the first aspect.

[0007] The fourth aspect of the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the quadruped robot controller based on reinforcement learning described in the first aspect.

[0008] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which, when executed by a processor, implements the training method for a quadruped robot controller based on reinforcement learning according to the first aspect.

[0009] The technical solution provided by the embodiments of the present application includes at least the following beneficial effects:

[0010] The present application provides a training method for a quadruped robot controller based on reinforcement learning. By using RGB image data as visual perception information, it can better adapt to complex lighting and obstacle environments, and can obtain more ground information. It can control the quadruped robot more accurately and efficiently, laying a solid foundation for the subsequent quadruped robot to safely and smoothly perform tasks.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0013] Figure 1 A flowchart of a training method for a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application;

[0014] Figure 2 A flowchart of another method for training a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application;

[0015] Figure 3 A flowchart of another method for training a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of a process for obtaining predicted position information of each joint provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of the structure of a training device for a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] The following embodiment is used to explain in detail the training method of the quadruped robot controller based on reinforcement learning of the present application.

[0021] Figure 1 A flowchart of a method for training a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application.

[0022] like Figure 1 As shown, the training method of the quadruped robot controller based on reinforcement learning proposed in this embodiment specifically includes the following steps:

[0023] S101. Controlling a quadruped robot to execute a first target control signal in a simulation environment to determine first sensor data and first elevation map data.

[0024] The first target control signal includes the target speed and target angular velocity

[0025] It should be noted that in each simulation step t, the movement of the quadruped robot and the contact dynamics between the quadruped robot and the ground are simulated in the simulation environment, and the first sensor data obtained by the quadruped robot executing the first target control signal in the simulation environment is obtained. And the first elevation map data .

[0026] The first sensor data includes but is not limited to inertial measurement unit (IMU) data and joint encoder data.

[0027] It should be noted that IMU data includes but is not limited to: body speed 、Body angular velocity: and fuselage posture ,The body posture is expressed by the projection of the direction vector of gravity in the robot’s own coordinate system.

[0028] It should be noted that the joint encoder data includes but is not limited to: Joint angle: and joint angular velocity .

[0029] It should be noted that the present application does not limit the specific method for obtaining the first elevation map data. Optionally, the first elevation map data can be obtained through a depth camera or a lidar.

[0030] S102. Train the first controller according to the first sensor data, the first elevation map data and the first target control signal to obtain a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network.

[0031] In an embodiment of the present application, the first sensor data can be input into the first internal perception network in the first controller to obtain a first coding vector, and the first elevation map data can be input into the first external perception network in the first controller to obtain a second coding vector. The first target vector, the first coding vector and the second coding vector corresponding to the first target control signal are spliced ​​to obtain a first spliced ​​vector. The first spliced ​​vector is input into the first controller network in the first controller to obtain the first predicted position information of each joint of the quadruped robot and the actual control signal of the quadruped robot. The first controller is trained according to the actual control signal and the first target control signal to obtain a first target controller.

[0032] S103, controlling the quadruped robot to execute a second target control signal in a simulation environment, and determining second sensor data, second elevation map data, and RGB image data.

[0033] It should be noted that in the related technologies, the visual perception information is obtained based on the depth camera and the lidar. The depth camera is prone to failure in strong light or low light environment, especially when encountering strong reflection or shadow area, the visual perception information is often not accurate enough, smooth, transparent or highly reflective materials will cause the depth camera to be unable to correctly obtain the depth information of the surface, resulting in large errors. In complex environments full of dust, smoke or water mist, the signal of the depth camera is easily interfered, resulting in blurred images or incomplete depth data, especially in scenes such as mines and fire rescue. The detection range of the lidar is limited, especially in multiple obstacles or complex environments, it is easy to produce blind spots. In addition, the laser beam will be affected when penetrating obstacles, resulting in ranging errors. Although the depth camera and lidar can provide the geometric structure of the terrain, they cannot evaluate the physical and mechanical properties of the ground. For example, when a quadruped robot walks on grass or slippery ground, the friction coefficient, hardness and other parameters of the ground will directly affect the movement and stability of the quadruped robot. For soft or unstable ground, such as sand and snow, existing perception methods lack the ability to identify the risk of collapse of these terrains, which may cause the quadruped robot to mistakenly enter dangerous areas and affect the execution of the task. In order to make up for the limitations of a single sensor, it is usually necessary to fuse information from multiple sensors such as depth cameras and lidars, which increases the complexity and computational burden of the system. Especially in real-time applications, data processing delays may cause the quadruped robot to respond untimely. The hardware cost of depth cameras and lidars is relatively high, especially high-precision lidars, making large-scale deployment of equipment difficult to achieve in some application scenarios.

[0034] In an embodiment of the present application, a camera on a quadruped robot can be used to collect red, green, and blue (RGB) image data. The camera on the quadruped robot can be used to directly imitate human visual perception to adapt to more complex lighting and obstacle environments, and more ground information can be obtained.

[0035] Optionally, since it is necessary to obtain RGB image data, a physical engine with rendering capabilities can be used as a basis for building a simulation environment.

[0036] For example, you can use Isaac Lab or a physics engine + rendering engine, such as Multi-Joint dynamics with Contact (MuJoCo) + UnReal Engine.

[0037] The second elevation map data and the RGB image data correspond to the same time.

[0038] S104, inputting the second sensor data, the second elevation map data and the second target control signal into the first target controller to obtain the target position information of each joint of the quadruped robot.

[0039] In an embodiment of the present application, the second sensor data can be input into the internal perception network of the first target to obtain an internal coding vector, and the second elevation map data can be input into the external perception network of the first target to obtain an external coding vector. The target vector, the internal coding vector and the external coding vector corresponding to the second target control signal are spliced ​​to obtain a spliced ​​vector, and the spliced ​​vector is input into the first target controller network to output the target position information of each joint of the quadruped robot.

[0040] For example, taking a 12-degree-of-freedom quadruped robot as an example, output the target position information of each joint of the quadruped robot .

[0041] S105 , constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal.

[0042] In an embodiment of the present application, the first target controller takes the elevation map data calculated based on the depth camera or lidar data as input, but the second controller takes the RGB image data collected by the RGB camera as input. The output data of the first target controller, that is, the first target controller, is used as a label, and imitation learning is used to train the second controller with RGB image data as input.

[0043] In an embodiment of the present application, the target position information, RGB image data, second sensor data and second target control signal of each joint can be associated to construct training data.

[0044] S106. Train the second controller to be trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a second target external perception network, and a second target controller network.

[0045] In the embodiment of the present application, after the training data is acquired, the second controller to be trained can be trained according to the training data to obtain a trained second target controller.

[0046] The training method of a quadruped robot controller based on reinforcement learning provided in the present application controls the quadruped robot to execute a first target control signal in a simulation environment, determines first sensor data and first elevation map data, trains the first controller according to the first sensor data, the first elevation map data and the first target control signal, and obtains a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network, controls the quadruped robot to execute a second target control signal in a simulation environment, determines second sensor data, second elevation map data and RGB image data, and inputs the second sensor data, the second elevation map data and the second target control signal into the first target controller In the process, the target position information of each joint of the quadruped robot is obtained, and training data is constructed according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal. According to the training data, the second controller to be trained is trained to obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a second target external perception network and a second target controller network. Therefore, the present application uses RGB image data as visual perception information, which can better adapt to complex lighting and obstacle environments, and can obtain more ground information, and can control the quadruped robot more accurately and efficiently, laying a solid foundation for the subsequent quadruped robot to safely and smoothly perform tasks.

[0047] Figure 2 A flowchart of a method for training a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application.

[0048] like Figure 2 As shown, the training method of the quadruped robot controller based on reinforcement learning proposed in this embodiment specifically includes the following steps:

[0049] S201. Controlling a quadruped robot to execute a first target control signal in a simulation environment, and determining first sensor data and first elevation map data.

[0050] The step S201 may be implemented by any implementation method in the embodiments of the present application, and will not be described in detail here.

[0051] S202, inputting the first sensor data into the first internal perception network in the first controller to obtain a first encoding vector, and inputting the first elevation map data into the first external perception network in the first controller to obtain a second encoding vector.

[0052] In the embodiment of the present application, after obtaining the first sensor data and the first elevation map data , the first sensor data Input to the first internal perception network, the first internal perception network encodes the sensor data to obtain a first encoding vector , the first elevation map data Input to the first external perception network, the first external perception network encodes the first elevation map data to obtain the second encoding vector .

[0053] S203. Concatenate the first target vector, the first encoding vector and the second encoding vector corresponding to the first target control signal to obtain a first concatenated vector, input the first concatenated vector into a first controller network in the first controller, and obtain first predicted position information of each joint of the quadruped robot and an actual control signal of the quadruped robot.

[0054] S204: Train the first controller according to the actual control signal and the first target control signal to obtain a first target controller.

[0055] In an embodiment of the present application, a reward function of the first controller can be obtained based on the actual control signal and the first target control signal, and the parameters of the first controller can be adjusted according to the reward function until the reward function meets the training end condition, and the first controller after the last parameter adjustment is determined as the first target controller.

[0056] Optionally, when the reward function value of the first controller reaches a preset threshold, it is determined that the reward function meets the training end condition.

[0057] In the embodiment of the present application, the target speed can be obtained from the first target control signal and target angular velocity , and obtain the first speed from the actual control signal , first angular velocity and torque , according to the target speed and actual speed , determine the speed reward function , and obtain the speed reward weight of the speed reward function , according to the target angular velocity and the actual angular velocity , determine the angular velocity reward function , and obtain the angular velocity reward weight of the angular velocity reward function , according to the actual torque , determine the moment reward function , and obtain the moment reward weight of the moment reward function , according to the speed reward function and speed reward weight, the angular velocity reward function and angular velocity reward weight, the torque reward function and the torque reward weight, the reward function of the first controller is obtained.

[0058] For example, the reward function of the first controller can be determined according to the following formula: :

[0059] in, is the reward function of the first controller, is the speed reward function, The speed reward weight, is the angular velocity reward function and is the angular velocity reward weight, is the moment reward function and is the moment reward weight.

[0060] S205. Control the quadruped robot to execute the second target control signal in the simulation environment, and determine the second sensor data, the second elevation map data, and the RGB image data.

[0061] S206, inputting the second sensor data, the second elevation map data and the second target control signal into the first target controller to obtain the target position information of each joint of the quadruped robot.

[0062] S207, constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal.

[0063] The steps S205-S207 may be implemented in any of the implementation methods in the embodiments of the present application and will not be described in detail here.

[0064] S208. Input the RGB image data, the second sensor data, and the second target control signal into the second controller to obtain the predicted position information of each joint of the quadruped robot.

[0065] In the embodiments of the present application, Figure 3 As shown, the second sensor data can be input into the second internal perception network in the second controller to obtain a third coding vector, and the RGB image data can be input into the second external perception network in the second controller to obtain a fourth coding vector, the second target vector, the third coding vector and the fourth coding vector corresponding to the second target control signal are spliced ​​to obtain a second spliced ​​vector, and the second spliced ​​vector is input into the second controller network in the second controller to obtain the predicted position information of each joint of the quadruped robot.

[0066] S209: Train the second controller according to the predicted position information of each joint and the target position information of each joint to obtain a trained second target controller.

[0067] In an embodiment of the present application, a loss function can be obtained based on the predicted position information of each joint and the target position information of each joint, and the parameters of the second controller can be adjusted based on the loss function until the loss function meets the training end conditions, and the second controller after the last parameter adjustment is determined as the second target controller.

[0068] For example, when the loss function value reaches a preset threshold, it is determined that the loss function meets the training end condition.

[0069] It should be noted that after the trained second target controller is obtained, the quadruped robot can be controlled based on the second target controller.

[0070] For example, Figure 4 As shown, the quadruped robot sensor data, RGB image data and target speed are input into the second target controller, and the second target controller outputs the position information of each joint, and the quadruped robot is controlled based on the position information of each joint.

[0071] The present application provides a training method for a quadruped robot controller based on reinforcement learning, which controls the quadruped robot to execute a first target control signal in a simulation environment, determines first sensor data and first elevation map data, inputs the first sensor data into a first internal perception network in the first controller to obtain a first coding vector, and inputs the first elevation map data into a first external perception network in the first controller to obtain a second coding vector, splices the first target vector, the first coding vector and the second coding vector corresponding to the first target control signal to obtain a first splicing vector, inputs the first splicing vector into a first controller network in the first controller, obtains first predicted position information of each joint of the quadruped robot and an actual control signal of the quadruped robot, trains the first controller according to the actual control signal and the first target control signal to obtain a first target controller, trains the first controller according to the actual control signal and the first target control signal to obtain a first target controller, and inputs the second sensor data, the second elevation map data and the second target control signal into a first external perception network in the first controller to obtain a second coding vector. The target control signal is input into the first target controller to obtain the target position information of each joint of the quadruped robot, and the training data is constructed according to the target position information of each joint, the RGB image data, the second sensor data, and the second target control signal. The RGB image data, the second sensor data, and the second target control signal are input into the second controller to obtain the predicted position information of each joint of the quadruped robot. The second controller is trained according to the predicted position information of each joint and the target position information of each joint to obtain a trained second target controller. Therefore, the present application uses RGB image data as visual perception information to better adapt to complex lighting and obstacle environments, and can obtain more ground information. By inputting RGB image data as visual perception information into the second target controller, the quadruped robot can be controlled more accurately and efficiently, reducing the complexity and cost of obtaining visual perception information, and laying a solid foundation for the subsequent quadruped robot to safely and smoothly perform tasks.

[0072] In order to implement the above embodiment, this embodiment provides a training device for a quadruped robot controller based on reinforcement learning. Figure 5 A schematic diagram of the structure of a training device for a quadruped robot controller based on reinforcement learning provided in an embodiment of the present application.

[0073] like Figure 5 As shown, the training device 1000 of the quadruped robot controller based on reinforcement learning includes: a first determination module 110, a first training module 120, a second determination module 130, a first acquisition module 140, a second acquisition module 150 and a second training module 160.

[0074] A first determination module 110 is used to control the quadruped robot to execute a first target control signal in a simulation environment, and determine first sensor data and first elevation map data;

[0075] A first training module 120 is used to train a first controller according to the first sensor data, the first elevation map data and the first target control signal to obtain a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network;

[0076] A second determination module 130 is used to control the quadruped robot to execute a second target control signal in a simulation environment, and determine second sensor data, second elevation map data, and RGB image data;

[0077] A first acquisition module 140 is used to input the second sensor data, the second elevation map data and the second target control signal into a first target controller to acquire target position information of each joint of the quadruped robot;

[0078] A second acquisition module 150, configured to construct training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal;

[0079] The second training module 160 is used to train the second controller to be trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a second target external perception network and a second target controller network.

[0080] According to one embodiment of the present application, the first training module 120 is also used to: input the first sensor data into the first internal perception network in the first controller to obtain a first encoding vector, and input the first elevation map data into the first external perception network in the first controller to obtain a second encoding vector; splice the first target vector, the first encoding vector and the second encoding vector corresponding to the first target control signal to obtain a first splicing vector, and input the first splicing vector into the first controller network in the first controller to obtain the first predicted position information of each joint of the quadruped robot and the actual control signal of the quadruped robot; train the first controller according to the actual control signal and the first target control signal to obtain the first target controller.

[0081] In one embodiment of the present application, the first training module 120 is further used to: obtain a reward function of the first controller based on the actual control signal and the first target control signal; adjust parameters of the first controller according to the reward function until the reward function meets the training end condition, and determine the first controller after the last parameter adjustment as the first target controller.

[0082] In one embodiment of the present application, the first training module 120 is also used to: obtain the target speed and target angular velocity from the first target control signal, and obtain the actual speed, actual angular velocity and actual torque from the actual control signal; determine the speed reward function based on the target speed and the actual speed, and obtain the speed reward weight of the speed reward function; determine the angular velocity reward function based on the target angular velocity and the actual angular velocity, and obtain the angular velocity reward weight of the angular velocity reward function; determine the torque reward function based on the actual torque, and obtain the torque reward weight of the torque reward function; obtain the reward function of the first controller based on the speed reward function and the speed reward weight, the angular velocity reward function and the angular velocity reward weight, the torque reward function and the torque reward weight.

[0083] In one embodiment of the present application, the second acquisition module 150 is further used to: associate the target position information of each joint, the RGB image data, the second sensor data and the second target control signal to construct training data.

[0084] In one embodiment of the present application, the second training module 160 is further used to: input the RGB image data, the second sensor data, and the second target control signal into the second controller to obtain the predicted position information of each joint of the quadruped robot; train the second controller according to the predicted position information of each joint and the target position information of each joint to obtain a trained second target controller.

[0085] In one embodiment of the present application, the second training module 160 is further used to: obtain a loss function based on the predicted position information of each joint and the target position information of each joint; adjust the parameters of the second controller according to the loss function until the loss function meets the training end condition, and determine the second controller after the last parameter adjustment as the second target controller.

[0086] In one embodiment of the present application, the second training module 160 is also used to: input the second sensor data into the second internal perception network in the second controller to obtain a third coding vector, and input the RGB image data into the second external perception network in the second controller to obtain a fourth coding vector; splice the second target vector corresponding to the second target control signal, the third coding vector and the fourth coding vector to obtain a second splicing vector, and input the second splicing vector into the second controller network in the second controller to obtain the predicted position information of each joint of the quadruped robot.

[0087] The training device of the quadruped robot controller based on reinforcement learning provided in the present application controls the quadruped robot to execute a first target control signal in a simulation environment, determines first sensor data and first elevation map data, trains the first controller according to the first sensor data, the first elevation map data and the first target control signal, and obtains a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network, controls the quadruped robot to execute a second target control signal in a simulation environment, determines second sensor data, second elevation map data and RGB image data, inputs the second sensor data, the second elevation map data and the second target control signal into the first target controller, obtains target position information of each joint of the quadruped robot, and obtains target position information of each joint of the quadruped robot according to the first sensor data, the first elevation map data and the first target control signal. According to the target position information of each joint, RGB image data, second sensor data and second target control signal, training data is constructed, and the second controller to be trained is trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target external perception network, a first target external perception network and a first target controller network. Thus, by using RGB image data as visual perception information, it can better adapt to complex lighting and obstacle environments, and can obtain more ground information. By inputting RGB image data as visual perception information into the second target controller, the quadruped robot can be controlled more accurately and efficiently, reducing the complexity and cost of obtaining visual perception information, and laying a solid foundation for the subsequent quadruped robot to safely and smoothly perform tasks.

[0088] In order to implement the above embodiment, the present application also proposes an electronic device 2000, such as Figure 6 As shown, it includes: a memory 210, a processor 220, and a computer program stored in the memory 210 and executable on the processor 220. When the processor executes the program, the training method of the quadruped robot controller based on reinforcement learning as described in the first aspect is implemented.

[0089] In order to implement the above-mentioned embodiments, the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the quadruped robot controller based on reinforcement learning described in the first aspect.

[0090] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the quadruped robot controller based on reinforcement learning described in the first aspect.

[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0092] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A training method for a quadruped robot controller based on reinforcement learning, characterized in that: The method comprises: Controlling the quadruped robot to execute a first target control signal in a simulation environment to determine first sensor data and first elevation map data; According to the first sensor data, the first elevation map data and the first target control signal, a first controller is trained to obtain a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network; Controlling the quadruped robot to execute a second target control signal in a simulation environment, and determining second sensor data, second elevation map data, and RGB image data; Inputting the second sensor data, the second elevation map data and the second target control signal into the first target controller to obtain target position information of each joint of the quadruped robot; Constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal; The second controller to be trained is trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target internal perception network, a second target external perception network, and a second target controller network.

2. The method according to claim 1, characterized in that The step of training the first controller according to the first sensor data, the first elevation map data and the first target control signal to obtain a trained first target controller includes: Inputting the first sensor data into a first internal perception network in the first controller to obtain a first encoding vector, and inputting the first elevation map data into a first external perception network in the first controller to obtain a second encoding vector; splicing a first target vector corresponding to a first target control signal, the first encoding vector, and the second encoding vector to obtain a first splicing vector, inputting the first splicing vector into a first controller network in the first controller, and obtaining first predicted position information of each joint of the quadruped robot and an actual control signal of the quadruped robot; The first controller is trained according to the actual control signal and the first target control signal to obtain the first target controller.

3. The method according to claim 2, characterized in that The step of training the first controller according to the actual control signal and the first target control signal to obtain the first target controller includes: Acquire a reward function of the first controller according to the actual control signal and the first target control signal; According to the reward function, the parameters of the first controller are adjusted until the reward function meets the training end condition, and the first controller after the last parameter adjustment is determined as the first target controller.

4. The method according to claim 3, characterized in that The step of acquiring a reward function of the first controller according to the actual control signal and the first target control signal includes: Obtaining a target speed and a target angular speed from the first target control signal, and obtaining an actual speed, an actual angular speed and an actual torque from the actual control signal; Determine a speed reward function according to the target speed and the actual speed, and obtain a speed reward weight of the speed reward function; Determining an angular velocity reward function according to the target angular velocity and the actual angular velocity, and obtaining an angular velocity reward weight of the angular velocity reward function; Determine a torque reward function according to the actual torque, and obtain a torque reward weight of the torque reward function; The reward function of the first controller is obtained according to the speed reward function and the speed reward weight, the angular velocity reward function and the angular velocity reward weight, the torque reward function and the torque reward weight.

5. The method according to claim 1, characterized in that The step of constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal comprises: The target position information of each joint, the RGB image data, the second sensor data and the second target control signal are associated to construct training data.

6. The method according to claim 5, characterized in that The step of training the second controller to be trained according to the training data to obtain a trained second target controller includes: Inputting the RGB image data, the second sensor data, and the second target control signal into the second controller to obtain predicted position information of each joint of the quadruped robot; The second controller is trained according to the predicted position information of each joint and the target position information of each joint to obtain a trained second target controller.

7. The method according to claim 6, characterized in that The step of training the second controller according to the predicted position information of each joint and the target position information of each joint to obtain a trained second target controller comprises: Acquire a loss function according to the predicted position information of each joint and the target position information of each joint; According to the loss function, the parameters of the second controller are adjusted until the loss function meets the training end condition, and the second controller after the last parameter adjustment is determined as the second target controller.

8. The method according to claim 6, characterized in that The step of inputting the RGB image data, the second sensor data, and the second target control signal into the second controller to obtain the predicted position information of each joint of the quadruped robot includes: Input the second sensor data into a second internal perception network in the second controller to obtain a third encoding vector, and input the RGB image data into a second external perception network in the second controller to obtain a fourth encoding vector; The second target vector corresponding to the second target control signal, the third encoding vector and the fourth encoding vector are spliced ​​to obtain a second spliced ​​vector, and the second spliced ​​vector is input into the second controller network in the second controller to obtain the predicted position information of each joint of the quadruped robot.

9. A training device for a quadruped robot controller based on reinforcement learning, characterized in that: The device comprises: A first determination module is used to control the quadruped robot to execute a first target control signal in a simulation environment, and determine first sensor data and first elevation map data; A first training module is used to train a first controller according to the first sensor data, the first elevation map data and the first target control signal to obtain a trained first target controller, wherein the first target controller includes a first target internal perception network, a first target external perception network and a first target controller network; A second determination module is used to control the quadruped robot to execute a second target control signal in a simulation environment, and determine second sensor data, second elevation map data, and RGB image data; A first acquisition module, used for inputting the second sensor data, the second elevation map data and the second target control signal into a first target controller to acquire target position information of each joint of the quadruped robot; A second acquisition module, used for constructing training data according to the target position information of each joint, the RGB image data, the second sensor data and the second target control signal; The second training module is used to train the second controller to be trained according to the training data to obtain a trained second target controller, wherein the second target controller includes a second target internal perception network, a second target external perception network and a second target controller network.

10. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

Images