Humanoid robot whole-body control reinforcement learning method and system based on parallel computing
By using parallel computation of CasADi expressions and CUDA kernels, the problems of whole-body coordinated control and CPU-GPU data interaction in traditional reinforcement learning frameworks are solved, realizing an efficient whole-body control strategy and improving GPU utilization and training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-08
AI Technical Summary
Traditional reinforcement learning control frameworks struggle to achieve coordinated control of the robot's entire body, and the data interaction between the CPU and GPU consumes a significant amount of time, leading to increased training time.
By constructing parallel computing CasADi expressions and CUDA kernels, parallel computation of joint space inertia tensor, center of mass position, and center of mass momentum is achieved. Multi-threaded computation is performed using GPU to avoid data interaction between CPU and GPU and improve GPU utilization. Reinforcement learning is then performed by combining the reward function for center of mass position and momentum calculation.
It improves GPU utilization, reduces data interaction latency between CPU and GPU, enables coordinated control of the robot's entire body, and improves training efficiency and control effectiveness.
Smart Images
Figure CN121589834B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, specifically to a reinforcement learning method and system for the whole-body control of humanoid robots based on parallel computing, which is applied to the reinforcement learning whole-body control of humanoid robots. Background Technology
[0002] The application of reinforcement learning (RL) in robot control can effectively avoid the limitations of traditional model-driven control methods in high-dimensional systems, such as inaccurate modeling and difficulty in coping with dynamic changes in complex environments. By introducing reinforcement learning into robot control systems, researchers can construct more flexible, efficient, and robust control frameworks, enabling robots to have stronger environmental understanding and adaptability. This has made reinforcement learning an important direction in modern robot control and laid the foundation for deploying autonomous robots in complex real-world scenarios.
[0003] However, in practical applications of whole-body robot control, it is often necessary to precisely adjust the motion of the robot's center of mass and momentum distribution in a high-dimensional space. Traditional reinforcement learning control frameworks, however, primarily reward or penalize information such as the position and velocity of robot links, lacking a compact dynamic description like center of mass momentum, making it difficult to achieve coordinated whole-body control. Secondly, current mainstream robot reinforcement learning platforms such as IsaacGym and IsaacLab do not provide an interface for obtaining the rigid body's center of mass momentum. Although the center of mass momentum can be calculated by calling mainstream dynamic libraries such as Pinocchio and RBDL, these libraries run on the CPU, while reinforcement learning training runs on the GPU. This results in significant time spent on data interaction between the CPU and GPU, greatly increasing training time. Summary of the Invention
[0004] The purpose of this invention is to provide a reinforcement learning method and system for the whole-body control of humanoid robots based on parallel computing, so as to solve the problems mentioned in the background art.
[0005] This invention provides the following technical solution: a reinforcement learning method for the whole-body control of a humanoid robot based on parallel computing, comprising the following operational steps:
[0006] Step 1: Construct the CasaADi expression for calculating the spatial inertia tensor of the joint link.
[0007] Preferably, in the robot's general robot description format, the inertial tensor at the center of mass of each link in the initial zero position is read; a symbolic variable representing the new center of mass position and the new total mass of the link are created, and the inertial tensor at the new center of mass position is calculated; the joint coordinates are defined to coincide with the link coordinates, and the inertial tensor of the link in the joint coordinates and the spatial inertial tensor of the joint in the local coordinate system are obtained; finally, the new center of mass positions of each link, the new total mass of each link, the spatial inertial tensor of each joint, and the total mass of the robot are used to construct the CasADi expression. .
[0008] Step 2: Construct the CasaADi symbolic expressions for the position and momentum of the center of mass.
[0009] Preferably, in the robot's general robot description format, the spatial transformation of each joint of the robot in its initial zero position is read; the spatial transformation of each joint is updated by initializing the rigid body composite inertia tensor of each joint; then, the rigid body composite inertia tensor is recursively updated through spatial transformation to obtain the rigid body composite inertia tensor at the center of mass; based on the position of the center of mass, the center of mass is spatially transformed in the world coordinate system, and then the transformation of the center of mass relative to each joint is obtained recursively; at the same time, the center of mass momentum matrix is obtained; finally, the position of the center of mass in the world coordinate system, the center of mass momentum, the generalized position containing the linear displacement and quaternion of the robot body and the motor angle of each joint, the generalized velocity containing the linear velocity and angular velocity of the robot body and the angular velocity of each joint motor, and the spatial inertia tensor of each joint are used to construct the CasADi symbolic expression. .
[0010] Step 3: Convert the CasADi expressions and CasADi symbolic expressions into a CUDA kernel and compile them into an executable file.
[0011] Preferably, the CasaADi calculation process utilizes a structure consisting of three parts: input memory, intermediate memory, and output memory. During the CasaADi calculation process, the input memory accesses the user-provided input, stores the input value in the intermediate memory, performs operations on the corresponding memory addresses according to symbolic operations in the intermediate memory, and stores the operation results in the intermediate memory again. Finally, after all symbolic operations are completed, the results saved in the intermediate memory are stored in the output memory, and the user obtains the calculation results through the output memory.
[0012] CasADi expression and CasADi symbolic expressions The calculation process corresponds to the addresses of each thread in CUDA, which are converted into CUDA kernels and compiled with CMake to obtain an executable file. In the IsaacGym / IsaacLab framework, GPU memory is pre-allocated according to the parallel environment and the executable file of the CUDA kernel is called to perform calculations. Parallelization of the calculation of joint space inertia tensor, center of mass position and center of mass momentum is achieved. Multiple threads of the GPU are used to calculate multiple CUDA kernels to obtain calculation results in multiple environments.
[0013] Step 4: During training environment initialization, import the CUDA kernel containing CasADi expressions and CasADi symbolic expressions and allocate GPU memory space.
[0014] Preferably, during initialization, the CasADi expression is loaded. and CasADi symbolic expressions Based on the number of parallel environments, the ID of each thread is obtained, and the GPU memory space is pre-allocated according to the workspace size required when calculating a single CasADi symbolic expression.
[0015] Step 5: After randomizing the mass and centroid region of the link, call the CasADi expression to calculate the joint space inertia tensor of each environment in parallel.
[0016] Preferably, within the IsaacGym / IsaacLab framework, the properties of rigid bodies, including their mass and center of mass position, are read for each environment, and the mass and center of mass domains are randomized. For each environment, the centers of mass of each rigid body are spliced together to construct new center of mass positions for each link. Then, the masses of each rigid body are spliced together to construct the new total mass of the connecting rods. Finally, the new center of mass positions of each link in each environment are determined. and the new total mass of each link Input into CasADi expression In the CUDA kernel, the output is the rigid body space inertial tensor after domain randomization. and the total mass of the robot .
[0017] Step 6: After each simulation step, call the CasADi symbolic expression to calculate the centroid position and centroid momentum of each environment in parallel.
[0018] Preferably, during each simulation step, the generalized position of each environment includes the linear displacement and quaternions of the robot body and the motor angles of each joint. Generalized velocity, including the linear and angular velocities of the robot body and the angular velocities of the motors at each joint. Spatial inertia tensor of each joint Input to CasADi symbolic expression The CUDA kernel is used to obtain the centroid location of each environment. and center of mass momentum Simultaneously through linear momentum and the total mass of the robot Obtain the velocity of the center of mass .
[0019] Step 7: Calculate the reward function by combining the position and momentum of the center of mass.
[0020] Preferably, the reward function includes a linear momentum tracking reward constructed in the walking strategy. Angular momentum reward The linear momentum change rate tracking penalty constructed in the standing strategy .
[0021] Step 8: Introduce the reward function into reinforcement learning training to obtain the robot's whole-body control strategy.
[0022] Preferably, the obtained robot whole-body control strategy includes guiding the robot to learn whole-body coordination when walking to ensure dynamic stability during walking and whole-body coordination when standing to improve the ability to resist disturbances during balance.
[0023] On the other hand, the present invention also provides a humanoid robot whole-body control reinforcement learning system based on parallel computing, which executes a humanoid robot whole-body control reinforcement learning method based on parallel computing, including: a joint space inertial tensor calculation module, a center of mass position and center of mass momentum calculation module, a CUDA kernel module, a reward module, and a reinforcement learning module.
[0024] On the other hand, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and movable on the processor, wherein the processor executes the program to implement a reinforcement learning method for the whole-body control of a humanoid robot based on parallel computing.
[0025] On the other hand, the present invention also provides a storage medium storing a computer program, which, when run by a processor, implements a reinforcement learning method for the whole-body control of a humanoid robot based on parallel computing.
[0026] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention realizes the parallel computation of the robot's joint space inertia tensor, center of mass position, and center of mass momentum in the IsaacGym / IsaacLab environment by converting the symbolic expressions of the joint space inertia tensor, center of mass position, and center of mass momentum into CUDA kernels. This not only avoids the latency and overhead of data interaction between the CPU and GPU caused by calling dynamics libraries, but also avoids the problem of low GPU utilization caused by fragmented operations in PyTorch, greatly improving GPU utilization. At the same time, the robot's whole-body control strategy is obtained by using the robot's center of mass position and center of mass momentum after reinforcement learning training, enabling reinforcement learning to achieve coordinated whole-body control. Attached Figure Description
[0027] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0028] Figure 1 This is a flowchart of the reinforcement learning method for whole-body control of a humanoid robot provided in an embodiment of the present invention;
[0029] Figure 2 This is a diagram showing the relationship between the coordinates of a rigid body joint (linkage rod) and the coordinates of its center of mass, provided in an embodiment of the present invention.
[0030] Figure 3 This is a schematic diagram of the spatial relationship between robot links provided in an embodiment of the present invention;
[0031] Figure 4 This is a schematic diagram of the robot's center of mass position and center of mass momentum provided in an embodiment of the present invention;
[0032] Figure 5 This is a schematic diagram of the method for converting CasADi symbolic expressions to CUDA kernels provided in an embodiment of the present invention;
[0033] Figure 6 This is a schematic diagram of the joint space inertia tensor calculation process for each environment provided in the embodiments of the present invention;
[0034] Figure 7 This is a schematic diagram illustrating the calculation process of the center of mass position and center of mass momentum for each environment provided in the embodiments of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] Example 1
[0037] Combination Figures 1-7 As shown, this embodiment provides the following technical solution: a reinforcement learning method for the whole-body control of a humanoid robot based on parallel computing, including the following operational steps:
[0038] Step 1: Construct the CasaADi expression for calculating the spatial inertia tensor of the joint link. ;
[0039] In this embodiment, the inertial tensor at the center of mass of each link of the robot at its initial zero position is first read from the robot's URDF (Universal Robot Description Format). Create a symbolic variable representing the new centroid position. The total mass of the new connecting rod .
[0040] For example, consider a connecting rod. Original total mass was The original centroid position is Assuming the change in mass is uniform, then the connecting rod New quality Compared with the original total mass The following relationship exists between them:
[0041]
[0042] in, The scaling factor is uniformly varying. Since the change in mass is uniform, it does not essentially cause a change in the centroid. The purpose of forcibly shifting the centroid is to increase the generalization ability of the policy during domain randomization (the maximum range of the shift is about 5cm in radius, so it will not significantly change the mass distribution of the rigid body). Therefore, at the new centroid position... inertial tensor under for:
[0043]
[0044] in, , representing the shift of the randomized centroid position relative to the original centroid position. Link In the joint The position of the centroid of the coordinate system (joint in URDF) and connecting rod (The coordinate systems coincide), in URDF, define the joint. With connecting rod Coordinates coincide, and the relationship between joint coordinates and the position of the center of mass is as follows: Figure 2 As shown, the joint is obtained. Linkage in coordinate system inertial tensor for:
[0045]
[0046] For joints The spatial inertia tensor in the local coordinate system is expressed as: :
[0047]
[0048] Therefore, the symbolic expression for constructing CasaADi is as follows:
[0049]
[0050] in The new center of mass positions for each link. The new total mass for each link.
[0051] For the spatial inertia tensor of each joint, This represents the total mass of the robot.
[0052] Step 2: Construct the CasaADi symbolic expressions for the position and momentum of the center of mass. ;
[0053] In this embodiment, the spatial transformations of each joint of the robot in its initial zero position (the transformations of the parent joint in the child joint system) are first read from the robot's URDF (Universal Robot Description Format):
[0054]
[0055] in This refers to the spatial transformation from the parent joint to the child joint. For joints Parent joint (joint) (the previous node). For joints The father joint in the joint Rotation below, For joints The father joint in the joint The spatial relationship between the links in the lower position is as follows: Figure 3 As shown. Following the algorithm proposed by David E. Orin and Ambarish Goswami, the position and momentum of the center of mass are calculated recursively. First, the rigid body composite inertia tensor of each joint is initialized. Update the spatial transformation of each joint:
[0056]
[0057] in, This refers to the spatial transformation from the parent joint to the child joint at a given moment. for The generalized position of the time-based robot (including the world position of the body, quaternions, and rotation of each joint relative to the zero position). The update transformation is determined based on the meaning of the generalized position.
[0058] The rigid body composite inertia tensor is recursively updated through spatial transformation. :
[0059]
[0060] in, for At any time, the joint The rigid body composite inertia tensor at point A is obtained recursively as the rigid body composite inertia tensor at the center of mass. :
[0061]
[0062] in, for The rigid body composite rotational inertia tensor at the center of mass at time t is given by the following formula: Current position This causes the center of mass to change in space within the world system:
[0063]
[0064] in In order to be in The spatial transformation from the center of mass to the floating base joints at any given time can be obtained recursively, and the transformation of the center of mass relative to each joint can then be derived:
[0065]
[0066] in, In order to be in The spatial transformation from the center of mass to joint i at time i. In order to be in The spatial transformation from the center of mass to the parent joint of joint i at time i.
[0067] The center-of-mass momentum matrix (CMM) is then obtained. :
[0068]
[0069]
[0070] in, For corresponding joints The motion subspace of , as defined, is in Center of mass momentum at time t Represented as:
[0071]
[0072] in, In order to be in Angular momentum of the center of mass at time t. In order to be in Momentum of the center of mass at time t. For including the robot body in The linear and angular velocities at any given moment, the generalized velocities of the angular velocities of each joint motor, and the relationship between the robot's center of mass and its momentum in the world frame are as follows: Figure 4 As shown, the CasaADi symbolic expression is constructed as follows:
[0073]
[0074] in, Let the position of the center of mass in the world coordinate system be denoted as . It is a generalized position that includes the linear displacement and quaternions of the robot body and the motor angles of each joint.
[0075] Step 3: Convert the CasaADi expression and CasADi symbolic expressions Convert to a CUDA kernel and compile into an executable file.
[0076] In this embodiment, the structure of the CasADi computation process is utilized: The CasADi computation process consists of three parts: input memory, intermediate memory (working area), and output memory. Input memory stores the user-provided input, and then stores the input value in intermediate memory. Intermediate memory performs symbolic operations on the corresponding memory addresses and stores the results back in intermediate memory. Finally, after all symbolic operations are completed, the results saved in intermediate memory are stored in output memory, allowing the user to retrieve the computation result. and The computation process corresponds to the addresses of various threads in CUDA, which are then converted into CUDA kernels and compiled using CMake to obtain an executable file. In frameworks such as IsaacGym / IsaacLab, GPU memory is pre-allocated according to the parallel environment, and the executable file of the CUDA kernel is called to perform the computation. In this way, the calculation of joint space inertia tensor, center of mass position, and center of mass momentum can be parallelized, utilizing multiple threads of the GPU to compute multiple CUDA kernels and obtain computation results from multiple environments.
[0077] For example, such as Figure 5 As shown, suppose a CasADi symbolic expression is , The following steps will be performed during the calculation:
[0078] Step 31: First, perform the input operation and read... And save it in workspace 0, then read it. Saved in workspace 1;
[0079] Step 32: Perform a summation operation on the contents of the two memory areas, working area 0 and working area 1, using the "+" operator, and store the result in working area 0;
[0080] Step 33: Based on the operator " Perform a square operation on memory in working area 0 and store the result in working area 0;
[0081] Step 34: Finally, perform the output operation, saving the contents of workspace 0 to the output memory for user access.
[0082] For example, the symbolic expressions generated by CasADi provide information such as the number of computation steps, the number of each operation, the input dimension, and the output dimension. Using this information, a CUDA kernel can be written to perform the same symbolic computation steps as CasADi on the corresponding GPU space based on the thread ID. In this invention, the same method is applied to the calculation of spatial inertia tensor and center-of-mass momentum.
[0083] Step 4: Import during training environment initialization. and The CUDA kernel is used to allocate GPU memory space;
[0084] In this embodiment, during initialization, loading... and Based on the number of parallel environments, the ID of each thread is obtained, and the GPU memory space is pre-allocated according to the workspace size required when calculating a single CasADi symbolic expression.
[0085] Step 5: After randomizing the mass and centroid region of the link, call... The joint space inertia tensor of each environment is computed in parallel;
[0086] In this embodiment, as Figure 6 As shown, the properties of the rigid body in each environment are read in IsaacGym / IsaacLab (the properties are the same as those in URDF), such as rigid body mass and centroid position. To improve the robustness of the trained model, the mass and centroid regions are randomized. For each environment, the centroids of each rigid body are concatenated to construct... The mass of each rigid body is pieced together to construct Each environment and Input to CUDA kernel (input-to-output method reference) Figure 5 As shown), the output is the rigid body space inertia tensor after domain randomization. and the total mass of the robot .
[0087] Step 6: Call after each simulation step The position and momentum of the center of mass for each environment are calculated in parallel.
[0088] In this embodiment, as Figure 7 As shown, during each simulation step, the generalized position of each environment is... ,speed and spatial inertial tensor Input to The CUDA kernel is used to obtain the centroid location of each environment. and center of mass momentum .
[0089] For the velocity of the center of mass Through linear momentum and the total mass of the robot We can obtain:
[0090]
[0091] Step 7: Calculate the reward function by combining the position and momentum of the center of mass.
[0092] In this embodiment, a walking strategy is used to construct a linear momentum tracking reward. , means as follows:
[0093]
[0094] in For speed command, This is a hyperparameter.
[0095] Angular momentum reward :
[0096]
[0097] in This is a hyperparameter.
[0098] Penalty for tracking the rate of change of momentum in the standing strategy construction line :
[0099]
[0100] in, For the desired center of mass position (local coordinate system of the organism), is the rotation matrix from the world coordinate system to the local coordinate system of the machine.
[0101] Step 8: Introduce the reward function into reinforcement learning training to obtain the robot's whole-body control strategy;
[0102] In this embodiment, the obtained robot whole-body control strategy includes guiding the robot to learn whole-body coordination when walking to ensure dynamic stability during walking and whole-body coordination when standing to improve the anti-disturbance ability during balance.
[0103] This invention achieves parallel computation of the robot's joint space inertia tensor, center of mass position, and center of mass momentum in the IsaacGym / IsaacLab environment by converting the symbolic expressions of the joint space inertia tensor, center of mass position, and center of mass momentum into a CUDA kernel. This not only avoids the latency and overhead of data interaction between the CPU and GPU caused by calling dynamics libraries, but also avoids the problem of low GPU utilization caused by fragmented operations in PyTorch, greatly improving GPU utilization. Furthermore, by using the robot's center of mass position and center of mass momentum after reinforcement learning training, a full-body control strategy for the robot can be obtained, enabling reinforcement learning to achieve coordinated full-body control.
[0104] Example 2
[0105] This implementation provides a humanoid robot whole-body control reinforcement learning control system based on parallel computing methods, including:
[0106] The joint spatial inertia tensor calculation module is used to calculate the centroid position of the link, the spatial inertia tensor of the joint after randomization of the mass domain, and the total mass of the robot.
[0107] The center of mass position and center of mass momentum calculation module is used to calculate the robot's center of mass position and center of mass momentum.
[0108] The CUDA kernel module is used to convert the CasADi symbolic expressions of joint space inertia tensor, center of mass position, and center of mass momentum into CUDA kernels and allocate GPU memory space to realize parallel computation of joint space inertia tensor, center of mass position, and center of mass momentum.
[0109] The reward module is used to calculate the robot's reward function and penalty based on its center of mass momentum.
[0110] The reinforcement learning module is used to train the robot by applying the rewards and penalties calculated by the reward module to obtain the robot's whole-body control strategy.
[0111] Example 3
[0112] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and movable on the processor. When the processor executes the program, it implements the steps in the aforementioned reinforcement learning method for whole-body control of a humanoid robot based on parallel computing.
[0113] Example 4
[0114] This embodiment provides a storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps in the above-described reinforcement learning method for the whole-body control of a humanoid robot based on parallel computing.
[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0116] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A reinforcement learning method for whole-body control of a humanoid robot based on parallel computing, characterized by: The following steps are included: Construct the CasaADi expression for calculating the spatial inertia tensor of the joint links; Construct the CasaADi symbolic expressions for the position and momentum of the center of mass; Convert CasADi expressions and CasADi symbolic expressions into CUDA kernels and compile them into executable files; During training environment initialization, the CUDA kernel for CasADi expressions and CasADi symbolic expressions is imported and GPU memory space is allocated; After randomizing the mass and centroid region of the link, the CasADi expression is called to compute the joint space inertia tensor of each environment in parallel. After each simulation step, the CasADi symbolic expression is called to calculate the centroid position and centroid momentum of each environment in parallel. The reward function is calculated by combining the position and momentum of the center of mass. The reward function includes linear momentum tracking reward and angular momentum reward constructed in the walking strategy, and linear momentum change rate tracking penalty constructed in the standing strategy. By incorporating a reward function into reinforcement learning training, a full-body control strategy for the robot can be obtained. The CasaADi expression for constructing the spatial inertia tensor of the joint links includes: In the general robot description format, the inertial tensor at the center of mass of each link of the robot is read at the initial zero position; Create a new symbolic variable representing the new center of mass position and the new total mass of the link. Calculate the inertia tensor at the new centroid position; Define the joint and link coordinates to coincide, and obtain the inertia tensor of the link in the joint coordinate system and the spatial inertia tensor of the joint in the local coordinate system. Finally, the new centroid positions of each link, the new total mass of each link, the spatial inertia tensor of each joint, and the total mass of the robot are used to construct the CasADi expression; The CasaADi notational expressions for constructing the position and momentum of the center of mass include: In the general robot description format, the spatial transformation of each joint of the robot in the initial zero position is read. The spatial transformation is the transformation of the parent joint in the child joint system. The spatial transformation of each joint is updated by initializing the rigid body composite inertia tensor of each joint. Then, through spatial transformation, the rigid body composite inertia tensor is recursively updated to obtain the rigid body composite inertia tensor at the center of mass. Based on the position of the center of mass, the center of mass undergoes spatial transformation in the world frame, and then the transformation of the center of mass relative to each joint is obtained recursively; at the same time, the momentum matrix of the center of mass is obtained. Finally, the CasaADi symbolic expression is constructed using the position of the center of mass in the world coordinate system, the momentum of the center of mass, the generalized position including the linear displacement and quaternion of the robot body and the motor angle of each joint, the generalized velocity including the linear velocity and angular velocity of the robot body and the angular velocity of each joint motor, and the spatial inertia tensor of each joint. Converting to a CUDA kernel and compiling it into an executable file involves utilizing the structure of the CasADi computation process, which consists of three parts: input memory, intermediate memory, and output memory. During the calculation process of CasaADi, the input memory accesses the input given by the user, stores the input value in the intermediate memory, performs operations on the corresponding memory address according to the symbolic operation, and stores the operation result in the intermediate memory again. Finally, after all the symbolic operations are completed, the result saved in the intermediate memory is stored in the output memory, and the user obtains the calculation result through the output memory. The computation process of CasADi expressions and CasADi symbolic expressions is mapped to the addresses of various threads in CUDA, converted into CUDA kernels, and compiled using CMake to obtain an executable file. In the IsaacGym / IsaacLab framework, GPU memory is pre-allocated according to the parallel environment, and the executable file of the CUDA kernel is called to perform the computation. Parallelization of the calculation of joint space inertia tensor, center of mass position, and center of mass momentum is achieved, and multiple CUDA kernels are calculated using multiple threads of the GPU to obtain the computation results of multiple environments.
2. The reinforcement learning method for whole-body control of a humanoid robot based on parallel computing as described in claim 1, characterized in that: Importing CUDA kernels containing CasADi expressions and CasADi symbolic expressions and allocating GPU memory space includes: During initialization, CasADi expressions and CasADi symbolic expressions are loaded, and the ID of each thread is obtained based on the number of parallel environments. The GPU memory space is pre-allocated based on the workspace size required for computation of a single CasADi symbolic expression.
3. The reinforcement learning method for whole-body control of a humanoid robot based on parallel computing as described in claim 2, characterized in that: Parallel computation of the joint space inertia tensor for each environment includes: Within the IsaacGym / IsaacLab framework, the properties of rigid bodies in each environment, including rigid body mass and centroid position, are read, and the mass and centroid domain are randomized. For each environment, the center of mass of each rigid body is spliced together to construct the new center of mass position of each link, and then the mass of each rigid body is spliced together to construct the new total mass of each link. Finally, the new centroid positions of each link in each environment and the new total mass of each link are input into the CUDA kernel of the CasADi expression, and the output is the rigid body space inertial tensor and the total mass of the robot after domain randomization.
4. The reinforcement learning method for whole-body control of a humanoid robot based on parallel computing as described in claim 3, characterized in that: Parallel computation of the center-of-mass position and momentum for each environment includes: During each simulation step, the generalized position of each environment, including the linear displacement and quaternion of the robot body and the motor angles of each joint, the generalized velocity including the linear velocity and angular velocity of the robot body and the angular velocity of each joint motor, and the spatial inertia tensor of each joint are input into the CUDA kernel of the CasADi symbolic expression to obtain the center of mass position and center of mass momentum of each environment. At the same time, the center of mass velocity is obtained from the linear momentum and the total mass of the robot.
5. A humanoid robot whole-body control reinforcement learning system based on parallel computing, characterized in that: The method for reinforcement learning of humanoid robot whole-body control based on parallel computing as described in any one of claims 1-4 includes: The joint spatial inertia tensor calculation module is used to calculate the position of the center of mass of the link and the spatial inertia tensor of the joint after the mass domain is randomized, as well as the total mass of the robot. The center of mass position and center of mass momentum calculation module is used to calculate the robot's center of mass position and center of mass momentum; The CUDA kernel module is used to convert the CaSADi symbolic expressions of joint space inertia tensor, center of mass position, and center of mass momentum into CUDA kernels and allocate GPU memory space to realize parallel computation of joint space inertia tensor, center of mass position, and center of mass momentum. The reward module is used to calculate the robot's reward function and penalty based on its center of mass momentum; The reinforcement learning module is used to train the robot by applying the rewards and penalties calculated by the reward module to obtain the robot's whole-body control strategy.
6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the reinforcement learning method for whole-body control of humanoid robots based on parallel computing as described in any one of claims 1-4.
7. A storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the reinforcement learning method for whole-body control of a humanoid robot based on parallel computing as described in any one of claims 1-4.