Double-arm body operation method of humanoid robot based on reinforcement learning

Abstract

The invention relates to a humanoid robot double-arm body operation method based on reinforcement learning, belongs to the field of robot cooperative control, and is characterized in that a strategy of trajectory block prediction and time integration fusion is used for double-arm cooperative control, and continuity and stability of double-arm operation are improved; in reinforcement learning control, a three-dimensional pose track generation and correction module is introduced, condition generation and denoising correction of a three-dimensional space are carried out on a track block layer, and geometric consistency and naturalness of a generated track are guaranteed; the invention further provides a reinforcement learning optimization framework and a simulation-reality migration process, and through system integration of reward, value guidance and migration processes, strategy deployability and safety are guaranteed; compared with the prior art, the method has the advantages that the naturalness, the collaboration and the success rate of double-arm operation can be remarkably improved, the generalization ability is high, and the good simulation-to-reality migration ability is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot collaborative control, and particularly relates to a humanoid robot dual-arm body operation method based on reinforcement learning. BACKGROUND

[0002] As a typical representative of embodied intelligence, humanoid robots have made significant progress in technology breakthroughs and commercial applications in recent years, and are reshaping the production and lifestyle in multiple fields. The common methods for existing humanoid robots to perform dual-arm embodied operation tasks include trajectory planning and imitation learning.

[0003] Trajectory planning achieves precise control through predefined paths and motion sequences, and is suitable for repetitive tasks in structured environments. Its core lies in generating optimal motion paths through mathematical modeling. In structured scenarios such as industrial assembly, this method shows significant advantages. However, this method has limited adaptability to dynamic environments. When encountering object position shifts or obstacle interference, manual re-planning of parameters is required, resulting in decreased efficiency. In addition, complex tasks such as flexible object grasping often require segmented planning and real-time adjustment, further increasing the computational burden. Therefore, the trajectory planning method is difficult to guarantee real-time and natural in high-dimensional complex environments.

[0004] Imitation learning enables robots to adapt to complex scenarios by observing human demonstrations or expert strategies. Its core advantage lies in its adaptability to unstructured tasks. This method records human operations through vision or motion capture systems, extracts key motion features through behavior cloning or inverse reinforcement learning, and finally generates executable robot strategies. However, imitation learning faces two major challenges: one is data dependency, which requires a large number of high-quality demonstration samples to cover operation variants; the other is strategy generalization, which will significantly decrease performance when environmental parameters exceed the training range. That is, imitation learning relies on demonstration data, has poor generalization, and is prone to error accumulation, making it difficult to guarantee the absolute accuracy of actions. In tasks requiring millimeter-level positioning, it may need to be combined with trajectory planning for post-processing.

[0005] Reinforcement Learning (RL) has the ability to optimize the policy through interaction, and can learn the optimal policy through the interaction between the agent and the environment without explicit programming. In high-dimensional action space, traditional methods need to manually design the dependency relationship between action dimensions, while RL can autonomously explore high-dimensional combined strategies, such as coordinating multiple joint movements in robot control. This flexibility makes RL irreplaceable in embodied intelligence; however, high-dimensional action space also brings significant challenges - dimension explosion leads to a sharp decline in exploration efficiency, policy convergence is difficult, and it is easy to fall into local optimum; in addition, the natural neglect of RL to action smoothness may cause mechanical vibration or operation interruption, further limiting its practicality in physical systems; in high-dimensional action space, RL faces three core challenges: insufficient convergence stability, lack of action smoothness, and difficulty in simulation to reality transfer.

[0006] Therefore, there is an urgent need for a dual-arm embodied operation method that combines action generation optimization and reinforcement learning training, which can not only ensure task completion but also enhance the naturalness of actions and the ability of real-world deployment. SUMMARY

[0007] In view of the defects of the prior art, the present application provides a dual-arm embodied operation method for humanoid robots based on reinforcement learning, aiming to solve the problems of unnatural dual-arm operation, insufficient collaboration, poor generalization ability, and unstable real-world deployment in the prior art.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a dual-arm embodied operation method for humanoid robots based on reinforcement learning, comprising the following steps: (1) obtaining robot multi-modal observation information, including visual data, joint state, force signal and target object pose; (2) trajectory generation and optimization: (2.1) constructing an action generation structure: using a neural network based on sequence modeling, inputting the robot multi-modal sequence of a certain number of steps in history, predicting the end-of-arm trajectory block of a certain number of steps in the future, and generating a smooth trajectory by weighted fusion of overlapping predicted trajectory blocks in the reasoning stage; (2.2) conditionally generating and denoising the end-of-arm trajectory block in three-dimensional space to ensure that the trajectory meets the geometric constraints and has naturalness; (2.3) projecting the generated end-of-arm trajectory into joint commands, combining inverse kinematics and constraint optimization methods to meet the joint limit, speed, acceleration and collision conditions; (3) reinforcement learning training: (3.1) constructing a comprehensive reward function, the comprehensive reward function including task completion reward, dual-arm collaboration consistency reward and action smoothness reward; (3.2) The action generation structure is trained using a reinforcement learning strategy optimization method to maximize the expected return; and in the interactive sampling stage of reinforcement learning, the value function is used to value weight the policy output action, guiding the sampling distribution to shift to the high value area; (4) Safe execution and deployment: (4.1) Training in a simulation environment using curriculum learning and domain randomization; (4.2) System identification and parameter correction before deployment, and policy distillation and delay compensation; (4.3) Deploy the optimized strategy to the real robot, and ensure execution safety through safety constraint layer and recovery mechanism.

[0009] Further, in step (2.1), in the inference stage, exponential weighted fusion is used for overlap prediction, and the rotation part is interpolated in three-dimensional rotation space to ensure continuity.

[0010] Further, step (2.2) specifically includes the following steps: First, add noise to the trajectory block in the SE(3) tangent space, denoted as ; Wherein, represents the noisy trajectory of the diffusion process at time step ; represents the original real trajectory; represents the noise scheduling coefficient; represents random noise subject to standard Gaussian distribution ; Then, train the denoising network to predict noise, and the loss function is , wherein, denotes the training loss of the denoising network; denotes the noise predicted by the network; denotes the real added noise, and in the inference stage, the natural trajectory is obtained by gradually denoising from Gaussian noise.

[0011] Further, the comprehensive reward function in step (3.1) is defined as: , wherein, denotes the task completion degree reward, denotes the dual-arm cooperation consistency reward, denotes the action smoothness reward, , , , , , ,

[0012] Further, in step (3.2), the policy network is updated using a policy gradient-based reinforcement learning algorithm, and the objective function of policy learning is ; wherein, represents a policy optimization objective function; represents a policy network parameter; represents a policy distribution; is a discount factor; represents the reward at the moment.

[0013] Further, in step (3.2), the value guiding mechanism weights the action sampling based on the policy output distribution and the state-action value evaluation, which is represented as: , wherein, represents the action distribution after reweighting after guiding; represents the action distribution of the original output of the policy network; represents a state-action value function; represents a guiding intensity coefficient.

[0014] Further, in step (4.1), domain randomization includes perturbation of mass, friction, damping, lighting, texture and sensor noise.

[0015] Further, the safety constraint layer in step (4.3) includes joint amplitude limiting, torque saturation, collision detection and failure recovery mechanism.

[0016] Further, the dual-arm body operation method further comprises: recording running data during real execution, and realizing continuous optimization of the policy through offline retraining or residual learning.

[0017] Further, in step (2.1), the action generation structure is a Transformer structure based on self-attention mechanism, which encodes the robot state sequence of a plurality of historical steps and predicts the dual-arm end trajectory block of a plurality of future steps.

[0018] Beneficial effects: Compared with the prior art, the present application can significantly improve the naturalness, collaboration and success rate of dual-arm operation, has strong generalization ability, and has good simulation-to-reality migration ability.

[0019] The application adopts a trajectory block-based dual-arm action sequence generation mechanism, represents dual-arm operation actions as future multi-step end trajectory blocks instead of single-step actions, and outputs long-time span trajectory sequences through time sequence modeling; in the reasoning phase, overlapping trajectories are fused by exponential weighting to generate smooth and continuous control instructions, which significantly reduces long-time sequence error accumulation problems; the trajectory block prediction + time integration fusion strategy is used for dual-arm coordinated control, which improves the continuity and stability of dual-arm operation compared with traditional single-step prediction or trajectory planning methods.

[0020] The application embeds three-dimensional trajectory denoising correction in reinforcement learning control, generates and corrects three-dimensional pose trajectories through a three-dimensional pose trajectory generation and correction module, performs conditional generation and denoising correction in three-dimensional space at the trajectory block level, ensures the geometric consistency and naturalness of generated trajectories, effectively avoids problems such as joint trajectory jitter and end conflict, makes dual-arm actions conform to human operation habits, and improves the naturalness of dual-arm actions; the mechanism is decoupled from the reinforcement learning framework and can embed any strategy network as a trajectory prior generator.

[0021] The application provides a reinforcement learning optimization framework and simulation-real world migration process, optimizes the reinforcement learning mechanism by designing a comprehensive reward function (task completion degree, dual-arm collaborative consistency, and action smoothness) and introducing a value guidance mechanism in sampling, so that the strategy converges faster and is more stable; a complete simulation-to-real migration process is constructed: in simulation, curriculum learning and domain randomization training are used, and after system identification, strategy distillation and delay compensation, it is deployed to real robots, and a safety constraint layer (including joint amplitude limiting, collision detection, and failure recovery) is introduced in the deployment stage to ensure execution safety; effectively solves the landing deployment problem of high-dimensional continuous action reinforcement learning in real dual-arm operation; through the system integration of reward + value guidance + migration process, the deployability and safety of the strategy are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0022] Fig. 1 It is the overall framework diagram of the humanoid robot dual-arm body operation method of the application. Fig. 2 It is the flowchart of the training-deployment of the application. DETAILED DESCRIPTION

[0023] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0024] As Figs. 1-2As shown, in this embodiment, a humanoid robot dual-arm body operation method based on reinforcement learning includes: acquiring multi-modal state information; generating smooth dual-arm trajectories by trajectory block prediction and weighted fusion; ensuring natural motion and geometric reasonableness through three-dimensional pose trajectory generation and correction; combining reward functions including task completion, collaborative consistency and motion smoothness for reinforcement learning optimization, and introducing a value guidance mechanism to improve returns in sampling; using curriculum learning and domain randomization training in simulation, deploying to real robots after system identification, policy distillation and delay compensation, and combining safety constraint layers to ensure execution safety; the specific steps are as follows.

[0025] Step (1) Environment perception and state acquisition: multi-modal state observation information of the robot is collected through sensors, including dual-arm joint angle and angular velocity, torso posture, force and tactile signals, three-dimensional pose of the target object, and visual environment information.

[0026] According to the acquired information, a state space is constructed, the state space contains image features , body state , , target pose , ; specifically, : image features at the current time (visual input); : joint angle vector at the current time; : joint angular velocity vector at the current time; : three-dimensional position of the target object; : three-dimensional pose (rotation matrix or quaternion) of the target object.

[0027] Based on the state space, define the action space, the action space is the end pose sequence of the future step; ; wherein, : future step action trajectory; : end position of the first step; : end pose of the first step.

[0028] Step (2) Trajectory generation and optimization.

[0029] (2.1) Construct an action generation structure, which is a neural network module based on time series modeling, the input of which is a historical state sequence, i.e. a robot multi-modal state sequence of the last several steps (including double-arm joint angle, angular velocity, force signal and target object pose, etc.), and the output is a double-arm end trajectory block of the future several steps; in the inference stage, there is an overlapping part between the trajectory blocks predicted at adjacent time points, and the system uses exponential weighted fusion on these overlapping trajectory segments to generate a final executed smooth trajectory, so as to reduce the accumulation of long-time series prediction error.

[0030] In this embodiment, the Transformer structure based on self-attention mechanism is adopted to encode the robot state sequence of the last several steps and predict the double-arm end trajectory block of the future several steps, so as to realize the continuous generation of long-time series action.

[0031] In the inference stage, the overlapping predicted trajectory blocks are exponentially weighted and averaged to generate a smooth and coherent double-arm action instruction, which reduces the accumulation of prediction error. Specifically, the strategy outputs future step action blocks, and uses the time-integrated exponential weighted average formula to weight and fuse the overlapping predictions, and the rotation part is logarithm-exponential averaged or uses SLERP interpolation in SO(3) tangent space to ensure continuity; wherein, : the fused step action; : the predicted action in the th trajectory block at the th step; : exponential decay coefficient for controlling the weight decrease in fusion.

[0032] (2.2) Three-dimensional pose trajectory generation and correction: conditionally generating and denoising the end trajectory block in three-dimensional space to ensure the geometric reasonableness and naturalness of the generated trajectory.

[0033] First, the trajectory block is forwardly added with noise in SE(3) tangent space, and the trajectory formula with noise is expressed as ; wherein, : the noisy trajectory at time step : the original real trajectory; : noise scheduling coefficient (decay coefficient); : random noise obeying standard Gaussian distribution.

[0034] Then, the denoising network is trained to predict the noise, and the loss function is: wherein, : training loss of the denoising network; : denotes the noise predicted by the network;​​ : Real added noise, gradually denoising from Gaussian noise in the inference stage to get natural trajectory.

[0035] In the trajectory generation process, noise is gradually added and denoised in the training stage to improve the naturalness of the generated trajectory.

[0036] (2.3) Project the generated end trajectory into joint commands, combine inverse kinematics and constraint optimization methods to meet joint limit, speed, acceleration and collision conditions.

[0037] Step (3) reinforcement learning training.

[0038] (3.1) Reward function design: construct a comprehensive reward function, including task completion reward, dual-arm collaboration consistency reward and action smoothness reward, to guide policy optimization.

[0039] The comprehensive reward function is defined as: , wherein, represents the task completion reward, represents the dual-arm collaboration consistency reward, represents the action smoothness reward, , , respectively represent , , corresponding weight coefficients, which can be determined by experience setting and parameter search, with priority to task completion, and then considering collaboration and action smoothness.

[0040] The selection of weight coefficients needs to be considered comprehensively according to the task. Initially, they can be set to =1.0, =0.5, =0.2, to ensure that task completion is prioritized. In training, they can be appropriately fine-tuned according to the learning curve. The adjustment principle is: if the task success rate is low, increase ; if the dual-arm collaborative action is out of sync, increase ; if the trajectory jitter is large, increase .

[0041] (3.2) Reinforcement learning optimization: use policy gradient-based reinforcement learning algorithm (such as PPO) to update the policy network, and introduce value guidance mechanism in the sampling process to make the action distribution biased towards high return area.

[0042] (3.2.1) Optimize the strategy using PPO algorithm: maximize the expected value of the reward function in reinforcement learning. The objective function formula of policy learning is: ; wherein, : policy optimization objective function; : policy network parameters; : policy distribution; : discount factor; : : reward at time step t.

[0043] (3.2.2) Q-value guided sampling: modify the policy distribution in the sampling stage, and guide the action sampling based on the value function. The formula of value guided sampling is: where, : re-weighted action distribution after guidance; : action distribution of the original output of the policy network; : state-action value function; : guidance intensity coefficient.

[0044] In the sampling stage of reinforcement learning, the state-action value of the action is used as the weight to re-weight the policy output distribution, so that the sampling result is biased towards high-value actions, thereby improving the efficiency of policy learning.

[0045] (4) Safe execution and deployment.

[0046] (4.1) Simulation training and domain randomization: train in a simulation environment, correct parameters through system identification, and use curriculum learning (from basic grasping, carrying to complex assembly) and domain randomization (dynamic parameters, friction coefficient, noise and time delay disturbance) to improve the generalization ability of the policy.

[0047] Curriculum learning: provide simple tasks (such as grasping stationary objects, light mass objects) at the beginning of training, and then gradually increase the difficulty of tasks (heavier targets, objects moving, more complex operation space), and gradually unlock tasks by controlling the training environment parameters.

[0048] Domain randomization: randomly perturb robot and environment parameters during training, including dynamic parameters, mass, friction coefficient, joint damping, joint control delay, sensor noise, lighting conditions, texture, etc., so that the policy can still work stably in a changing environment.

[0049] (4.2) System identification and policy transfer: calibrate the real robot dynamics parameters through system identification before real deployment, and perform policy distillation and delay compensation to improve the real-time performance and stability of the policy.

[0050] (4.3) Real deployment and safety protection: deploy the optimized policy on the real robot, and guarantee the operation safety through safety constraint layer and recovery mechanism.

[0051] The safety constraint layer includes joint amplitude, torque saturation, collision detection and failure recovery mechanism. The joint amplitude limits the joint angle, angular velocity and torque command of the strategy output, and the part exceeding the threshold is truncated to the safe range.

[0052] Collision detection: using force sensors or current / torque monitoring, stopping movement immediately and retreating to a safe position when detecting abnormal contact force.

[0053] Failure recovery mechanism: if the task fails (such as grasping failure, object falling, position error too large, etc.), trigger the recovery strategy to move the dual arms back to the initial safe posture and wait for re-execution.

[0054] Through the above steps (1)-(4), the invention can realize the collaborative control of humanoid robot dual arms with body operation, which can significantly improve the naturalness, collaboration and success rate of dual arm operation, and has good simulation to reality migration ability.

[0055] Further, on the basis of the above steps (1)-(4), the method of the invention can also have an online fine-tuning link: record the running data during real-time execution, and realize continuous optimization of the strategy through offline retraining or residual learning; it should be noted that this online fine-tuning link is a further optimization of the method of the invention, and can be flexibly selected in actual use, not a necessary step.

[0056] Compared with the prior art, the invention has the following advantages: (1) Naturalness improvement: through three-dimensional pose trajectory correction and reward function constraint, guiding the robot to generate more natural actions close to human operation habits; (2) Collaboration enhancement: using sequence trajectory block prediction and fusion mechanism to ensure smooth and consistent dual arm motion and improve collaborative operation performance; (3) Strong generalization ability: combining curriculum learning and domain randomization to make the robot adapt to different object shapes, materials and external disturbances, improving generalization and stability.

[0057] The above is only the preferred embodiment of the invention, not any form of limitation on the invention, although the invention has been disclosed as above, however, it is not intended to limit the invention, any skilled person in the art, without departing from the scope of the technical solution of the invention, can make some changes or modifications to the above disclosed technical content to make equivalent embodiments with equivalent changes, but as long as it does not deviate from the technical solution of the invention, any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the invention, all still belong to the scope of the technical solution of the invention.

Claims

1. A method for body operation of a humanoid robot dual arm based on reinforcement learning, characterized by, The method comprises the following steps: (1) acquiring robot multi-modal observation information, including visual data, joint state, force signal and target object pose; (2) trajectory generation and optimization: (2.1) constructing an action generation structure: adopting a neural network based on sequence modeling, predicting future several steps of double-arm end trajectory blocks by inputting several steps of historical robot multi-modal sequences, and generating a smooth trajectory by weighted fusion of overlapping predicted trajectory blocks in the reasoning stage; (2.2) conditionally generating and denoising and correcting the double-arm end trajectory blocks in three-dimensional space to ensure that the trajectory meets the geometric constraints and has naturalness; (2.3) projecting the generated end trajectory into joint commands, combining inverse kinematics and constraint optimization methods to meet the joint limit, speed, acceleration and collision conditions; (3) reinforcement learning training: (3.1) constructing a comprehensive reward function, which includes task completion reward, double-arm collaboration consistency reward and action smoothness reward; (3.2) training the action generation structure by using a reinforcement learning strategy optimization method to maximize the expected return; And in the interactive sampling stage of reinforcement learning, the value function is used to weight the value of the action output by the strategy, guiding the sampling distribution to shift to the high-value area; (4) safe execution and deployment: (4.1) training in a simulation environment using curriculum learning and domain randomization; (4.2) performing system identification and parameter correction before deployment, and performing strategy distillation and delay compensation; (4.3) deploying the optimized strategy to a real robot, and ensuring execution safety through a safety constraint layer and a recovery mechanism.

2. The method of claim 1, wherein the method is based on reinforcement learning. In step (2.1), in the reasoning stage, exponential weighted fusion is used for overlapping prediction, and interpolation is performed in the three-dimensional rotation space to ensure continuity.

3. The method according to claim 1 or 2, wherein, Step (2.2) specifically comprises the following steps: First, the trajectory block is forwardly added with noise in the SE(3) tangent space, denoted as ; wherein, represents a noisy trajectory of the diffusion process at time step ; represents the original true trajectory; represents a noise schedule coefficient; represents a random noise subject to a standard Gaussian distribution ; Then, the denoising network is trained to predict the noise, and the loss function is wherein, denotes the training loss of the denoising network; denotes the noise predicted by the network; denotes the real added noise, and the natural trajectory is obtained by gradually denoising from the Gaussian noise in the inference stage.

4. The method of claim 1, wherein the method is based on reinforcement learning. The comprehensive reward function in step (3.1) is defined as: wherein, represents the task completion degree reward, represents the dual-arm coordination consistency reward, represents the motion smoothness reward, , , respectively represent , , corresponding weight coefficients.

5. The method of claim 1, wherein the method is based on reinforcement learning. In step (3.2), the policy network is updated using a policy gradient-based reinforcement learning algorithm, and the objective function for policy learning is ; wherein represents a policy optimization objective function; represents a policy network parameter; represents a policy distribution; is a discount factor; represents a reward at time t.

6. The method of claim 1, wherein the method is based on reinforcement learning. In step (3.2), the value guiding mechanism weights the action sampling based on the policy output distribution and the state-action value estimate, denoted as: where, denotes the guided re-weighted action distribution; denotes the action distribution of the original output of the policy network; denotes the state-action value function; denotes the guiding intensity coefficient.

7. The method of claim 1, wherein the method is based on reinforcement learning. In step (4.1), domain randomization includes perturbation of mass, friction, damping, illumination, texture and sensor noise.

8. The method of claim 1, wherein the method is based on reinforcement learning. The safety constraint layer in step (4.3) includes joint amplitude limiting, torque saturation, collision detection and failure recovery mechanism.

9. The method of claim 1, wherein the method is based on reinforcement learning. The double-arm body operation method further comprises: recording running data during real execution, and realizing continuous optimization of the strategy through offline retraining or residual learning.

10. The method of claim 1, wherein the method is based on reinforcement learning. In step (2.1), the action generation structure is a Transformer structure based on self-attention mechanism, which encodes several steps of historical robot state sequences and predicts several steps of double-arm end trajectory blocks.

Images