A humanoid robot dual-arm body operation method based on reinforcement learning

Abstract

The present application relates to a kind of humanoid robot dual-arm body operation method based on reinforcement learning, belong to the field of robot collaborative control, using trajectory block prediction+time integration fusion strategy, for dual-arm coordinated control, improve the continuity and stability of dual-arm operation;In reinforcement learning control, by introducing three-dimensional pose trajectory generation and correction module, three-dimensional space condition generation and denoising correction are carried out at trajectory block level, to ensure the geometric consistency and naturalness of generated trajectory;The present application also provides reinforcement learning optimization framework and simulation-real migration process, through the system integration of reward+value guide+migration process, to ensure the deployability and safety of strategy;Compared with prior art, the present application can significantly improve the naturalness, collaboration and success rate of dual-arm operation, with strong generalization ability, and has good simulation to reality migration ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot collaborative control, and particularly relates to a humanoid robot dual-arm body operation method based on reinforcement learning. BACKGROUND

[0002] As a typical representative of embodied intelligence, humanoid robots have made significant progress in technology breakthroughs and commercial applications in recent years, and are reshaping the production and lifestyle in multiple fields. The common methods for existing humanoid robots to perform dual-arm embodied operation tasks include trajectory planning and imitation learning.

[0003] Trajectory planning achieves precise control through predefined paths and motion sequences, and is suitable for repetitive tasks in structured environments. The core of trajectory planning is to generate an optimal motion path through mathematical modeling. In structured scenarios such as industrial assembly, this method shows significant advantages. However, this method has limited adaptability to dynamic environments. When encountering object position shifts or obstacle interference, manual re-planning of parameters is required, resulting in decreased efficiency. In addition, complex tasks such as flexible object grasping often require segmented planning and real-time adjustment, further increasing the computational burden. Therefore, trajectory planning methods are difficult to guarantee real-time and natural in high-dimensional complex environments.

[0004] Imitation learning enables robots to adapt to complex scenarios by observing human demonstrations or expert strategies. The core advantage of imitation learning is its adaptability to unstructured tasks. This method records human operations through vision or motion capture systems, extracts key motion features through behavior cloning or inverse reinforcement learning, and finally generates executable robot strategies. However, imitation learning faces two major challenges: one is data dependency, which requires a large number of high-quality demonstration samples to cover operation variants; the other is strategy generalization, when environmental parameters exceed the training range, performance will decrease significantly. That is, imitation learning relies on demonstration data, has poor generalization, and is prone to error accumulation, making it difficult to guarantee the absolute accuracy of motion. In tasks requiring millimeter-level positioning, trajectory planning may need to be combined for post-processing.

[0005] Reinforcement Learning (RL) has the ability to optimize the policy through interaction, and can learn the optimal policy through the interaction between the agent and the environment without explicit programming. In high-dimensional action space, traditional methods need to manually design the dependency relationship between action dimensions, while RL can autonomously explore high-dimensional combined strategies, such as coordinating multiple joint movements in robot control. This flexibility makes RL irreplaceable in embodied intelligence; however, high-dimensional action space also brings significant challenges - dimension explosion leads to a sharp decline in exploration efficiency, policy convergence is difficult, and it is easy to fall into local optimum; in addition, the natural neglect of RL to action smoothness may cause mechanical vibration or operation interruption, further limiting its practicality in physical systems; in high-dimensional action space, RL faces three core challenges: insufficient convergence stability, lack of action smoothness, and difficulty in simulation to reality transfer.

[0006] Therefore, there is an urgent need for a dual-arm embodied operation method that combines action generation optimization and reinforcement learning training, which can not only ensure task completion but also enhance the naturalness of actions and the ability of real-world deployment. SUMMARY

[0007] In view of the defects of the prior art, the present application provides a dual-arm embodied operation method for humanoid robots based on reinforcement learning, which aims to solve the problems of unnatural dual-arm operation, insufficient coordination, poor generalization ability and unstable real-world deployment in the prior art.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a dual-arm embodied operation method for humanoid robots based on reinforcement learning, comprising the following steps:

[0009] (1) obtaining robot multi-modal observation information, including visual data, joint state, force sensation signal and target object pose;

[0010] (2) trajectory generation and optimization:

[0011] (2.1) constructing an action generation structure: using a neural network based on sequence modeling, inputting the robot multi-modal sequence of a certain number of historical steps, predicting the future trajectory block of the dual-arm end for a certain number of steps, and generating a smooth trajectory by weighted fusion of overlapping predicted trajectory blocks in the reasoning stage;

[0012] (2.2) conditionally generating and denoising the dual-arm end trajectory block in three-dimensional space to ensure that the trajectory meets the geometric constraints and has naturalness;

[0013] (2.3) projecting the generated end trajectory into joint commands, combining inverse kinematics and constraint optimization methods to meet the joint limit, speed, acceleration and collision conditions;

[0014] (3) Reinforcement learning training:

[0015] (3.1) Construct a comprehensive reward function, which includes task completion reward, dual-arm coordination consistency reward and action smoothness reward;

[0016] (3.2) Train the action generation structure using a reinforcement learning strategy optimization method to maximize the expected return; and in the interactive sampling stage of reinforcement learning, use the value function to weight the value of the policy output action, guiding the sampling distribution to shift to the high value area;

[0017] (4) Safe execution and deployment:

[0018] (4.1) Train in a simulation environment using curriculum learning and domain randomization;

[0019] (4.2) Perform system identification and parameter correction before deployment, and perform policy distillation and delay compensation;

[0020] (4.3) Deploy the optimized policy to a real robot, and ensure execution safety through a safety constraint layer and a recovery mechanism.

[0021] Further, in step (2.1), in the inference stage, exponential weighted fusion is used for overlap prediction, and the rotation part is interpolated in three-dimensional rotation space to ensure continuity.

[0022] Further, step (2.2) specifically includes the following steps:

[0023] First, add noise to the trajectory block in the SE(3) tangent space, denoted as ;

[0024] wherein represents the noisy trajectory of the diffusion process at time step ; represents the original real trajectory; represents the noise scheduling coefficient; represents random noise subject to a standard Gaussian distribution ;

[0025] Then, train the denoising network to predict noise, and the loss function is , wherein represents the training loss of the denoising network; represents the noise predicted by the network; represents the real added noise, and in the inference stage, the natural trajectory is obtained by gradually denoising from Gaussian noise.

[0026] Further, the comprehensive reward function in step (3.1) is defined as: , wherein, represents a task completion degree reward, represents a dual-arm coordination consistency reward, represents a motion smoothness reward, , , respectively represent , , corresponding weight coefficients.

[0027] Further, in step (3.2), the policy network is updated using a policy gradient-based reinforcement learning algorithm, and the target function of policy learning is ; wherein represents a policy optimization target function; represents a policy network parameter; represents a policy distribution; is a discount factor; represents a reward at time .

[0028] Further, in step (3.2), the value guidance mechanism weights the action sampling based on the policy output distribution and the state-action value evaluation, which is represented as: , wherein represents a reweighted action distribution after guidance; represents an action distribution of the original output of the policy network; represents a state-action value function; represents a guidance intensity coefficient.

[0029] Further, in step (4.1), domain randomization includes perturbation of mass, friction, damping, lighting, texture, and sensor noise.

[0030] Further, the safety constraint layer in step (4.3) includes joint amplitude limiting, torque saturation, collision detection, and failure recovery mechanism.

[0031] Further, the dual-arm body operation method further comprises: recording running data during real execution, and realizing continuous optimization of the policy through offline retraining or residual learning.

[0032] Further, in step (2.1), the action generation structure is a Transformer structure based on self-attention mechanism, which encodes the robot state sequence of a plurality of historical steps and predicts the dual-arm end trajectory block of a plurality of future steps.

[0033] Beneficial effects:

[0034] Compared with the prior art, the application can significantly improve the naturalness, cooperation and success rate of double-arm operation, has strong generalization ability and good simulation-to-reality migration ability.

[0035] The application adopts a double-arm action sequence generation mechanism based on trajectory blocks, represents double-arm operation actions as future multi-step end trajectory blocks instead of single-step actions, outputs long-time span trajectory sequences through time sequence modeling, uses exponential weighted fusion to generate smooth and continuous control instructions for overlapping trajectories in the reasoning phase, significantly reduces long-time sequence error accumulation problems, and uses a trajectory block prediction + time integration fusion strategy for double-arm coordinated control, which improves the continuity and stability of double-arm operation compared with traditional single-step prediction or trajectory planning methods.

[0036] The application embeds three-dimensional trajectory denoising correction in reinforcement learning control, introduces a three-dimensional pose trajectory generation and correction module, performs conditional generation and denoising correction in three-dimensional space at the trajectory block level, guarantees the geometric consistency and naturalness of generated trajectories, effectively avoids problems such as joint trajectory jitter and end conflict, makes double-arm actions conform to human operation habits, and improves the naturalness of double-arm actions; the mechanism is decoupled from the reinforcement learning framework and can embed any strategy network as a trajectory prior generator.

[0037] The application provides a reinforcement learning optimization framework and a simulation-to-reality migration process, optimizes the reinforcement learning mechanism by designing a comprehensive reward function (task completion degree, double-arm cooperation consistency, and action smoothness) and introducing a value guidance mechanism in sampling, so that the strategy converges faster and is more stable, builds a complete simulation-to-reality migration process: uses curriculum learning and domain randomization training in simulation, deploys to real robots after system identification, strategy distillation and delay compensation, and introduces a safety constraint layer (including joint amplitude limiting, collision detection, and failure recovery) in the deployment stage to ensure execution safety, effectively solves the landing deployment problem of high-dimensional continuous action reinforcement learning in real double-arm operation, and guarantees the deployability and safety of the strategy through the system integration of reward + value guidance + migration process. BRIEF DESCRIPTION OF DRAWINGS

[0038] Fig. 1 It is the overall framework diagram of the double-arm body operation method of the humanoid robot of the application.

[0039] Fig. 2 It is the flowchart of the training-deployment of the application. DETAILED DESCRIPTION

[0040] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0041] As Figs. 1-2As shown, in this embodiment, a humanoid robot dual-arm body operation method based on reinforcement learning includes: acquiring multi-modal state information; generating smooth dual-arm trajectories by trajectory block prediction and weighted fusion; ensuring natural motion and geometric reasonableness through three-dimensional pose trajectory generation and correction; combining reward functions including task completion, collaborative consistency and motion smoothness for reinforcement learning optimization, and introducing a value guidance mechanism to improve returns in sampling; using curriculum learning and domain randomization training in simulation, deploying to real robots after system identification, policy distillation and delay compensation, and combining safety constraint layers to ensure execution safety; the specific steps are as follows.

[0042] Step (1) Environment perception and state acquisition: multi-modal state observation information of the robot is collected through sensors, including dual-arm joint angle and angular velocity, torso posture, force and tactile signals, three-dimensional pose of the target object, and visual environment information.

[0043] According to the acquired information, a state space is constructed, the state space contains image features , body state , , target pose , ; specifically, : image features at the current time (visual input); : joint angle vector at the current time; : joint angular velocity vector at the current time; : three-dimensional position of the target object; : three-dimensional pose of the target object (rotation matrix or quaternion).

[0044] Based on the state space, define the action space, the action space is the end pose sequence of the future step; ; wherein, : future step action trajectory; : end position of the first step; : end pose of the first step.

[0045] Step (2) Trajectory generation and optimization.

[0046] (2.1) Construct an action generation structure, which is a neural network module based on time series modeling, the input of which is a historical state sequence, i.e. a robot multi-modal state sequence of the last several steps (including double-arm joint angle, angular velocity, force signal and target object pose, etc.), and the output is a double-arm end trajectory block of the future several steps; in the inference stage, there is an overlapping part between the trajectory blocks predicted at adjacent time points, and the system uses exponential weighted fusion to generate a final executed smooth trajectory for these overlapping trajectory segments, so as to reduce the accumulation of long-time series prediction error.

[0047] In this embodiment, the Transformer structure based on self-attention mechanism is adopted to encode the robot state sequence of the last several steps and predict the double-arm end trajectory block of the future several steps, so as to realize the continuous generation of long-time series action.

[0048] In the inference stage, the overlapping predicted trajectory blocks are exponentially weighted and averaged to generate a smooth and coherent double-arm action instruction, which reduces the accumulation of prediction error. Specifically, the strategy outputs future step action blocks, and uses the time-integrated exponential weighted average formula to weight and fuse the overlapping predictions, and the rotation part is logarithm-exponential averaged or uses SLERP interpolation in SO(3) tangent space to ensure continuity; wherein, : the fused step action; : the th trajectory block in the th step prediction action; : exponential decay coefficient for controlling the weight decrease in fusion.

[0049] (2.2) Three-dimensional pose trajectory generation and correction: conditionally generating and denoising the end trajectory block in three-dimensional space to ensure the geometric reasonableness and naturalness of the generated trajectory.

[0050] First, the trajectory block is forwardly added with noise in the SE(3) tangent space, and the trajectory formula with noise is expressed as ; wherein, : the noisy trajectory at time step : the original real trajectory; : noise scheduling coefficient (decay coefficient); : random noise obeying standard Gaussian distribution

[0051] Then, the denoising network is trained to predict the noise, and the loss function is: , wherein, : the training loss of the denoising network; : represents the noise predicted by the network;​​ : Real added noise, gradually denoising from Gaussian noise in the inference stage to get natural trajectory.

[0052] In the trajectory generation process, noise is gradually added and denoised in the training stage to improve the naturalness of the generated trajectory.

[0053] (2.3) Project the generated end trajectory into joint commands, combine inverse kinematics and constraint optimization methods to meet joint limit, speed, acceleration and collision conditions.

[0054] Step (3) reinforcement learning training.

[0055] (3.1) Reward function design: construct a comprehensive reward function, including task completion reward, dual-arm collaboration consistency reward and action smoothness reward, to guide policy optimization.

[0056] The comprehensive reward function is defined as: , wherein represents the task completion reward, represents the dual-arm collaboration consistency reward, represents the action smoothness reward, , , respectively represent , , corresponding weight coefficients, which can be determined by experience setting and parameter search, with priority to task completion, and then considering collaboration and action smoothness.

[0057] The selection of weight coefficients needs to be considered comprehensively according to the task. Initially, they can be set as =1.0, =0.5, =0.2, to ensure that task completion is prioritized. In training, they can be appropriately fine-tuned according to the learning curve. The adjustment principle is: if the task success rate is low, increase ; if the dual-arm collaborative action is out of sync, increase ; if the trajectory jitter is large, increase .

[0058] (3.2) Reinforcement learning optimization: use policy gradient-based reinforcement learning algorithm (such as PPO) to update the policy network, and introduce value guidance mechanism in the sampling process to make the action distribution biased towards high return area.

[0059] (3.2.1) Optimize the strategy using PPO algorithm: maximize the expected value of the reward function in reinforcement learning. The objective function formula of policy learning is: ; wherein : policy optimization objective function; : policy network parameters; : policy distribution; : discount factor; : : reward at time step t.

[0060] (3.2.2) Q-value guided sampling: modify the policy distribution in the sampling stage, and guide the action sampling based on the value function. The formula of value guided sampling is: where, : re-weighted action distribution after guidance; : action distribution of the original output of the policy network; : state-action value function; : guidance intensity coefficient.

[0061] In the sampling stage of reinforcement learning, the state-action value of the action is calculated as the weight to re-weight the policy output distribution, so that the sampling result is biased towards high-value actions, thereby improving the policy learning efficiency.

[0062] (4) Safe execution and deployment.

[0063] (4.1) Simulation training and domain randomization: train in a simulation environment, correct parameters through system identification, and use curriculum learning (from basic grasping, carrying to complex assembly) and domain randomization (dynamic parameters, friction coefficient, noise and time delay disturbance) to improve the policy generalization ability.

[0064] Curriculum learning: provide simple tasks (such as grasping stationary objects, light mass objects) at the beginning of training, and then gradually increase the difficulty of tasks (heavier targets, objects moving, more complex operation space), and gradually unlock tasks by controlling the training environment parameters.

[0065] Domain randomization: randomly perturb robot and environment parameters during training, including dynamic parameters, mass, friction coefficient, joint damping, joint control delay, sensor noise, lighting conditions, texture, etc., so that the policy can still work stably in a changing environment.

[0066] (4.2) System identification and policy transfer: calibrate the real robot dynamics parameters through system identification before real deployment, and perform policy distillation and delay compensation to improve the real-time performance and stability of the policy.

[0067] (4.3) Real deployment and safety protection: deploy the optimized policy on the real robot, and guarantee the operation safety through safety constraint layer and recovery mechanism.

[0068] ​The safety constraint layer includes joint amplitude, torque saturation, collision detection and failure recovery mechanism. The joint amplitude limits the joint angle, angular velocity and torque command of the strategy output, and the part exceeding the threshold is truncated to the safe range.

[0069] Collision detection: use force sensor or current / torque monitoring, stop movement immediately and retreat to a safe position when abnormal contact force is detected.

[0070] Failure recovery mechanism: if task failure is detected (such as grasping failure, object falling, large position error, etc.), trigger the recovery strategy to move the dual arms back to the initial safe posture and wait for re-execution.

[0071] Through the above steps (1)-(4), the invention can realize the collaborative control of humanoid robot dual arms with body operation, which can significantly improve the naturalness, collaboration and success rate of dual arm operation, and has good simulation to reality migration ability.

[0072] Further, on the basis of the above steps (1)-(4), the method of the invention can also have an online fine-tuning link: record the running data during real-time execution, and realize continuous optimization of the strategy through offline retraining or residual learning; it should be noted that this online fine-tuning link is a further optimization of the method of the invention, which can be flexibly selected in actual use, and is not a necessary step.

[0073] Compared with the prior art, the invention has the following advantages:

[0074] (1) Naturalness improvement: through three-dimensional pose trajectory correction and reward function constraint, guide the robot to generate natural movements closer to human operation habits;

[0075] (2) Collaboration enhancement: use sequence trajectory block prediction and fusion mechanism to ensure smooth and consistent dual arm movements and improve collaborative operation performance;

[0076] (3) Strong generalization ability: combine curriculum learning and domain randomization to enable the robot to adapt to different object shapes, materials and external disturbances, and improve generalization and stability.

[0077] The above is only the preferred embodiment of the invention, and does not limit the invention in any form. Although the invention has been disclosed as above, it is not intended to limit the invention. Any skilled person in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the invention, and any simple modification, equivalent change and modification of the above embodiment based on the technical essence of the invention are still within the scope of the invention.

Claims

1. A method for dual-arm embodied manipulation of a humanoid robot based on reinforcement learning, characterized in that, Includes the following steps: (1) Acquire multimodal observation information of the robot, including visual data, joint status, force signals and target object pose; (2) Trajectory generation and optimization: (2.1) Constructing the motion generation structure: Using a sequence modeling-based neural network, the robot predicts the trajectory blocks of the two arms end in the next few steps by inputting the robot's multimodal sequence of several historical steps, and weighted fusion of the overlapping predicted trajectory blocks to generate a smooth trajectory during the inference stage; (2.2) In three-dimensional space, the trajectory blocks at the ends of the two arms are generated and denoised to ensure that the trajectory conforms to geometric constraints and has naturalness; (2.3) Project the generated end trajectory as joint commands, and combine inverse kinematics and constraint optimization methods to satisfy joint limits, velocity, acceleration and collision conditions; (3) Strengthen learning and training: (3.1) Construct a comprehensive reward function, which includes task completion reward, bi-arm coordination consistency reward and motion smoothness reward; (3.2) The action generation structure is trained using a reinforcement learning strategy to maximize the expected reward; Furthermore, during the interactive sampling phase of reinforcement learning, the value function is used to weight the policy output actions, guiding the sampling distribution to shift towards high-value regions. (4) Security Execution and Deployment: (4.1) Training is performed in a simulation environment using course learning and domain randomization; (4.2) Before deployment, system identification and parameter correction should be performed, and strategy distillation and latency compensation should be carried out; (4.3) Deploy the optimized strategy to the real robot and ensure execution safety through a safety constraint layer and recovery mechanism.

2. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, In step (2.1), during the inference stage, the overlapping prediction is fused by exponential weighting, and the rotation part is interpolated in the three-dimensional rotation space to ensure continuity.

3. A method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1 or 2, characterized in that, Step (2.2) specifically includes the following steps: First, the trajectory block is positively denoised in the SE(3) tangent space, denoted as: ; in, This indicates the diffusion process at time step The noisy trajectory; Represents the original true trajectory; Indicates the noise dispatch coefficient; It indicates that it follows a standard Gaussian distribution. Random noise; Then, train the denoising network to predict noise, with the loss function being... ,in, This represents the training loss of the denoising network; This represents the noise in the network predictions; This represents the noise actually added. During the inference phase, noise is gradually removed starting from Gaussian noise to obtain the natural trajectory.

4. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, The comprehensive reward function in step (3.1) is defined as follows: ,in, Indicates the reward for task completion. This indicates a reward for the coordination and consistency of both arms. Indicates reward for smoothness of movement. , , They represent , , The corresponding weighting coefficients.

5. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, In step (3.2), a policy gradient-based reinforcement learning algorithm is used to update the policy network. The objective function of policy learning is: ;in, This represents the objective function for policy optimization. Indicates the policy network parameters; Represents the policy distribution; Discount factor; express A reward for every moment.

6. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, In step (3.2), the value guidance mechanism weights the action sampling based on the policy output distribution and state-action value assessment, as follows: ,in, This indicates the action distribution that has been reweighted after guidance; This represents the action distribution of the original output of the policy network; State-action value function; This represents the guiding strength coefficient.

7. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, In step (4.1), domain randomization includes perturbations of mass, friction, damping, illumination, texture, and sensor noise.

8. The method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, The safety constraint layer in step (4.3) includes joint limiting, torque saturation, collision detection, and failure recovery mechanisms.

9. A method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, The dual-arm embodied operation method also includes: recording operational data during actual execution and continuously optimizing the strategy through offline retraining or residual learning.

10. A method for dual-armed humanoid robot manipulation based on reinforcement learning according to claim 1, characterized in that, In step (2.1), the action generation structure is a Transformer structure based on a self-attention mechanism, which encodes the robot state sequence of several historical steps and predicts the end-effector trajectory blocks of the two arms for several future steps.

Images