A method and related apparatus for generating robot motion based on human-robot collaboration enhancement

By using human-machine collaborative reinforcement learning, robots can make autonomous decisions in complex environments and integrate human experience, solving the problem of insufficient adaptability in existing technologies and achieving efficient and safe task execution and skill enhancement.

CN123103671APending Publication Date: 2026-11-17INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611516500.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-28
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing basic robot models are not adaptable enough to new tasks, new environments and complex operation scenarios. Supervised fine-tuning methods have high data acquisition costs and low efficiency. Reinforcement learning methods are limited in application in real robot systems. The quality of human intervention depends on the operator's experience and lacks long-term accumulation and knowledge distillation capabilities.

Method used

A robot motion generation method based on human-machine collaboration reinforcement is adopted. By combining a pre-trained world motion model with historical interaction experience data for reinforcement learning optimization and reward distillation training, the future predicted world state and robot motion are generated. Based on the task execution risk value, autonomous execution or human-machine collaborative intervention mode is selected, and human input actions are integrated to improve autonomous decision-making and safety.

Benefits of technology

It improves the robot's autonomous decision-making ability and robustness in complex tasks, reduces the burden of human operation, realizes the dynamic integration of robot autonomous strategies and human experience, and improves task success rate and generalization ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103671A_ABST
    Figure CN123103671A_ABST
Patent Text Reader

Abstract

This invention belongs to the field of robot intelligent control and discloses a method and related apparatus for generating robot actions based on human-machine collaboration reinforcement. Based on task description information, current world state information, and robot actions, a pre-trained world action model is generated using reinforcement learning optimization and reward distillation training based on historical interaction experience data samples. This model generates predicted future world state information and predicted robot actions. Based on the predicted future world state information and predicted robot actions, a task execution risk value is obtained, and an autonomous execution mode or a human-machine collaborative intervention mode is selected. When the human-machine collaborative intervention mode is selected, a human-machine collaboration request information is generated, and human input actions are obtained. The predicted robot actions are then fused with the human input actions to obtain the robot actions. This enables the robot to shift from passive response to active decision-making, dynamically integrating autonomous strategies and human experience, effectively improving task execution quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control, and relates to a method and related device for generating robot motion based on human-machine collaboration enhancement. Background Technology

[0002] With the rapid development of embodied intelligence and general robotics, a new generation of robot foundation models, represented by Vision-Language-Action (VLA) and World Action Model (WAM), are gradually demonstrating strong task generalization and complex operation capabilities. However, most existing robot foundation models rely on large-scale offline data training, and they still suffer from insufficient adaptability when facing new tasks, new environments, and complex contact operation scenarios.

[0003] Currently, the main technical approaches for robot skill transfer and task adaptation are supervised fine-tuning and reinforcement learning. Supervised fine-tuning typically requires collecting large amounts of high-quality teleoperation teaching data, resulting in high data acquisition costs, low efficiency, and difficulty in covering long-tail states in complex scenarios. While reinforcement learning can continuously optimize policy performance through environmental interaction, its application in real-world robot systems remains significantly limited due to issues such as high trial-and-error costs, sparse rewards, low exploration efficiency, and poor training stability. In recent years, some research has attempted to improve the data efficiency of reinforcement learning in real-world robots using human-robot collaborative control and human intervention mechanisms. However, existing methods often rely on direct human intervention in the robot's control process, and the quality of human operation is highly dependent on the operator's experience, making it difficult to guarantee the consistency and expert level of intervention data. Furthermore, most existing methods treat human intervention only as a temporary error correction mechanism, lacking the ability to accumulate long-term human experience and distill knowledge, thus preventing the robot from continuously improving its skill level. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for generating robot motion based on human-machine collaboration enhancement.

[0005] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention provides a method for generating robot actions based on human-machine collaboration reinforcement, comprising: acquiring task description information, current world state information, and robot actions; generating future predicted world state information and predicted robot actions based on a pre-trained world action model, according to the task description information, current world state information, and robot actions; obtaining a task execution risk value based on the future predicted world state information and predicted robot actions; selecting an autonomous execution mode or a human-machine collaborative intervention mode based on the task execution risk value; when selecting the autonomous execution mode, obtaining robot actions based on the predicted robot actions; when selecting the human-machine collaborative intervention mode, generating human-machine collaboration request information and acquiring human input actions, and fusing the predicted robot actions with the human input actions to obtain robot actions; wherein the pre-trained world action model is obtained by: acquiring a world action model and historical interaction experience data samples, and performing reinforcement learning optimization and reward distillation training on the world action model based on the historical interaction experience data samples to obtain the pre-trained world action model; the historical interaction experience data samples are collected based on historical task execution processes and include world state information, robot actions, and action reward information.

[0006] Optionally, the world state information includes environmental observation information, robot body state information, and interaction information; the environmental observation information includes one or more of RGB images, depth images, point cloud information, and semantic information; the robot body state information includes one or more of joint angles, joint velocities, end effector poses, and robot motion history information; the interaction information includes one or more of contact information, force information, torque information, slip information, friction state information, and task stage information.

[0007] Optionally, obtaining the task execution risk value based on the predicted future world state information and the predicted robot actions includes: obtaining the risk value of each preset risk scenario based on the predicted future world state information and the predicted robot actions, and weighting and summing the risk values ​​of each preset risk scenario to obtain the task execution risk value; wherein, the preset risk scenarios include one or more of the following: task failure risk, abnormal contact risk, action deviation risk, collision risk, slippage risk, environmental change risk, and strategy uncertainty risk.

[0008] Optionally, the step of selecting the autonomous execution mode or the human-machine collaborative intervention mode based on the task execution risk value includes: selecting the autonomous execution mode when the task execution risk value is not greater than a preset risk threshold; and selecting the human-machine collaborative intervention mode when the task execution risk value is greater than the preset risk threshold.

[0009] Optionally, fusing the predicted robot action with the human input action to obtain the robot action includes: obtaining the confidence level and the degree of change in the world state of the predicted robot action, and combining it with the task execution risk value to obtain the fusion weight of the predicted robot action and the human input action, and fusing the predicted robot action with the human input action according to the fusion weight to obtain the robot action.

[0010] Optionally, before fusing the predicted robot action with the human input action, the method further includes: correcting the human input action based on the current world state information, the predicted future world state information, and the human input action, using a preset human input action correction model; wherein the preset human input action correction model is obtained by modeling the mapping relationship between the human input action and the expert action and combining it with pre-training.

[0011] Optionally, it also includes: obtaining risk event identification results based on a preset risk event identification model according to the robot's actions and the predicted future world state information; and when the risk event identification result indicates the existence of a risk event, generating robot risk elimination actions to replace the robot's actions based on the existing risk event; wherein, the risk events include collision events, assembly failure events, abnormal contact events, and task deviation events; and the robot risk elimination actions include compensation actions, recovery actions, or search actions.

[0012] Optionally, the step of optimizing the world action model using reinforcement learning and reward distillation training based on historical interaction experience data samples to obtain a pre-trained world action model includes: optimizing the world action model using online reinforcement learning based on historical interaction experience data samples to obtain a first optimized world action model; and performing offline reward distillation training on the world action model based on historical interaction experience data samples to obtain a second optimized world action model; and fusing the model parameters of the first optimized world action model and the second optimized world action model to obtain a pre-trained world action model; wherein, in the offline reward distillation training, high-reward trajectory data and low-reward trajectory data are obtained based on historical interaction experience data samples, and high-reward trajectory data are used to form positive samples, and low-reward trajectory data are used to form negative samples; the world action model is trained with the optimization objective of increasing the probability of the world action model outputting the corresponding robot action under positive samples and decreasing the probability of outputting the corresponding robot action under negative samples.

[0013] Optionally, the action reward information is constructed using a staged dense reward format; wherein, the stage includes one or more of the following: target search stage, target approach stage, posture alignment stage, initial contact stage, stable contact stage, operation execution stage, and task completion stage; and the reward includes one or more of the following: distance reward, posture reward, contact reward, progress reward, task completion reward, and safety reward.

[0014] Optionally, when generating future predicted world state information and predicted robot actions based on a pre-trained world action model using task description information, current world state information, and robot actions, the process further includes: acquiring short-term interaction memory information and long-term skill memory information, and using these as auxiliary information, along with the task description information, current world state information, and robot actions, inputting them into the pre-trained world action model to generate future predicted world state information, predicted robot actions, and predicted interaction information; wherein, short-term interaction memory information includes one or more of the following: robot state change information, robot action change information, contact change information, human intervention events, and risk change information within a preset time window; long-term skill memory information includes one or more of the following: task trajectory information generated during historical task execution, successful experience trajectory, failed experience trajectory, task stage transition information, and key interaction event information.

[0015] Optionally, it also includes: collecting current interaction experience data samples based on the current task execution process, and performing reinforcement learning optimization and reward distillation training on the pre-trained world action model based on the current interaction experience data samples and historical interaction experience data samples, and updating the pre-trained world action model.

[0016] In a second aspect, the present invention provides a robot motion generation system based on human-machine collaboration reinforcement, comprising: a data acquisition module for acquiring task description information, current world state information, and robot motion; a motion prediction module for generating future predicted world state information and predicted robot motion based on a pre-trained world motion model, according to the task description information, current world state information, and robot motion; a risk assessment module for obtaining a task execution risk value based on the future predicted world state information and predicted robot motion; selecting an autonomous execution mode or a human-machine collaborative intervention mode based on the task execution risk value; and a motion optimization module for obtaining robot motion based on the predicted robot motion when the autonomous execution mode is selected; and generating human-machine collaboration request information and acquiring human input motion when the human-machine collaborative intervention mode is selected, and fusing the predicted robot motion with the human input motion to obtain robot motion; wherein the pre-trained world motion model is obtained by: acquiring a world motion model and historical interaction experience data samples, and performing reinforcement learning optimization and reward distillation training on the world motion model based on the historical interaction experience data samples to obtain the pre-trained world motion model; the historical interaction experience data samples are collected based on historical task execution processes and include world state information, robot motion, and motion reward information.

[0017] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the robot motion generation method based on human-machine collaboration enhancement.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the robot motion generation method based on human-machine collaboration enhancement.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention relates to a robot motion generation method based on human-machine collaboration enhancement. Based on task description information, current world state information, and robot actions, it generates predicted future world state information and predicted robot actions using a pre-trained world action model. This enables the robot to have long-term predictive capabilities regarding environmental state changes, task execution results, and risk evolution, improving its autonomous decision-making ability and robustness in complex tasks. Next, based on the predicted future world state information and predicted robot actions, a task execution risk value is obtained. Based on this risk value, an autonomous execution mode or a human-machine collaborative intervention mode is selected, enabling the robot to shift from passive response to proactive predictive decision-making. It maintains autonomous and efficient execution under low-risk conditions and introduces timely human intervention under high-risk conditions. The predicted robot actions are fused with human input actions to obtain the final robot action. This method ensures task execution safety while fully utilizing human experience constraints and autonomous decision-making capabilities, improving the quality of human intervention and data utilization efficiency, reducing the human operational burden in complex tasks, and achieving a dynamic fusion between robot autonomous strategies and human experience. The pre-trained world action model is obtained by optimizing the world action model through reinforcement learning and reward distillation training based on historical interaction experience data samples. This enables the continuous accumulation and efficient utilization of high-value interaction experience under real robot conditions, improving the robot's policy learning efficiency and sample utilization efficiency. It allows the robot to continuously acquire high-value interaction experience under limited human intervention and continuously improve its operational capabilities and task generalization capabilities through a closed loop of online exploration and offline distillation. This invention is applicable to various complex task scenarios such as plug-in assembly, tool operation, button interaction, door handle operation, compliant assembly, complex contact operations, and humanoid robot dual-arm collaborative operation. It can effectively improve the robot's task success rate, autonomy, generalization ability, and engineering application performance, and has good application value and promotion prospects. Attached Figure Description

[0020] Figure 1 This is a flowchart of a robot motion generation method based on human-machine collaboration enhancement according to an embodiment of the present invention.

[0021] Figure 2 This is a block diagram of a robot motion generation system based on human-machine collaboration enhancement, according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a robot motion generation method based on human-machine collaboration enhancement is provided to solve the problem that existing methods usually rely on direct human intervention in robot control, and the quality of human operation is highly dependent on the operator's experience, making it difficult to adapt to complex task scenarios.

[0025] Specifically, the robot motion generation method based on human-machine collaboration enhancement of the present invention includes the following steps: S1: Obtain task description information, current world state information, and robot actions.

[0026] S2: Based on the task description information, the current world state information, and the robot's actions, generate the predicted world state information and predicted robot actions for the future based on the pre-trained world action model.

[0027] S3: Based on the predicted world state information and predicted robot actions, obtain the task execution risk value; select the autonomous execution mode or the human-machine collaborative intervention mode based on the task execution risk value.

[0028] S4: When the autonomous execution mode is selected, the robot action is obtained based on the predicted robot action; when the human-machine collaborative intervention mode is selected, human-machine collaborative request information is generated and human input action is obtained, and the predicted robot action and human input action are fused to obtain the robot action.

[0029] The pre-trained world action model is obtained by acquiring the world action model and historical interaction experience data samples, and performing reinforcement learning optimization and reward distillation training on the world action model based on the historical interaction experience data samples to obtain the pre-trained world action model; the historical interaction experience data samples are collected based on the historical task execution process and include world state information, robot actions and action reward information.

[0030] Specifically, in step S1, the task description information refers to information describing the task that the robot needs to perform. This information may include task objectives, task type, task constraints, task stage information, etc. For example, the task description information may be provided in the form of natural language instructions, task identifiers, target images, or task parameters. For instance, in a mating assembly task, the task description information may include the location of the holes to be mated, the required mating depth, and the allowable contact force range; in a button operation task, the task description information may include the location of the target button and the pressing stroke.

[0031] Specifically, in step S1, the current world state information refers to all the information about the external environment and the robot's own state that the robot has perceived and recorded from the start of the task to the current moment.

[0032] In one possible implementation, the world state information includes environmental observation information, robot body state information, and interaction information. The environmental observation information includes one or more of RGB images, depth images, point cloud information, and semantic information. The robot body state information includes one or more of joint angles, joint velocities, end effector pose, and robot motion history information. The interaction information includes one or more of contact information, force information, torque information, slip information, friction state information, and task stage information.

[0033] Explanatoryly, since complex robot operation tasks typically involve multimodal information such as visual perception, body state perception, and contact interaction perception, by dividing world state information into three categories—environmental observation information, robot body state information, and interaction information—it can comprehensively cover all types of information required during task execution. This enables the comprehensive collection and structured organization of multimodal information during robot task execution, providing a complete and accurate data foundation for subsequent world action model prediction.

[0034] Specifically, in step S1, the robot action up to the current moment refers to the sequence of actions performed by the robot from the start of the task to the current moment. For example, the robot action may include the target angle, target velocity, target torque of each joint, or the target pose of the end effector.

[0035] For example, world state information and robot actions can be acquired through the robot's own sensor system, vision acquisition system, and external monitoring system, and then processed through time synchronization and state alignment to form a unified data sequence. Explain, time synchronization and state alignment processes ensure that data from different sources are aligned in time and consistent in space, thereby providing uniformly formatted input data for the world action model.

[0036] Specifically, in step S2, the pre-trained world action model is a pre-trained model capable of learning the dynamic evolutionary relationship between robot actions, environmental states, and the task execution process. Specifically, the pre-trained world action model learns the coupling relationship between robot actions and changes in environmental states by jointly modeling world state information and robot actions, thereby enabling the prediction of future world states and robot actions.

[0037] Specifically, the pre-trained world action model is obtained by performing reinforcement learning optimization and reward distillation training on the world action model based on historical interaction experience data samples. The world action model can adopt a commonly used world action model in the field of robot control. Interpretably, the world action model simultaneously learns the laws governing environmental state changes, robot action generation, and the dynamic laws governing the interaction between the robot and the environment during modeling, and uses the predicted future state information to assist in predicting robot actions. The laws governing environmental state changes refer to the evolution of the environment's state under robot intervention or without robot intervention; the laws governing robot action generation refer to the way appropriate actions are generated based on the current state; and the dynamic laws governing interaction refer to the changes in contact forces, contact positions, and other interaction quantities between the robot and the environment. By simultaneously learning these three types of laws, the world action model can achieve coupled modeling of world state information prediction and robot action generation within a unified framework.

[0038] Optionally, the world-action model can adopt a joint state-action modeling approach. This approach learns the coupling relationship between state evolution and action generation by uniformly encoding world state information sequences and robot action sequences. During the training phase, it utilizes future state supervision to learn the dynamic changes in the latent space, while directly generating predicted world state information and predicted robot actions during the inference phase. This eliminates the need for rolling future state predictions or future imagination during testing, thereby improving inference efficiency and real-time performance. The joint state-action modeling approach means that world states and robot actions are jointly encoded and processed in the same model, rather than establishing separate state prediction and action generation models.

[0039] Specifically, in step S3, the task execution risk value is a comprehensive quantitative assessment result of various risk situations that may occur when the robot performs the predicted robot actions.

[0040] In one possible implementation, obtaining the task execution risk value based on the predicted future world state information and predicted robot actions includes: obtaining risk values ​​for each preset risk scenario based on the predicted future world state information and predicted robot actions; and weighting and summing the risk values ​​of each preset risk scenario to obtain the task execution risk value. The preset risk scenarios include one or more of the following: task failure risk, abnormal contact risk, action deviation risk, collision risk, slippage risk, environmental change risk, and strategy uncertainty risk. Explained, by weighting and summing various risk scenarios, the overall risk level of task execution can be comprehensively reflected, thereby providing a quantitative basis for subsequent mode selection.

[0041] For example, the weights can be set based on the task type, task stage, and historical experience. For instance, in the initial contact stage of a mating assembly task, the weights of collision risk and abnormal contact risk can be set higher; in the stable contact stage, the weights of slippage risk and task failure risk can be set higher.

[0042] In one possible implementation, the step of selecting the autonomous execution mode or the human-machine collaborative intervention mode based on the task execution risk value includes: selecting the autonomous execution mode when the task execution risk value is not greater than a preset risk threshold; and selecting the human-machine collaborative intervention mode when the task execution risk value is greater than the preset risk threshold.

[0043] Explained, the preset risk threshold is a risk value boundary used to distinguish between low and high risk. When the task execution risk value is not greater than the preset risk threshold, it indicates that the risk of the robot performing the task according to the predicted robot actions is within an acceptable range. In this case, the autonomous execution mode is selected, and the robot autonomously completes the task control, thereby reducing the proportion of human intervention and improving execution efficiency. When the task execution risk value is greater than the preset risk threshold, it indicates that there may be a high risk in the robot performing the task according to the predicted robot actions. In this case, the human-robot collaborative intervention mode is selected, automatically triggering the human collaborative control mechanism and sending an intervention request to the operator, thereby introducing human experience constraints to reduce risk. By selecting the execution mode based on the comparison result of the task execution risk value and the preset risk threshold, the automatic and dynamic switching between robot autonomous control and human intervention is realized, ensuring both execution efficiency in low-risk situations and task safety in high-risk situations. For example, in a plug-in assembly task, when a collision risk or abnormal contact risk is predicted in the future, the task execution risk value will increase and exceed the preset risk threshold. At this time, the system automatically switches to the human-robot collaborative intervention mode.

[0044] Specifically, in step S4, when the autonomous execution mode is selected, the predicted robot action output by the pre-trained world action model is directly used as the robot action for execution. When the human-machine collaborative intervention mode is selected, human-machine collaborative request information is generated and an intervention request is sent to the operator. The operator inputs manual actions through teleoperation devices, force feedback devices, master-slave control systems, or teaching pendants, and then the predicted robot action is fused with the manual input action to obtain the robot action.

[0045] In one possible implementation, fusing the predicted robot action with the human input action to obtain the robot action includes: obtaining the confidence level of the predicted robot action and the degree of change in the world state, and combining it with the task execution risk value to obtain the fusion weight of the predicted robot action and the human input action; and fusing the predicted robot action with the human input action according to the fusion weight to obtain the robot action.

[0046] Specifically, the confidence level of the predicted robot action is used to characterize the reliability of the pre-trained world action model for the current predicted action, while the degree of change in the world state is used to characterize the deviation of the current world state from the historical world state. By determining the fusion weights by comprehensively considering the task execution risk value, the confidence level of the predicted robot action, and the degree of change in the world state, dynamic fusion between human actions and autonomous robot actions can be achieved. For example, when the task risk is low, autonomous robot actions take the lead; when the task risk increases, the weight of human actions gradually increases, thereby achieving dynamic switching between autonomous robot control and human control.

[0047] In summary, this invention, based on a human-machine collaboration-enhanced robot motion generation method, forms a complete closed loop from multimodal data acquisition, future state prediction, risk assessment to human-machine collaborative motion generation through the coordination of steps S1 to S4. The task description information, current world state information, and robot motion provided in step S1 provide a complete input foundation for the prediction in step S2. The predicted future world state and predicted robot motion generated in step S2 provide a basis for the risk assessment in step S3. The task execution risk value obtained in step S3 provides a quantitative basis for the mode selection in step S4. Step S4 generates robot motion based on the mode selection result, realizing a dynamic switch between autonomous execution and human-machine collaborative intervention. These steps work together to enable the robot to continuously acquire high-value interactive experiences under limited human intervention, and improve task success rate and autonomy while ensuring task execution safety.

[0048] In one possible implementation, before fusing the predicted robot action with the human input action, the method further includes: correcting the human input action based on a preset human input action correction model according to the current world state information, the predicted future world state information, and the human input action; wherein the preset human input action correction model is obtained by modeling the mapping relationship between the human input action and the expert action and combining it with pre-training.

[0049] Specifically, the manual input action correction model is used to automatically correct and optimize manual input actions, gradually bringing them closer to expert actions. This model is based on modeling the mapping relationship between manual input actions and expert actions, combined with pre-training. Interpretably, because the quality of manual input actions highly depends on the operator's experience, different operators may input actions with significant differences, making it difficult to guarantee the consistency and expert level of intervention data. Through the manual input action correction model, the mapping relationship between manual-assisted actions and expert actions can be learned using the current environmental state, future state prediction results, and manual input actions, thereby achieving automatic correction of manual actions, improving the quality of manual intervention and data utilization efficiency.

[0050] In one possible implementation, the robot motion generation method based on human-machine collaboration enhancement further includes: obtaining a risk event identification result based on a preset risk event identification model according to the robot motion and the predicted future world state information; and when the risk event identification result indicates the existence of a risk event, generating a robot risk elimination action to replace the robot motion according to the existing risk event; wherein the risk event includes collision events, assembly failure events, abnormal contact events, and task deviation events; and the robot risk elimination action includes compensation actions, recovery actions, or search actions.

[0051] Specifically, the risk event identification model is used to identify potential future risk events based on robot actions and predicted future world state information. When the risk event identification result indicates the existence of a risk event, a risk mitigation action is generated to replace the robot's actions. For example, in a plug-in assembly task, if the risk event identification result indicates a collision event, a compensation action can be generated to adjust the position of the robot's end effector to avoid a collision; if the risk event identification result indicates an assembly failure event, a recovery action can be generated to remove the part and retry the assembly; if the risk event identification result indicates a task deviation event, a search action can be generated to re-search for the hole position.

[0052] Explained in this embodiment, the risk event identification model identifies potential risk events in advance and generates risk mitigation actions, thereby enabling early intervention and proactive elimination of potential risk events and improving the safety and success rate of the robot when performing complex tasks.

[0053] In one possible implementation, the step of performing reinforcement learning optimization and reward distillation training on the world action model based on historical interaction experience data samples to obtain a pre-trained world action model includes: performing online reinforcement learning optimization on the world action model based on historical interaction experience data samples to obtain a first optimized world action model; performing offline reward distillation training on the world action model based on historical interaction experience data samples to obtain a second optimized world action model; and fusing the model parameters of the first optimized world action model and the second optimized world action model to obtain a pre-trained world action model; wherein, in the offline reward distillation training, high-reward trajectory data and low-reward trajectory data are obtained based on historical interaction experience data samples, and the high-reward trajectory data is used to form positive samples, and the low-reward trajectory data is used to form negative samples; the world action model is trained with the optimization objective of increasing the probability of the world action model outputting the corresponding robot action under positive samples and decreasing the probability of outputting the corresponding robot action under negative samples.

[0054] Specifically, online reinforcement learning optimization refers to optimizing the world action model's policy online using historical interaction experience data samples, enabling the world action model to continuously update its policy parameters during interaction with the environment. Examples of online reinforcement learning optimization algorithms include proximal policy optimization, soft actor commentator, or dual-delay deep deterministic policy gradient. Offline reward distillation training refers to training the world action model offline using historical interaction experience data samples, enabling the world action model to learn action policies corresponding to high-reward trajectories from historical experience. Specifically, the model parameters of the first and second optimized world action models are fused to obtain a pre-trained world action model. Examples of model parameter fusion methods include weighted averaging, parameter interpolation, or model ensemble.

[0055] Specifically, high-reward trajectory data refers to trajectory data with high total action reward information in historical interaction experience data samples, while low-reward trajectory data refers to trajectory data with low total action reward information in historical interaction experience data samples. Interpretably, by using high-reward trajectory data to construct positive samples and low-reward trajectory data to construct negative samples, the world action model can learn high-reward action policies and avoid low-reward action policies, thereby improving the robustness and generalization ability of the policy.

[0056] Explained in this embodiment, through the synergistic optimization of online reinforcement learning optimization and offline reward distillation training, the model parameters obtained from online reinforcement learning optimization and offline reward distillation training are fused together, achieving efficient training of the world action model. This utilizes both the real-time nature of online exploration and the efficiency of offline learning, comprehensively leveraging the advantages of both, thereby improving the performance and training efficiency of the pre-trained world action model.

[0057] In one possible implementation, the action reward information is constructed using a staged dense reward format; wherein, the stages include one or more of the following: target search stage, target approach stage, posture alignment stage, initial contact stage, stable contact stage, operation execution stage, and task completion stage; and the rewards include one or more of the following: distance reward, posture reward, contact reward, progress reward, task completion reward, and safety reward.

[0058] Specifically, the staged dense reward system divides the task execution process into multiple stages, generating corresponding reward information at each stage based on the task execution status. Among these, distance reward represents the distance between the robot's end effector and the task target; attitude reward represents the deviation between the robot's end effector attitude and the target attitude; contact reward represents the contact state between the robot and the task target; progress reward represents the progress of task execution; task completion reward represents whether the task has been completed; and safety reward represents the safety during task execution. For example, the distance reward is calculated based on the Euclidean distance or norm distance between the robot's end effector and the task target; the smaller the distance, the larger the distance reward. The attitude reward is calculated based on the rotation angle deviation or quaternion deviation between the robot's end effector attitude and the target attitude; the smaller the deviation, the larger the attitude reward. The contact reward is calculated based on the degree of matching between the contact force, contact position, or contact state between the robot and the task target and the desired contact state; the higher the degree of matching, the larger the contact reward. The progress reward is calculated based on the progress of the current task stage relative to the previous task stage or the percentage of task completion; the greater the progress, the larger the progress reward. The task completion reward is set to a positive constant when the task is successfully completed and set to zero or a negative value when the task is not completed. The safety reward is calculated based on safety indicators such as whether the contact force is within a safe range, whether a collision occurs, or whether slippage occurs during task execution; the higher the safety, the larger the safety reward.

[0059] Interpretive, by constructing action reward information using a staged dense reward model, it achieves comprehensive coverage of rewards at each stage and of each type during task execution. This provides denser and more accurate reward signals for reinforcement learning optimization of world action models, thereby solving the problem of sparse rewards in real robot reinforcement learning and improving policy learning efficiency and sample utilization efficiency.

[0060] In one possible implementation, when generating future predicted world state information and predicted robot actions based on a pre-trained world action model using task description information, current world state information, and robot actions, the step further includes: acquiring short-term interaction memory information and long-term skill memory information, and using them as auxiliary information, along with task description information, current world state information, and robot actions, to jointly input into the pre-trained world action model to generate future predicted world state information, predicted robot actions, and predicted interaction information; wherein, short-term interaction memory information includes one or more of the following: robot state change information, robot action change information, contact change information, human intervention events, and risk change information within a preset time window; long-term skill memory information includes one or more of the following: task trajectory information generated during historical task execution, successful experience trajectory, failed experience trajectory, task stage transition information, and key interaction event information.

[0061] Specifically, short-term interaction memory information is used to record the robot's interaction process within the most recent time window. The preset time window refers to a pre-defined time range; for example, the preset time window can be set to the most recent 10 seconds, the most recent 30 seconds, or the most recent 1 minute. Robot state change information records the changes in the robot's state within the preset time window; robot action change information records the changes in the robot's actions within the preset time window; contact change information records the changes in the contact state between the robot and the environment within the preset time window; human intervention events record human intervention events that occur within the preset time window; and risk change information records the changes in the task execution risk value within the preset time window.

[0062] Long-term skill memory information is used to record the skill knowledge generated during the execution of historical tasks. Among them, task trajectory information is used to record the task trajectory generated during the execution of historical tasks, successful experience trajectory is used to record the experience trajectory of successfully completing tasks during the execution of historical tasks, failure experience trajectory is used to record the experience trajectory of failing to complete tasks during the execution of historical tasks, task stage transition information is used to record the transition relationship between task stages during the execution of historical tasks, and key interaction event information is used to record key interaction events that occurred during the execution of historical tasks.

[0063] Interpretive, by inputting short-term interaction memory information and long-term skill memory information as auxiliary information into the pre-trained world action model, it realizes the continuous use of historical interaction processes, task stage information and skill knowledge. This enables the pre-trained world action model to make full use of historical interaction processes and skill knowledge when predicting future states and actions, thereby improving long-term reasoning ability and task execution stability in long-term tasks.

[0064] In one possible implementation, the robot motion generation method based on human-machine collaboration reinforcement further includes: collecting current interaction experience data samples based on the current task execution process, and performing reinforcement learning optimization and reward distillation training on the pre-trained world motion model based on the current interaction experience data samples and historical interaction experience data samples, thereby updating the pre-trained world motion model.

[0065] Specifically, the current interaction experience data sample refers to the interaction experience data collected during the current task execution, including world state information, robot actions, and action reward information generated during the current task execution. By combining the current interaction experience data sample with historical interaction experience data samples, reinforcement learning optimization and reward distillation training are performed on the pre-trained world action model. This enables the pre-trained world action model to continuously accumulate new interaction experiences, achieving continuous learning and iterative optimization of robot skills. This allows the robot to continuously accumulate high-value interaction experiences during actual task execution, realizing the ability to transfer from human-assisted execution to autonomous execution, and gradually improving the success rate and generalization ability in complex tasks.

[0066] In summary, this invention presents a robot motion generation method based on human-robot collaboration reinforcement. Compared to traditional methods that rely solely on offline teaching data or pure reinforcement learning exploration, this method constructs a unified closed loop encompassing future prediction, risk assessment, human-robot collaboration, online exploration, reward distillation, and continuous evolution. This enables robots to learn efficiently and continuously optimize in complex real-world environments. It is applicable to various robot application scenarios, including industrial assembly, equipment inspection, tool operation, humanoid robot dual-arm collaborative work, and complex contact operations. It effectively reduces manual teaching costs and improves robot autonomy, robustness, and generalization capabilities, demonstrating promising prospects.

[0067] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0068] See Figure 2 In another embodiment of the present invention, a robot motion generation system based on human-machine collaboration enhancement is provided, which can be used to implement the above-mentioned robot motion generation method based on human-machine collaboration enhancement. Specifically, the robot motion generation system based on human-machine collaboration enhancement includes a data acquisition module, a motion prediction module, a risk assessment module, and a motion optimization module.

[0069] The system comprises the following modules: a data acquisition module for acquiring task description information, current world state information, and robot actions; a motion prediction module for generating predicted future world state information and predicted robot actions based on a pre-trained world motion model, using the task description information, current world state information, and robot actions; a risk assessment module for obtaining a task execution risk value based on the predicted future world state information and predicted robot actions; and a selection of either an autonomous execution mode or a human-machine collaborative intervention mode based on the task execution risk value. The motion optimization module, when selecting the autonomous execution mode, generates robot actions based on the predicted robot actions; and when selecting the human-machine collaborative intervention mode, generates human-machine collaboration request information, acquires human input actions, and fuses the predicted robot actions with the human input actions to obtain the robot actions. The pre-trained world motion model is obtained by acquiring a world motion model and historical interaction experience data samples, and then performing reinforcement learning optimization and reward distillation training on the world motion model based on the historical interaction experience data samples. The historical interaction experience data samples are collected based on historical task execution processes and include world state information, robot actions, and action reward information.

[0070] All relevant content of each step involved in the aforementioned embodiments of the robot motion generation method based on human-machine collaboration enhancement can be referenced to the functional description of the corresponding functional module of the robot motion generation system based on human-machine collaboration enhancement in the embodiments of the present invention, and will not be repeated here.

[0071] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0072] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a robot motion generation method based on human-machine collaboration enhancement.

[0073] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the robot motion generation method based on human-machine collaboration enhancement in the above embodiments.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating robot actions based on human-machine collaboration enhancement, characterized in that, include: Obtain task description information, as well as the current world state and robot actions; Based on the task description information, the current world state information, and the robot's actions, and using a pre-trained world action model, generate future predicted world state information and predicted robot actions. Based on the predicted world state information and the predicted robot actions, the task execution risk value is obtained; Choose between autonomous execution mode or human-machine collaborative intervention mode based on the task execution risk value; When the autonomous execution mode is selected, the robot action is obtained based on the predicted robot action; when the human-robot collaborative intervention mode is selected, human-robot collaborative request information is generated and human input actions are obtained, and the predicted robot action and human input actions are fused to obtain the robot action. The pre-trained world action model is obtained by acquiring the world action model and historical interaction experience data samples, and performing reinforcement learning optimization and reward distillation training on the world action model based on the historical interaction experience data samples to obtain the pre-trained world action model; the historical interaction experience data samples are collected based on the historical task execution process and include world state information, robot actions and action reward information.

2. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The world state information includes environmental observation information, robot body state information, and interaction information; The environmental observation information includes one or more of RGB images, depth images, point cloud information, and semantic information; the robot body state information includes one or more of joint angles, joint velocities, end effector poses, and robot motion history information; the interaction information includes one or more of contact information, force information, torque information, slip information, friction state information, and task stage information.

3. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The step of obtaining the task execution risk value based on the predicted world state information and the predicted robot actions includes: obtaining the risk value of each preset risk situation based on the predicted world state information and the predicted robot actions, and weighting and summing the risk values ​​of each preset risk situation to obtain the task execution risk value; wherein, the preset risk situations include one or more of the following: task failure risk, abnormal contact risk, action deviation risk, collision risk, slippage risk, environmental change risk, and strategy uncertainty risk.

4. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The selection of autonomous execution mode or human-machine collaborative intervention mode based on task execution risk value includes: When the risk value of task execution is not greater than the preset risk threshold, the autonomous execution mode is selected; when the risk value of task execution is greater than the preset risk threshold, the human-machine collaborative intervention mode is selected.

5. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The process of fusing predicted robot actions with human-input actions to obtain robot actions includes: The confidence level and the degree of change in the world state of the predicted robot action are obtained. Combined with the task execution risk value, the fusion weight of the predicted robot action and the human input action is obtained. The predicted robot action and the human input action are then fused according to the fusion weight to obtain the robot action.

6. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, Before fusing the predicted robot action with the human input action, the process further includes: correcting the human input action based on the current world state information, the predicted future world state information, and the human input action, using a preset human input action correction model; wherein the preset human input action correction model is obtained by modeling the mapping relationship between the human input action and the expert action and combining it with pre-training.

7. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, Also includes: Based on the robot's actions and information about the predicted future world state, and using a pre-defined risk event identification model, the risk event identification results are obtained. And when the risk event identification result indicates the existence of a risk event, a robot risk elimination action is generated to replace the robot action based on the existing risk event; wherein, the risk event includes collision event, assembly failure event, abnormal contact event, and task deviation event; the robot risk elimination action includes compensation action, recovery action, or search action.

8. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The process of optimizing the world action model using reinforcement learning and reward distillation training based on historical interaction experience data samples to obtain a pre-trained world action model includes: The world action model is optimized through online reinforcement learning based on historical interaction experience data samples to obtain the first optimized world action model; the world action model is then trained offline with reward distillation based on historical interaction experience data samples to obtain the second optimized world action model; and the model parameters of the first optimized world action model and the second optimized world action model are fused to obtain the pre-trained world action model. In the offline reward distillation training, high-reward trajectory data and low-reward trajectory data are obtained based on historical interaction experience data samples. High-reward trajectory data are used to form positive samples, and low-reward trajectory data are used to form negative samples. The world action model is trained with the optimization objective of increasing the probability of the world action model outputting the corresponding robot action under positive samples and decreasing the probability of outputting the corresponding robot action under negative samples.

9. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, The action reward information is constructed using a staged dense reward format; wherein, the stage includes one or more of the following: target search stage, target approach stage, posture alignment stage, initial contact stage, stable contact stage, operation execution stage, and task completion stage; the reward includes one or more of the following: distance reward, posture reward, contact reward, progress reward, task completion reward, and safety reward.

10. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, When generating predicted world state information and predicted robot actions based on a pre-trained world action model, according to the task description information, the current world state information, and the robot's actions, the process further includes: Short-term interactive memory information and long-term skill memory information are acquired and used as auxiliary information. Together with task description information, current world state information and robot actions, they are input into a pre-trained world action model to generate future predicted world state information, predicted robot actions and predicted interaction information. The short-term interactive memory information includes one or more of the following: robot state change information, robot action change information, contact change information, human intervention events, and risk change information within a preset time window; the long-term skill memory information includes one or more of the following: task trajectory information generated during the execution of historical tasks, successful experience trajectory, failed experience trajectory, task stage transition information, and key interactive event information.

11. The robot motion generation method based on human-machine collaboration enhancement according to claim 1, characterized in that, Also includes: Based on the current task execution process, collect current interaction experience data samples, and based on the current interaction experience data samples and historical interaction experience data samples, perform reinforcement learning optimization and reward distillation training on the pre-trained world action model to update the pre-trained world action model.

12. A robot motion generation system based on human-machine collaboration enhancement, characterized in that, include: The data acquisition module is used to acquire task description information, as well as the current world state information and robot actions. The action prediction module is used to generate future predicted world state information and predicted robot actions based on the task description information, the current world state information and robot actions, and a pre-trained world action model. The risk assessment module is used to obtain the task execution risk value based on the predicted future world state information and the predicted robot actions; Choose between autonomous execution mode or human-machine collaborative intervention mode based on the task execution risk value; The motion optimization module is used to obtain robot motion based on predicted robot motion when the autonomous execution mode is selected; when the human-robot collaborative intervention mode is selected, it generates human-robot collaborative request information and obtains human input motion, and fuses the predicted robot motion with the human input motion to obtain robot motion. The pre-trained world action model is obtained by acquiring the world action model and historical interaction experience data samples, and performing reinforcement learning optimization and reward distillation training on the world action model based on the historical interaction experience data samples to obtain the pre-trained world action model; the historical interaction experience data samples are collected based on the historical task execution process and include world state information, robot actions and action reward information.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the robot motion generation method based on human-machine collaboration enhancement as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot motion generation method based on human-machine collaboration enhancement as described in any one of claims 1 to 11.