Pyramid fusion-based embodied data-driven industrial robot training system
By using an industrial robot training system driven by pyramid fusion embodied data, the problem of discrepancies between human operation data and robot control commands has been solved, thereby improving the reliability and adaptability of robot operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2026-06-04
- Publication Date
- 2026-07-14
Smart Images
Figure CN122378733A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of industrial robots, embodied intelligence, and multimodal data processing technology, specifically an industrial robot training system driven by pyramid fusion embodied data. Background Technology
[0002] Industrial robots are evolving from fixed-program control to intelligent systems with learning capabilities. In tasks such as assembly, gripping, insertion, handling, and inspection, robots not only need to perform repetitive actions but also adapt to changes in workpiece position, operational procedures, and the surrounding environment. Traditional programming and teaching methods suffer from the challenge of directly translating human operational data, visual perception, and verbal experience into robot training actions due to the physical differences between them. This results in low reliability of industrial robot control after training. Summary of the Invention
[0003] The purpose of this invention is to address the problem of low reliability in industrial robot control after training, caused by the difficulty in directly translating worker actions, visual perception, and language experience into robot training actions due to the physical differences between human operational data and robot control commands. This invention provides an industrial robot training system driven by pyramid fusion embodied data.
[0004] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0005] The industrial robot training system based on pyramid fusion embodied data includes a multimodal human demonstration data acquisition module, a data synchronization and standardization module, a semantic-action fusion modeling module, a human-machine cross-shape alignment module, a pyramid fusion data management module, a strategy training module, and a strategy deployment and feedback module.
[0006] The multimodal human demonstration data acquisition module is used to collect raw human demonstration data;
[0007] The data synchronization and standardization module is used to time-align the original human demonstration data and then divide it into multiple behavioral segments.
[0008] The semantic-action fusion modeling module is used to encode the synchronized original human demonstration data and then fuse it to obtain a unified semantic-action representation, and to transform each behavioral fragment into a structured training sample.
[0009] The cross-body alignment module is used to acquire the robot's end effector trajectory, gripper opening degree, and joint trajectory;
[0010] The pyramid fusion data management module is used to construct an embodied data pyramid and input structured training samples, unified semantic-action representations, end-effector trajectories, gripper opening and closing degrees, and joint trajectories into the pyramid to obtain the fused training samples output by the pyramid.
[0011] The policy training module trains a policy model based on the fused training samples and obtains the execution policy output by the policy model.
[0012] The strategy deployment and feedback module deploys and executes the strategy, and inputs the execution results into the pyramid fusion data management module for further training.
[0013] Furthermore, the multimodal human demonstration data acquisition module specifically performs the following steps:
[0014] First, the multimodal human demonstration data acquisition module receives the industrial task configuration and, based on the industrial task configuration, uses multimodal acquisition equipment to collect raw human demonstration data in a real production environment.
[0015] The industrial task configuration includes one or more of the following: assembly, gripping, insertion, handling, sorting, and inspection, and records the task number, process stage, operation object, and target robot type.
[0016] The acquisition device includes one or more of the following: a head-mounted vision device, an eye-tracking unit, a voice acquisition unit, an inertial measurement unit, a motion capture unit, and an edge synchronization controller;
[0017] The original human demonstration data includes first-person perspective video, gaze point trajectory, voice commands, human motion trajectory, head posture, tool or workpiece status, and environmental status.
[0018] Furthermore, the data synchronization and standardization module specifically performs the following steps:
[0019] Step 21: Establish a unified time reference and align the original human demonstration data with the time reference.
[0020] Step 22: Based on the time-aligned data, resample the data at different sampling frequencies, and divide the continuous operation into multiple behavior segments according to the start, contact, release, and completion events of the action;
[0021] Step 23: Calibrate the coordinate relationship between the data acquisition device and the robot base, and convert data from different sources to a unified reference coordinate system;
[0022] Step 24: Generate a standardized record for each behavior segment, the record including the object of operation, action trajectory, gaze target, voice intention, contact state, and environmental state.
[0023] Furthermore, the semantic-action fusion modeling module specifically performs the following steps:
[0024] Step 31: Encode the synchronized raw human demonstration data;
[0025] Step 32: Based on standardized records, construct a behavior graph by combining the operation object, human motion trajectory, tool, workpiece, voice intent, and contact state. The edges in the behavior graph represent temporal relationships, spatial relationships, contact relationships, and semantic relationships.
[0026] Step 33: Based on the behavior graph, fuse the encoded features through graph neural networks, temporal models or cross-modal attention models to obtain a unified semantic-action representation;
[0027] Step 34: Divide each behavior fragment into a structured training sample, which includes task labels, semantic features, action features, source data index, and quality score.
[0028] Furthermore, the human-machine cross-shape alignment module specifically performs the following steps:
[0029] Step 41: Read the structural parameters of the target industrial robot, including the number of robot joints, joint range, end effector type, gripper parameters, and control interface;
[0030] Step 42: Extract the human hand trajectory, tool pose, and relative pose of the manipulated object from the behavior map, and map the human hand trajectory, tool pose, and relative pose of the manipulated object into the robot end effector trajectory, gripper opening and closing degree, and joint trajectory.
[0031] Furthermore, the embodied data pyramid is a three-layered embodied data pyramid. The first layer of the three-layered embodied data pyramid is basic task data, including historical task data, general operation segments, and existing robot training samples. The second layer is human natural demonstration data, including video, voice, gaze, and action data of skilled workers, as well as unified semantic-action representations. The third layer is high-fidelity robot data, including real robot execution data and teleoperation data. The teleoperation data includes end effector trajectory, gripper opening and closing degree, and joint trajectory.
[0032] Furthermore, the strategy training dummy performs the following steps:
[0033] Step 61: Input the fused training samples into the policy model for training to learn the mapping relationship from task state to robot action;
[0034] Step 62: Output the trained strategy as joint trajectory, end-effector pose trajectory, gripper control quantity, velocity control quantity, or skill invocation command.
[0035] Furthermore, the strategy model includes a behavior cloning model, a diffusion strategy model, a visual-language-action model, or a temporal Transformer strategy model.
[0036] Furthermore, the training employs one or more of the following training methods: behavior cloning, action alignment, contrastive learning, and policy distillation.
[0037] Furthermore, the strategy deployment and feedback module specifically performs the following steps:
[0038] After the robot performs a task, it records the task success rate, trajectory deviation, abnormal interruption, human feedback and environmental changes, and feeds the execution results back to the pyramid fusion data management module. Based on the execution feedback, it updates the sample quality score and policy confidence, and triggers incremental training or human review to continuously optimize the robot's strategy.
[0039] The beneficial effects of this invention are:
[0040] This application first transforms raw multimodal data into structured training samples through time synchronization, spatial calibration, and semantic-action fusion modeling, thereby improving the usability of human operational experience. Then, through a human-robot cross-shape alignment mechanism, human actions are mapped to robot end-effector trajectories, gripper movements, or joint control commands. This is then fused and trained using a data pyramid, enabling the strategy to learn general motion patterns while also utilizing worker experience, thus enhancing the reliability of industrial robot control. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the overall framework of this application;
[0042] Figure 2 This is a flowchart illustrating the human-machine cross-shape alignment module.
[0043] Figure 3 This is a demonstration diagram of the model's real-machine reasoning. Detailed Implementation
[0044] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.
[0045] Specific Implementation Method 1: The industrial robot training system based on pyramid fusion embodied data driven by this implementation method includes a multimodal human demonstration data acquisition module, a data synchronization and standardization module, a semantic-action fusion modeling module, a human-machine cross-shape alignment module, a pyramid fusion data management module, a strategy training module, and a strategy deployment and feedback module.
[0046] The multimodal human demonstration data acquisition module is used to collect natural operational data of skilled workers in real industrial production environments. Skilled workers perform tasks according to normal work procedures, and the system records first-person visual flow, gaze points, voice commands, human movement trajectories, tool status, and environmental status, adding task numbers, timestamps, and source identifiers to the data.
[0047] The data synchronization and standardization module addresses issues related to inconsistent data acquisition times, coordinates, and formats across different devices. This module aligns multi-source data to a unified timeline and reference coordinate system, and segments continuous operations into structured behavioral fragments according to action phases.
[0048] The semantic-action fusion modeling module is used to understand human operational intentions and action processes. This module organizes visual targets, language instructions, human actions, tool and workpiece states into behavior graphs and generates unified semantic-action representations for subsequent training.
[0049] The human-robot cross-body alignment module is used to convert human actions into robot-executable actions. Based on the target robot's structure and control interface, this module maps the human hand trajectory, tool posture, and object position to the robot's end effector trajectory, gripper action, or joint trajectory, and ensures the action's executableness through inverse kinematics or trajectory optimization.
[0050] The pyramid fusion data management module manages data from different sources and of varying quality. Basic task data provides general movement patterns, natural human demonstration data provides operational intentions and processes, and high-fidelity robot data provides realistic execution constraints. The system automatically adjusts the usage ratio of different data levels according to the training phase.
[0051] The policy training module is used to train robot policies based on pyramid fusion data. The training process includes learning actions from demonstration samples, aligning human and robot actions, correcting control errors using real robot data, and generating a deployable policy model.
[0052] The strategy deployment and feedback module is used to deploy the trained strategies to real industrial robots. After the robot performs a task, the system records the execution results and anomaly information, and feeds the feedback data back to the platform for subsequent incremental training and strategy optimization.
[0053] S1, Multimodal Human Demonstration Data Acquisition
[0054] S11. The multimodal human demonstration data acquisition module receives industrial task configurations, which include one or more of assembly, gripping, insertion, handling, sorting, and inspection, and records the task number, process stage, operation object, and target robot type.
[0055] S12. Skilled workers wear or use multimodal acquisition devices in a real production environment. The acquisition devices include one or more of the following: head-mounted vision devices, eye-tracking units, voice acquisition units, inertial measurement units, motion capture units, and edge synchronization controllers.
[0056] S13, the multimodal human demonstration data acquisition module synchronously acquires first-person perspective video, gaze trajectory, voice commands, human motion trajectory, head posture, tool or workpiece status, and environmental status to form raw human demonstration data (acquired through S12).
[0057] S2, Data Synchronization and Standardization
[0058] S21. The data synchronization and standardization module establishes a unified time reference and performs time alignment on video, audio, eye tracking, motion capture, and sensor data (data collected in S13).
[0059] S22, the data synchronization and standardization module resamples data at different sampling frequencies. Based on the data collected in S13, and according to the start, contact, release and completion events of the action, the continuous operation is divided into multiple behavior segments.
[0060] S23, the data synchronization and standardization module calibrates the coordinate relationship between the acquisition device and the robot base, and converts data from different sources to a unified reference coordinate system.
[0061] S24. The data synchronization and standardization module generates a standardized record for each behavior segment. The record includes the operation object, action trajectory, gaze target, voice intention, contact state, and environmental state.
[0062] S3, Semantic-Action Fusion Modeling
[0063] The S31 and semantic-action fusion modeling modules encode the data collected in S13.
[0064] S32, the semantic-action fusion modeling module is based on standardized records and constructs a behavior graph by combining the operation object, human action trajectory, tool, workpiece, voice intent, and contact state. The edges in the behavior graph represent temporal relationships, spatial relationships, contact relationships, and semantic relationships.
[0065] The S33 semantic-action fusion modeling module utilizes behavior graphs and fuses the features encoded in S31 through graph neural networks, temporal models, or cross-modal attention models to obtain a unified semantic-action representation.
[0066] S34. The semantic-action fusion modeling module transforms each behavioral fragment into a structured training sample, which includes task labels, semantic features, action features, source data index, and quality score.
[0067] S4, Human-Computer Cross-Shape Alignment
[0068] S41, the human-machine cross-body alignment module reads the structural parameters of the target industrial robot, including the number of robot joints, joint range, end effector type, gripper parameters and control interface.
[0069] S42. The human-robot cross-body alignment module establishes the correspondence between human actions and robot actions. It extracts the human hand trajectory, tool posture, and relative pose of the manipulated object from the behavior map, and maps the human hand trajectory, tool posture, and relative pose of the manipulated object to the robot end effector trajectory, gripper opening and closing degree, and joint trajectory.
[0070] S5, Pyramid Fusion Embodied Data Management
[0071] S51, the pyramid fusion data management module constructs a three-layer embodied data pyramid. The first layer is basic task data, including historical task data, general operation segments, and existing robot training samples; the second layer is human natural demonstration data, including video, voice, gaze, and motion data of skilled workers, as well as unified semantic-action representations; the third layer is high-fidelity robot data, including real robot execution data and teleoperation data (end-effector trajectory, gripper opening and closing degree, and joint trajectory).
[0072] The structured training samples, unified semantic-action representations, end-effector trajectories, gripper opening and closing angles, and joint trajectories are input into the pyramid, and the pyramid outputs the fused training samples.
[0073] S52, the Pyramid Fusion Data Management Module adds data source tags, task tags, quality scores, executability scores, and task similarity scores to data at different levels.
[0074] S53. Basic task data is used to provide general action priors, human natural demonstration data is used to provide operational intentions and task processes, and high-fidelity robot data is used to correct control accuracy and real execution effects.
[0075] S54, the pyramid fusion data management module dynamically adjusts the usage ratio of the three types of data according to the training phase. In the early stage, it focuses on basic task data and human demonstration data, and in the later stage, it focuses on real robot data to improve the executability of the strategy in real scenarios.
[0076] S6, Strategy Training, Deployment, and Feedback
[0077] S61. The policy training module inputs the training samples after pyramid fusion into the policy model. The policy model includes a behavior cloning model, a diffusion policy model, a vision-language-action model, or a temporal Transformer policy model.
[0078] S62. The policy training module uses one or more training methods, such as behavior cloning, action alignment, contrastive learning, and policy distillation, to learn the mapping relationship from task state to robot action.
[0079] S63. The deployment and feedback module outputs the trained strategy as joint trajectory, end-effector pose trajectory, gripper control quantity, speed control quantity, or skill call command according to the target robot control interface.
[0080] S64. After the robot performs the task, the system records the task success rate, trajectory deviation, abnormal interruption, human feedback and environmental changes, and sends the execution results back to the data management module.
[0081] S65. The deployment and feedback module updates the sample quality score and policy confidence based on the execution feedback, and triggers incremental training or manual review to continuously optimize the robot policy. Example
[0082] This embodiment uses the task of plugging and assembling automotive parts as an example to illustrate the implementation process of the present invention.
[0083] First, a head-mounted vision device, eye-tracking unit, voice acquisition unit, IMU, and motion capture device are configured at the assembly station. Technicians perform insertion and assembly operations according to the normal process flow, while the system simultaneously captures video, line-of-sight, voice, arm movements, head posture, and environmental conditions.
[0084] Secondly, the system performs time synchronization and spatial calibration on the collected data, aligning video frames, eye-tracking trajectories, audio clips, and motion trajectories to the same time axis, and unifying the human coordinate system, visual coordinate system, tooling coordinate system, and robot base coordinate system. The system divides the assembly process into behavioral segments such as part picking, positioning, alignment, insertion, confirmation, and release.
[0085] Then, the system performs semantic-action modeling for each behavioral segment. The picking phase focuses on identifying the target workpiece and the grasping action; the alignment phase focuses on identifying the relative posture between the workpiece and the insertion interface; and the insertion phase focuses on recording the motion trajectory, speed changes, and contact state.
[0086] Next, the system reads the parameters of the target robotic arm and gripper, converts the human hand trajectory into the robotic arm end-effector pose, converts the grasping action into gripper opening and closing control, and generates robot execution instructions that meet safety constraints through trajectory optimization.
[0087] Finally, the system inputs basic task data, natural human demonstration data, and a small amount of real robot mating data into the training module to generate a mating assembly strategy. After the strategy is deployed, the robot performs the mating task, and the system records the success rate, trajectory deviation, and abnormal situations, feeding the results back to the platform for subsequent optimization.
[0088] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.
Claims
1. An industrial robot training system based on pyramid fusion embodied data-driven approach, characterized in that... The system includes a multimodal human demonstration data acquisition module, a data synchronization and standardization module, a semantic-action fusion modeling module, a human-machine cross-body alignment module, a pyramid fusion data management module, a strategy training module, and a strategy deployment and feedback module. The multimodal human demonstration data acquisition module is used to collect raw human demonstration data; The data synchronization and standardization module is used to time-align the original human demonstration data and then divide it into multiple behavioral segments. The semantic-action fusion modeling module is used to encode the synchronized original human demonstration data and then fuse it to obtain a unified semantic-action representation, and to transform each behavioral fragment into a structured training sample. The cross-body alignment module is used to acquire the robot's end effector trajectory, gripper opening degree, and joint trajectory; The pyramid fusion data management module is used to construct an embodied data pyramid and input structured training samples, unified semantic-action representations, end-effector trajectories, gripper opening and closing degrees, and joint trajectories into the pyramid to obtain the fused training samples output by the pyramid. The policy training module trains a policy model based on the fused training samples and obtains the execution policy output by the policy model. The strategy deployment and feedback module deploys and executes the strategy, and inputs the execution results into the pyramid fusion data management module for further training.
2. The industrial robot training system based on pyramid fusion embodied data driven according to claim 1, characterized in that... The multimodal human demonstration data acquisition module specifically performs the following steps: First, the multimodal human demonstration data acquisition module receives the industrial task configuration and, based on the industrial task configuration, collects raw human demonstration data in a real production environment using multimodal acquisition equipment. The industrial task configuration includes one or more of the following: assembly, gripping, insertion, handling, sorting, and inspection, and records the task number, process stage, operation object, and target robot type. The acquisition device includes one or more of the following: a head-mounted vision device, an eye-tracking unit, a voice acquisition unit, an inertial measurement unit, a motion capture unit, and an edge synchronization controller; The original human demonstration data includes first-person perspective video, gaze point trajectory, voice commands, human motion trajectory, head posture, tool or workpiece status, and environmental status.
3. The industrial robot training system based on pyramid fusion embodied data driven according to claim 2, characterized in that... The data synchronization and standardization module specifically performs the following steps: Step 21: Establish a unified time reference and align the original human demonstration data with the time reference. Step 22: Based on the time-aligned data, resample the data at different sampling frequencies, and divide the continuous operation into multiple behavior segments according to the start, contact, release, and completion events of the action; Step 23: Calibrate the coordinate relationship between the data acquisition device and the robot base, and convert data from different sources to a unified reference coordinate system; Step 24: Generate a standardized record for each behavior segment, the record including the object of operation, action trajectory, gaze target, voice intention, contact state, and environmental state.
4. The industrial robot training system based on pyramid fusion embodied data driven according to claim 3, characterized in that... The semantic-action fusion modeling module specifically performs the following steps: Step 31: Encode the synchronized raw human demonstration data; Step 32: Based on standardized records, construct a behavior graph by combining the operation object, human motion trajectory, tool, workpiece, voice intent, and contact state. The edges in the behavior graph represent temporal relationships, spatial relationships, contact relationships, and semantic relationships. Step 33: Based on the behavior graph, fuse the encoded features through graph neural networks, temporal models or cross-modal attention models to obtain a unified semantic-action representation; Step 34: Divide each behavior fragment into a structured training sample, which includes task labels, semantic features, action features, source data index, and quality score.
5. The industrial robot training system based on pyramid fusion embodied data driven according to claim 4, characterized in that... The human-machine cross-body alignment module specifically performs the following steps: Step 41: Read the structural parameters of the target industrial robot, including the number of robot joints, joint range, end effector type, gripper parameters, and control interface; Step 42: Extract the human hand trajectory, tool pose, and relative pose of the manipulated object from the behavior map, and map the human hand trajectory, tool pose, and relative pose of the manipulated object into the robot end effector trajectory, gripper opening and closing degree, and joint trajectory.
6. The industrial robot training system based on pyramid fusion embodied data driven according to claim 5, characterized in that... The embodied data pyramid is a three-layered embodied data pyramid. The first layer of the three-layered embodied data pyramid is basic task data, including historical task data, general operation segments, and existing robot training samples. The second layer is human natural demonstration data, including video, voice, gaze, and motion data of skilled workers, as well as unified semantic-action representations. The third layer is high-fidelity robot data, including real robot execution data and teleoperation data. The teleoperation data includes end effector trajectory, gripper opening and closing degree, and joint trajectory.
7. The industrial robot training system based on pyramid fusion embodied data driven according to claim 6, characterized in that... The strategy training dummy body performs the following steps: Step 61: Input the fused training samples into the policy model for training to learn the mapping relationship from task state to robot action; Step 62: Output the trained strategy as joint trajectory, end-effector pose trajectory, gripper control quantity, velocity control quantity, or skill invocation command.
8. The industrial robot training system based on pyramid fusion embodied data driven according to claim 7, characterized in that... The strategy model includes the behavior cloning model, the diffusion strategy model, the visual-language-action model, or the temporal Transformer strategy model.
9. The industrial robot training system based on pyramid fusion embodied data driven according to claim 8, characterized in that... The training employs one or more of the following training methods: behavior cloning, action alignment, contrastive learning, and policy distillation.
10. The industrial robot training system based on pyramid fusion embodied data driven according to claim 9, characterized in that... The strategy deployment and feedback module specifically performs the following steps: After the robot performs a task, it records the task success rate, trajectory deviation, abnormal interruption, human feedback and environmental changes, and feeds the execution results back to the pyramid fusion data management module. Based on the execution feedback, it updates the sample quality score and policy confidence, and triggers incremental training or human review to continuously optimize the robot's strategy.