Robot dynamic operation method and system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, and robot

Abstract

The application discloses a kind of robot dynamic operation method, system and robot based on single static demonstration dynamic enhancement and dynamic adaptive inference, method includes: first, by geometric reconstruction and key frame decomposition, single static demonstration is converted into four categories operation stage;Further, trajectory editing and motion planning are used to generate dynamic enhancement trajectory dataset with stage label in batches;Finally, the diffusion strategy model containing stage classification head is trained, and the reasoning calculation resource and action block length are adaptively adjusted according to the predicted stage when deploying.The application can realize high-quality dynamic operation training only by single static demonstration, significantly reduces the data acquisition cost;At the same time, through the adaptive inference mechanism of stage perception, the re-planning frequency is improved in the dynamic target alignment stage, and the computing overhead is reduced in the non-key stage, effectively solving the data scarcity and real-time bottleneck in robot dynamic operation, and improving the success rate and engineering landing ability of dynamic task.

Description

Technical Field

[0001] This invention belongs to the technical field of robot operation, specifically relating to a robot dynamic operation method, system, and robot based on single static demonstration dynamic enhancement and dynamic adaptive reasoning. Background Technology

[0002] With the rapid development of artificial intelligence, 3D vision perception, and robot learning technologies, robots are moving from fixed industrial scenarios to real-world, dynamic environments such as home services, flexible manufacturing, logistics sorting, and open scientific research experiments. In these scenarios, robots not only need to perform routine operations such as grasping, placing, striking, and blocking stationary objects, but also need to respond quickly to continuously moving targets. For example, robots may need to grasp moving objects on a conveyor belt, intercept a ball about to enter a goal, catch a bottle about to fall, place objects on a moving platform, or use tools to strike a moving target. These tasks require robots to continuously perceive moving targets within a closed-loop perception-action cycle, predict the target's state, and generate precise actions at the appropriate time.

[0003] Existing robot operation learning methods mainly rely on large-scale expert demonstration data, including imitation learning, diffusion strategies, vision-language-action models, and reinforcement learning control methods. These methods perform well in static desktop operations and quasi-static tasks, but dynamic operation tasks have higher data requirements and real-time requirements. On the one hand, dynamic tasks involve multiple combinations of factors such as target position, velocity, trajectory, response timing, and contact window, and manual teleoperation or kinematic teaching cannot cover enough dynamic changes. On the other hand, dynamic targets require policies with low-latency reasoning capabilities; if the policy generates actions too slowly, the robot will experience tracking lag and miss interaction opportunities.

[0004] To reduce data acquisition costs, existing research has attempted to expand robot demonstration data using methods such as 2D image enhancement, 3D reconstruction, simulation generation, state space transformation, and trajectory editing. However, 2D image enhancement cannot guarantee 3D geometric consistency; implicit 3D reconstruction methods typically only improve the visual appearance without adjusting the motion trajectory; traditional geometric enhancement is mostly limited to overall rigid body transformation, making it difficult to generate dynamic temporal sequences of moving targets; simulation data generation may have discrepancies between simulation and reality; and existing trajectory editing methods typically generate quasi-static changes, which cannot effectively cover complex operation stages such as dynamic target alignment, contact interaction, and safe transfer.

[0005] Furthermore, while generative motion models such as diffusion strategies can generate high-quality actions, their inference process requires multiple steps of denoising and sampling, resulting in significant computational overhead. Existing strategies typically employ a fixed number of denoising steps and a fixed action block execution length, failing to dynamically allocate inference computational resources based on task stages. During dynamic target tracking, robots require short action blocks and frequent replanning; during contact interaction, robots need continuous action and high precision; and during reset or transport stages, robots can utilize lower computational budgets. The lack of a stage-adaptive inference mechanism leads to insufficient response in critical stages or computational redundancy in non-critical stages.

[0006] In summary, existing robot dynamic manipulation technologies still suffer from problems such as high cost of dynamic demonstration data acquisition, low efficiency of single demonstration utilization, insufficient three-dimensional consistency of augmented data, lack of physical constraints in dynamic trajectory generation, high latency in strategy reasoning, and unreasonable allocation of computing resources at different task stages. There is an urgent need for a robot dynamic manipulation technology solution that can generate high-quality dynamic training data from a single static demonstration and adaptively adjust the reasoning frequency according to the dynamic interaction stage during deployment. Summary of the Invention

[0007] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a robot dynamic operation method, system, and robot based on dynamic enhancement and dynamic adaptive reasoning of a single static demonstration. Through geometric reconstruction and four-stage trajectory editing, a single static demonstration is automatically expanded into a large amount of physically consistent dynamic operation training data. At the same time, the dynamic stage perception strategy is trained by using the stage labels naturally generated during the enhancement process, and the diffusion inference calculation and action block execution steps are adaptively adjusted according to the predicted stage during deployment, thereby achieving low-cost and high-response-speed robot dynamic operation.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for dynamic robot operation based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning, comprising the following steps:

[0010] S1. Collect a single static robot operation demonstration to obtain demonstration data including scene point cloud, end effector pose, and robot body state;

[0011] S2, Execute single... on the target object Figure 3 3D mesh reconstruction, coarse registration with directed bounding boxes, and fine registration with multi-resolution ICP are used to generate normalized object point cloud assets; robot point cloud assets are rendered based on robot URDF model and forward kinematics to ensure geometric consistency.

[0012] S3. Decompose the static demonstration data into multiple keyframes, and define four types of operation stages based on the keyframes, including static object acquisition stage, dynamic target alignment stage, contact interaction execution stage, and pose condition transfer stage.

[0013] S4. Sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer.

[0014] S5. A dynamic perception strategy that integrates diffusion action generation and stage classification is trained based on augmented datasets;

[0015] S6. During deployment, based on the real-time prediction stage, the diffusion denoising step number, Taylor acceleration ratio, and action block execution length are adaptively adjusted to achieve low-latency closed-loop control.

[0016] As a preferred technical solution, in step S1, the demonstration data also includes one or more of the following: RGB-D image, depth map, camera intrinsic and extrinsic parameters, gripper status, voice commands, or image of the task object.

[0017] As a preferred technical solution, in step S2, ICP fine registration is accomplished by minimizing the following formula:

[0018]

[0019] in, For the corresponding point index, The number of corresponding points participating in the registration. For the observation point cloud, the first One point, For the first vertex or sampling point in a 3D mesh One corresponding point, For the spatial transformation to be estimated, This represents the Euclidean norm.

[0020] As a preferred technical solution, in step S3, the key frame includes an operation start frame, an object contact frame, a contact state frame, a target interaction frame, a target transfer frame, and a reset frame.

[0021] As a preferred technical solution, in step S3, the static object acquisition stage is used to acquire static tools or objects;

[0022] The dynamic target alignment stage is used to generate the robot's tracking and alignment trajectory for moving targets;

[0023] The contact interaction execution phase is used to reuse the high-precision local contact actions in the source demonstration.

[0024] The pose condition transfer phase is used for safe transfer, transport, or reset between different operation phases.

[0025] As a preferred technical solution, step S4 further includes the following steps:

[0026] The static object acquisition stage achieves spatial generalization by applying a rigid body transformation to the source stage trajectory:

[0027]

[0028] The dynamic target alignment stage extracts the target-relative pose of the end effector relative to the target object from the source demonstration terminal keyframes. Given the synthetic target object's trajectory Contact offset with the task The desired end effector pose is:

[0029]

[0030] The motion planner then generates a smooth tracking trajectory under velocity constraints, reachability space constraints, and joint constraints, and is able to perform high-frequency replanning to cope with rapid target changes;

[0031] During the contact interaction execution phase, the source contact segment is represented relative to the source anchoring coordinate system. The local trajectory is transformed to the target anchoring coordinate system in the enhanced trajectory. :

[0032]

[0033] This allows for the reuse of precise contact behavior from the source demonstration under different spatial positions and orientations;

[0034] During the pose condition transfer stage, the source key pose is... By applying spatial offset, the enhanced target pose is obtained:

[0035]

[0036] The motion planner plans from the current state to the target pose, completing safe transportation, attitude switching, reset or obstacle avoidance;

[0037] Where k is the enhanced trajectory number and t is the time step; the left side of equation (4) represents the transformed scene point cloud and end effector pose, respectively, and the static stage rigid body transformation on the right side represents the spatial generalization transformation obtained by sampling the k-th enhanced trajectory; the left side of equation (5) represents the desired end effector pose, and the right side is composed of the k-th target object motion trajectory, task contact offset and source demonstration target relative pose in sequence; the source anchoring coordinate system and target anchoring coordinate system in equation (6) are used to complete the coordinate transformation of the contact segment; the spatial offset in equation (7) is used to obtain the enhanced target pose of the k-th enhanced trajectory.

[0038] As a preferred technical solution, the dynamic sensing strategy in step S5 is as follows:

[0039] Using point cloud and robot body state as input, the motion blocks are generated by conditional diffusion denoising network after being fused by encoder;

[0040] The motion generation was trained using diffusion denoising loss.

[0041] The stage classification head is trained using stage labels automatically generated during the augmentation process. The total loss is a weighted sum of the action generation loss and the stage classification loss, enabling the policy to output both action and stage predictions simultaneously.

[0042] As a preferred technical solution, in step S6, the number of diffusion denoising steps, the Taylor acceleration ratio, and the action block execution length are adaptively adjusted as follows:

[0043] Taylor expansion is used to approximate the DDIM denoising step, the complete forward computation residual is cached, and the Taylor coefficients are estimated using finite difference to skip part of the network computation.

[0044] The dynamic target alignment phase employs short action blocks to increase the replanning frequency;

[0045] Long action blocks are used to maintain continuity during the contact interaction phase;

[0046] The pose condition transfer phase is skipped using aggressive computation to reduce overhead;

[0047] Conservative scheduling is used in the static object acquisition phase to avoid early failures.

[0048] Secondly, the present invention provides a robot dynamic operating system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, which is applied to the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, including a demonstration acquisition module, a geometric reconstruction module, a keyframe decomposition module, a dynamic trajectory editing module, a data generation module, a dynamic perception strategy training module, and a stage adaptive reasoning module.

[0049] The demonstration acquisition module is used to acquire demonstration data including scene point cloud, end effector pose, and robot body state for a single static robot operation demonstration.

[0050] The geometric reconstruction module is used to perform single-object reconstruction on the target object. Figure 3 3D mesh reconstruction, coarse registration with directed bounding boxes, and fine registration with multi-resolution ICP are used to generate normalized object point cloud assets; robot point cloud assets are rendered based on robot URDF model and forward kinematics to ensure geometric consistency.

[0051] The keyframe decomposition module is used to decompose the static demonstration data into multiple keyframes and define four types of operation stages based on the keyframes, including the static object acquisition stage, the dynamic target alignment stage, the contact interaction execution stage, and the pose condition transfer stage.

[0052] The dynamic trajectory editing module is used to sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer.

[0053] The dynamic perception strategy training module is used to train a dynamic perception strategy that integrates diffusion action generation and stage classification based on an augmented dataset.

[0054] The stage-adaptive inference module is used to adaptively adjust the diffusion denoising steps, Taylor acceleration ratio, and action block execution length according to the real-time prediction stage during deployment, thereby achieving low-latency closed-loop control.

[0055] Thirdly, the present invention provides a robot, the robot comprising:

[0056] At least one processor; and,

[0057] A memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] 1. This invention can significantly reduce the cost of dynamic operation data acquisition. In the experiment, it takes about 100 minutes to manually acquire 50 dynamic demonstrations and about 400 minutes to acquire 200 dynamic demonstrations; this invention only needs to acquire one static demonstration every 30 seconds, and then automatically enhances 50 trajectories in about 7 minutes and enhances 200 trajectories in about 30 minutes, which significantly reduces the time and manpower required for manual remote operation.

[0061] 2. This invention can improve the data utilization efficiency of a single demonstration. In simulation tasks, when training directly with one dynamic demonstration, the success rates for dynamic tapping, conveyor belt grasping, object capture, whack-a-mole, and target blocking are 27%, 0%, 8%, 3%, and 12%, respectively; when training with 50 dynamic demonstrations, the success rates for the above tasks are 55%, 85%, 50%, 83%, and 87%, respectively; after enhancing one static demonstration to 200 trajectories, the success rates for the above tasks reach 52%, 82%, 68%, 80%, and 87%, respectively. This demonstrates that in simulation, a low-cost static demonstration can be fully utilized to obtain a batch of high-quality dynamic demonstration datasets.

[0062] 3. This invention can improve the performance of real-world dynamic operation deployment. In real-world tasks, when training directly with one dynamic demo, the success rates for dynamic tapping, conveyor belt grabbing, shipboard placement, and bottle capture are 6.67%, 30%, 0%, and 0%, respectively; when training with 50 dynamic demos, the success rates are 68.9%, 70.3%, 3.33%, and 73.3%, respectively; after enhancing one static demo to 200 trajectories, the success rates increase to 87.8%, 73.3%, 66.7%, and 83.3%, respectively. This demonstrates that in real-world deployment, a high-quality dynamic demo dataset can be obtained in batches using only one low-cost static demo.

[0063] 4. This invention can maintain the 3D geometric consistency of the enhanced data. The average success rate is 76.7% when using complete object mesh reconstruction and robot reconstruction; the success rate drops to 10.0% after removing the object mesh; and the success rate drops to 6.67% after removing both the object and robot reconstruction, indicating that geometric reconstruction effectively avoids the geometric distortion caused by direct editing of single-view point clouds.

[0064] 5. This invention can reduce dynamic policy inference latency and improve closed-loop response. A single standard DDIM denoising step complete forward computation takes approximately 46ms, while a Taylor approximation step takes approximately 22ms, resulting in a latency reduction of approximately 52.2% per approximation step. In various dynamic tasks, this invention reduces inference time compared to the DP3 baseline; for example, conveyor belt grasping is reduced from 45.0ms to 32.9ms, object capture from 61.4ms to 36.0ms, whack-a-mole from 44.4ms to 30.6ms, and target blocking from 44.6ms to 33.9ms.

[0065] 6. This invention can improve the staged generalization ability of complex dynamic tasks. Through four types of stage operators, this invention can flexibly instantiate tasks such as dynamic tapping, conveyor belt grasping, object capture, whack-a-mole, target blocking, shipboard placement, and bottle capture, and supports different combinations of looping, skipping, rearrangement, and boundary reachability enhancement, thereby improving the scalability and engineering application value of the method. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of a robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning, according to an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the dynamic sensing strategy according to an embodiment of the present invention;

[0069] Figure 3 This is a block diagram of a robot dynamic operating system based on a single static demonstration, dynamic enhancement, and dynamic adaptive reasoning, according to an embodiment of the present invention.

[0070] Figure 4 This is a structural diagram of the robot according to an embodiment of the present invention. Detailed Implementation

[0071] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0072] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0073] like Figure 1As shown, this example provides a robot dynamic manipulation method based on single static demonstration dynamic enhancement and dynamic adaptive inference. Through geometric reconstruction and four-stage trajectory editing, a single static demonstration is automatically expanded into a large amount of physically consistent dynamic manipulation training data. Simultaneously, the method utilizes naturally generated stage labels during enhancement to train a dynamic stage perception strategy, and adaptively adjusts the diffusion inference calculation and action block execution steps based on the predicted stage during deployment, thereby achieving low-cost, high-response-speed robot dynamic manipulation. The method specifically includes the following steps:

[0074] S1. Collect a single static robot operation demonstration to obtain demonstration data including scene point cloud, end effector pose, and robot body state.

[0075] Specifically, a static robot operation demonstration is collected, which records the scene point cloud, end effector pose, and robot body state at each moment, and can be represented as:

[0076]

[0077] in, For scene point clouds, For the end effector pose, This refers to the robot's physical state.

[0078] Furthermore, depending on the platform configuration, the demonstration data may also include RGB-D images, depth maps, camera intrinsic and extrinsic parameters, gripper status, voice commands, and images of the task object.

[0079] S2, Execute single... on the target object Figure 3 3D mesh reconstruction, coarse registration with directed bounding boxes, and fine registration with multi-resolution ICP are used to generate normalized object point cloud assets; robot point cloud assets are rendered based on robot URDF model and forward kinematics to ensure geometric consistency.

[0080] Specifically, to avoid geometric loss caused by direct editing of point clouds from a single viewpoint, this embodiment performs geometric reconstruction on the target object and the robot separately. The geometric reconstruction specifically includes:

[0081] For the target object, this invention generates a 3D mesh model from a single RGB image and registers this model with the observed point cloud of the target object. The registration process includes coarse initialization based on directed bounding boxes and fine registration using multi-resolution ICP; the fine ICP registration is accomplished by minimizing the following formula:

[0082]

[0083] in, For the corresponding point index, The number of corresponding points participating in the registration. For the observation point cloud, the first One point, For the first vertex or sampling point in a 3D mesh One corresponding point, For the spatial transformation to be estimated, This represents the Euclidean norm. After registration, the 3D mesh is resampled into a normalized object point cloud asset, allowing for arbitrary spatial transformations in subsequent augmentation.

[0084] For robots, this embodiment reads the robot's URDF model and joint states, and renders the robot's point cloud using forward kinematics:

[0085]

[0086] In Equation (3), the left-hand side represents the robot point cloud asset at time t, RenderRobot(·) is a function for forward kinematic rendering based on the URDF model and joint states, URDF is the Unified Robot Description Format Model, and the robot body state in Equation (3) is the joint or body observation at time t. This method avoids using noisy and incomplete sensor robot point clouds, ensuring that the robot body geometry and joint states are consistent in the enhanced trajectory.

[0087] S3. Decompose the static demonstration data into multiple keyframes, and define four types of operation stages based on the keyframes, including static object acquisition stage, dynamic target alignment stage, contact interaction execution stage, and pose condition transfer stage.

[0088] Specifically, a single static demonstration contains key local priors regarding robot-object interaction. This invention extracts keyframes through a small amount of manual annotation or automatic event recognition, such as the operation start frame, object contact frame, post-contact state frame, target interaction frame, target transfer frame, and reset frame. Based on these keyframes, the source demonstration is decomposed into the following four stages:

[0089] The static object acquisition phase, Phi_static, is used to acquire static tools or objects.

[0090] The dynamic target alignment stage Phi_dyn is used to generate the robot's tracking and alignment trajectory for moving targets;

[0091] Phi_inter, the contact interaction execution phase: used to reuse high-precision local contact actions from the source demonstration;

[0092] Phi_trans: Used for safe transfer, transport, or reset between different operation stages.

[0093] S4. Sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer.

[0094] Specifically, for the first Enhanced trajectories: This invention samples enhancement parameters from task-related distributions. It controls the initial pose, speed, direction, trajectory, contact anchor point, target transfer, and number of cycles of the target object.

[0095] The static object acquisition stage achieves spatial generalization by applying a rigid body transformation to the source stage trajectory:

[0096]

[0097] In Equation (4), the left side represents the scene point cloud and end effector pose of the k-th augmented trajectory at time t, and the static stage rigid body transformation on the right side is the spatial generalization transformation obtained by sampling the k-th augmented trajectory, where k is the augmented trajectory number and t is the time step.

[0098] The dynamic target alignment stage extracts the target-relative pose of the end effector relative to the target object from the source demonstration terminal keyframes. Given a combination The trajectory of the target object e Contact offset with the task The desired end effector pose is:

[0099]

[0100] In equation (5), the left side represents the desired end effector pose of the k-th enhanced trajectory at time t, and the right side consists of the synthesized target object motion trajectory, the task contact offset, and the target relative pose in the keyframe of the source demonstration terminal. The motion planner then generates a smooth tracking trajectory under velocity constraints, reachability space constraints, and joint constraints, and can perform high-frequency replanning to cope with rapid target changes. When the error between the current end effector pose and the desired end effector pose is less than a preset alignment threshold, the contact interaction execution stage is entered; otherwise, replanning continues.

[0101] During the contact interaction execution phase, the source contact segment is represented relative to the source anchoring coordinate system. The local trajectory is transformed to the target anchoring coordinate system in the enhanced trajectory. :

[0102]

[0103] In equation (6), A is the source anchoring coordinate system. For the first The target anchoring coordinate system of the enhanced trajectory is used. The inverse transformation of A is used to first transform the source contact segment to the local anchoring coordinate system, and then map it to the enhanced trajectory from the target anchoring coordinate system.

[0104] This allows for the reuse of precise contact behaviors from the source demonstration, such as tapping, gripping, blocking, and releasing, in different spatial locations and orientations.

[0105] During the pose condition transfer stage, the source key pose is... By applying spatial offset, the enhanced target pose is obtained:

[0106]

[0107] Wherein, the left side of equation (7) is the first The enhanced target pose of the enhanced trajectory is derived from the source key pose, which is the stage target pose annotated or identified in the static demonstration. The spatial offset is the target transformation sampled during the pose condition transition stage. The motion planner plans from the current state to the target pose, completing safe transportation, attitude switching, reset, or obstacle avoidance.

[0108] Furthermore, this invention can flexibly combine four types of stage operators according to task interaction logic, with typical combinations including:

[0109] Dynamic tapping:

[0110]

[0111] Object capture:

[0112]

[0113] Conveyor belt grabbing:

[0114]

[0115] In this equation, the left side of equations (8), (9) and (10) represents the phase combination sequence of dynamic tapping, object capture and conveyor belt grasping tasks, respectively. The four types of phase operators on the right side correspond to the static object acquisition phase, dynamic target alignment phase, contact interaction execution phase and pose condition transfer phase, respectively. L is the number of loops, the arrow indicates the phase execution order, and the subscripts 1 and 2 indicate instances of the same phase in different sub-steps.

[0116] Real-world shipboard placement tasks can also introduce an accessibility-enhanced dynamic target alignment stage. When a moving target is out of the reach of the robotic arm, the target is projected to the nearest reachable boundary point, allowing the robot to continuously track within a safe range of motion.

[0117] Furthermore, after generating the enhanced trajectory, this invention merges the object point cloud, robot point cloud, and environment point cloud, uses nearest neighbor search to delete redundant or outlier points, and performs sampling of a fixed number of points. Preferably, each frame samples 2048 points, saving 3D coordinates and color information. Each enhanced trajectory is saved in HDF5 format, including point cloud observations, end-effector pose, joint motion, gripper state, stage boundaries, stage labels, enhancement parameters, and transformation matrix, etc.

[0118] S5. A dynamic perception strategy that integrates diffusion action generation and stage classification is trained based on an augmented dataset.

[0119] Specifically, this invention is based on training a diffusion-based action generation strategy using augmented data, such as... Figure 2 As shown, the policy input includes single-view point cloud and robot body state. The point cloud is encoded by a point cloud encoder such as PointNet, and the robot state is encoded by a lightweight multilayer perceptron. After the two types of features are fused, a conditional diffusion denoising network is used to generate action blocks.

[0120] Motion generation training uses diffusion denoising loss:

[0121]

[0122] In equation (11), the left-hand side represents the diffusion denoising loss, ε represents the Gaussian noise added to the action, and the noise prediction network with parameter θ is used to predict this noise. The action variable in the equation is the noisy action in the t-th diffusion step. This represents the expectation for the training samples and the noise.

[0123] Meanwhile, this invention utilizes stage labels automatically generated by the enhanced process to train a lightweight stage classification head:

[0124]

[0125]

[0126] In Equation (12), the left side is the stage probability distribution output by the stage classification head at time t, which is different from the scene point cloud in Equation (1); the right side of Equation (12) indicates that the classification head outputs the distribution based on the fused observation features; the stage label in Equation (13) is the stage label at time t, and T is the trajectory length.

[0127] The total loss is:

[0128]

[0129] in, For total training losses, The stage classification loss weights are used. Thus, the policy not only learns action generation, but also learns to identify whether the current operation is in the stage of static object acquisition, dynamic target alignment, contact interaction execution, or pose condition transition.

[0130] S6. During deployment, based on the real-time prediction stage, the diffusion denoising step number, Taylor acceleration ratio, and action block execution length are adaptively adjusted to achieve low-latency closed-loop control.

[0131] Specifically, during deployment, the stage classification head outputs the current prediction stage, and the system adjusts the diffusion inference computation and action block execution steps accordingly. This invention can employ an inference acceleration mechanism based on Taylor expansion. The intermediate features of adjacent DDIM denoising steps exhibit smooth variation characteristics, thus allowing the residual block output of complete forward computation steps to be cached, and Taylor coefficients estimated using finite difference estimation.

[0132]

[0133] For the denoising step that skips the full computation, intermediate features are predicted using the following formula:

[0134]

[0135] In equation (15), the intermediate feature represents the first... The DDIM denoising step outputs intermediate features or residual blocks of the network, with Taylor coefficients representing finite difference coefficients of the corresponding order, K being the maximum unfolding order, and δ being the step size offset for skipping the complete forward computation. express factorial.

[0136] Different scheduling strategies are used in different stages. The dynamic target alignment stage uses short action blocks to increase the frequency of replanning; the contact interaction execution stage uses longer action blocks to maintain action continuity; the pose condition transfer stage uses more aggressive computation skipping to reduce the computational overhead of non-critical stages; and the static object acquisition stage uses more conservative scheduling to avoid early grasping failures.

[0137] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0138] Based on the same ideas as the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning in the above embodiments, the present invention also provides a robot dynamic operating system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning. This system can be used to execute the above-described robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning. For ease of explanation, the structural schematic diagram of the robot dynamic operating system embodiment based on single static demonstration dynamic enhancement and dynamic adaptive reasoning only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0139] Please see Figure 3 In another embodiment of this application, a robot dynamic operating system 100 based on single static demonstration dynamic enhancement and dynamic adaptive reasoning is provided. The system includes a demonstration acquisition module 101, a geometric reconstruction module 102, a keyframe decomposition module 103, a dynamic trajectory editing module 104, a dynamic perception strategy training module 105, and a stage adaptive reasoning module 106.

[0140] The demonstration acquisition module 101 is used to acquire a single static robot operation demonstration and obtain demonstration data including scene point cloud, end effector pose, and robot body state.

[0141] The geometric reconstruction module 102 is used to perform single-object reconstruction on the target object. Figure 3 3D mesh reconstruction, coarse registration with directed bounding boxes, and fine registration with multi-resolution ICP are used to generate normalized object point cloud assets; robot point cloud assets are rendered based on robot URDF model and forward kinematics to ensure geometric consistency.

[0142] The keyframe decomposition module 103 is used to decompose the static demonstration data into multiple keyframes and define four types of operation stages based on the keyframes, including the static object acquisition stage, the dynamic target alignment stage, the contact interaction execution stage, and the pose condition transfer stage.

[0143] The dynamic trajectory editing module 104 is used to sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer.

[0144] The dynamic perception strategy training module 105 is used to train a dynamic perception strategy that integrates diffusion action generation and stage classification based on an augmented dataset.

[0145] The stage adaptive inference module 106 is used to adaptively adjust the diffusion denoising steps, Taylor acceleration ratio and action block execution length according to the real-time prediction stage during deployment, so as to achieve low-latency closed-loop control.

[0146] It should be noted that the robot dynamic operating system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning of the present invention corresponds one-to-one with the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning of the present invention. The technical features and beneficial effects described in the embodiments of the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning are applicable to the embodiments of robot dynamic operation based on single static demonstration dynamic enhancement and dynamic adaptive reasoning. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0147] Furthermore, in the above embodiments of the robot dynamic operating system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the robot dynamic operating system based on single static demonstration dynamic enhancement and dynamic adaptive reasoning is divided into different program modules to complete all or part of the functions described above.

[0148] Please see Figure 3 In one embodiment, a robot is provided that implements a robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning. The robot 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a robot dynamic operation program 203 based on single static demonstration dynamic enhancement and dynamic adaptive reasoning.

[0149] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the robot 200, such as the robot 200's portable hard drive. In other embodiments, the first memory 202 can also be an external storage device of the robot 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the robot 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the robot 200. The first memory 202 can be used not only to store application software and various types of data installed on the robot 200, such as the code of the robot dynamic operation program 203 based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, but also to temporarily store data that has been output or will be output.

[0150] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the robot, connecting various components of the robot through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the robot 200 and process data.

[0151] Figure 4 Only a robot with parts is shown; those skilled in the art will understand that... Figure 4 The structure shown does not constitute a limitation on the robot 200, and may include fewer or more parts than shown, or combine certain parts, or have different arrangements of parts.

[0152] The robot dynamic operation program 203 stored in the first memory 202 of the robot 200, based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0153] Collect a single static robot operation demonstration to obtain demonstration data including scene point cloud, end effector pose, and robot body state;

[0154] Execute single on target object Figure 3 3D mesh reconstruction, coarse registration with directed bounding boxes, and fine registration with multi-resolution ICP are used to generate normalized object point cloud assets; robot point cloud assets are rendered based on robot URDF model and forward kinematics to ensure geometric consistency.

[0155] The static demonstration data is decomposed into multiple keyframes, and four types of operation stages are defined based on the keyframes, including the static object acquisition stage, the dynamic target alignment stage, the contact interaction execution stage, and the pose condition transfer stage.

[0156] By sampling dynamic enhancement parameters, multiple physically executable dynamic operation trajectories and stage labels are generated through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer.

[0157] A dynamic perception strategy that integrates diffusion action generation and stage classification is trained based on augmented datasets;

[0158] During deployment, the diffusion denoising steps, Taylor acceleration ratio, and action block execution length are adaptively adjusted based on the real-time prediction stage to achieve low-latency closed-loop control.

[0159] Furthermore, if the modules / units integrated into the robot 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning, characterized in that: Includes the following steps: S1. Collect a single static robot operation demonstration to obtain demonstration data including scene point cloud, end effector pose, and robot body state; S2. Perform single-image 3D mesh reconstruction, directed bounding box coarse registration and multi-resolution ICP fine registration on the target object to generate normalized object point cloud assets; Robot point cloud assets are rendered using the robot's URDF model and forward kinematics to ensure geometric consistency; S3. Decompose the static demonstration data into multiple keyframes, and define four types of operation stages based on the keyframes, including static object acquisition stage, dynamic target alignment stage, contact interaction execution stage, and pose condition transfer stage. S4. Sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer. S5. A dynamic perception strategy that integrates diffusion action generation and stage classification is trained based on augmented datasets; S6. During deployment, based on the real-time prediction stage, the diffusion denoising step number, Taylor acceleration ratio, and action block execution length are adaptively adjusted to achieve low-latency closed-loop control.

2. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S1, the demonstration data also includes one or more of the following: RGB-D image, depth map, camera intrinsic and extrinsic parameters, gripper status, voice commands, or image of the task object.

3. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S2, ICP fine registration is accomplished by minimizing the following formula: in, For the corresponding point index, The number of corresponding points participating in the registration. For the observation point cloud, the first One point, For the first vertex or sampling point in a 3D mesh One corresponding point, For the spatial transformation to be estimated, This represents the Euclidean norm.

4. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S3, the keyframes include the operation start frame, the object contact frame, the contact state frame, the target interaction frame, the target transfer frame, and the reset frame.

5. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S3, the static object acquisition stage is used to acquire static tools or objects; The dynamic target alignment stage is used to generate the robot's tracking and alignment trajectory for moving targets; The contact interaction execution phase is used to reuse the high-precision local contact actions in the source demonstration. The pose condition transfer phase is used for safe transfer, transport, or reset between different operation phases.

6. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, Step S4 also includes the following steps: The static object acquisition stage achieves spatial generalization by applying a rigid body transformation to the source stage trajectory: The dynamic target alignment stage extracts the target-relative pose of the end effector relative to the target object from the source demonstration terminal keyframes. Given the synthetic target object's trajectory Contact offset with the task The desired end effector pose is: The motion planner then generates a smooth tracking trajectory under velocity constraints, reachability space constraints, and joint constraints, and is able to perform high-frequency replanning to cope with rapid target changes; During the contact interaction execution phase, the source contact segment is represented relative to the source anchoring coordinate system. The local trajectory is transformed to the target anchoring coordinate system in the enhanced trajectory. : This allows for the reuse of precise contact behavior from the source demonstration under different spatial positions and orientations; During the pose condition transfer stage, the source key pose is... By applying spatial offset, the enhanced target pose is obtained: The motion planner plans from the current state to the target pose, completing safe transportation, attitude switching, reset or obstacle avoidance; Where k is the enhanced trajectory number and t is the time step; the left side of equation (4) represents the transformed scene point cloud and end effector pose, respectively, and the static stage rigid body transformation on the right side represents the spatial generalization transformation obtained by sampling the k-th enhanced trajectory; the left side of equation (5) represents the desired end effector pose, and the right side is composed of the k-th target object motion trajectory, task contact offset and source demonstration target relative pose in sequence; the source anchoring coordinate system and target anchoring coordinate system in equation (6) are used to complete the coordinate transformation of the contact segment; the spatial offset in equation (7) is used to obtain the enhanced target pose of the k-th enhanced trajectory.

7. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S5, the dynamic perception strategy is specifically as follows: Using point cloud and robot body state as input, the motion blocks are generated by conditional diffusion denoising network after being fused by encoder; The motion generation was trained using diffusion denoising loss. The stage classification head is trained using stage labels automatically generated during the augmentation process. The total loss is a weighted sum of the action generation loss and the stage classification loss, enabling the policy to output both action and stage predictions simultaneously.

8. The robot dynamic operation method based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning according to claim 1, characterized in that, In step S6, the number of diffusion denoising steps, the Taylor acceleration ratio, and the action block execution length are adaptively adjusted, specifically as follows: Taylor expansion is used to approximate the DDIM denoising step, the complete forward computation residual is cached, and the Taylor coefficients are estimated using finite difference to skip part of the network computation. The dynamic target alignment phase employs short action blocks to increase the replanning frequency; Long action blocks are used to maintain continuity during the contact interaction phase; The pose condition transfer phase is skipped using aggressive computation to reduce overhead; Conservative scheduling is used in the static object acquisition phase to avoid early failures.

9. A robot dynamic operating system based on single static demonstration, dynamic enhancement, and dynamic adaptive reasoning, characterized in that: The robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning, as described in any one of claims 1-8, includes a demonstration acquisition module, a geometric reconstruction module, a keyframe decomposition module, a dynamic trajectory editing module, a dynamic perception strategy training module, and a stage adaptive reasoning module. The demonstration acquisition module is used to acquire demonstration data including scene point cloud, end effector pose, and robot body state for a single static robot operation demonstration. The geometric reconstruction module is used to perform single-image 3D mesh reconstruction, directed bounding box coarse registration, and multi-resolution ICP fine registration on the target object to generate normalized object point cloud assets; the robot point cloud assets are rendered based on the robot URDF model and forward kinematics to ensure geometric consistency. The keyframe decomposition module is used to decompose the static demonstration data into multiple keyframes and define four types of operation stages based on the keyframes, including the static object acquisition stage, the dynamic target alignment stage, the contact interaction execution stage, and the pose condition transfer stage. The dynamic trajectory editing module is used to sample dynamic enhancement parameters and generate multiple physically executable dynamic operation trajectories and stage labels through rigid body transformation, target relative pose tracking, anchored contact segment replay, and motion planning transfer. The dynamic perception strategy training module is used to train a dynamic perception strategy that integrates diffusion action generation and stage classification based on an augmented dataset. The stage-adaptive inference module is used to adaptively adjust the diffusion denoising steps, Taylor acceleration ratio, and action block execution length according to the real-time prediction stage during deployment, thereby achieving low-latency closed-loop control.

10. A robot, characterized in that, The robot includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the robot dynamic operation method based on single static demonstration dynamic enhancement and dynamic adaptive reasoning as described in any one of claims 1-8.

Images