A robot low-step-count diffusion closed-loop execution method

Abstract

The application discloses a robot low-step diffusion closed-loop execution method, comprising the following steps: collecting multi-modal teaching data to form a data set; reading the teaching trajectory from the data set and constructing a training sample according to a time window to perform offline training; using the offline trained model, using the current multi-modal observation and task instruction as a condition, using a few-step diffusion sampling to generate a joint speed action block, and issuing a joint speed command; after executing the specified number of actions according to the joint speed command, re-observing and re-planning to form a new action block, and after boundary timing fusion with the unexecuted part, generating a new joint speed command to continue executing the action. The application solves the problems of high deployment delay of multi-modal strategy, insufficient responsiveness of fixed action block, obvious boundary jitter of frequent re-planning, discontinuous joint speed command and scattered collection and execution system in the prior art.

Description

Technical Field

[0001] This invention relates to the field of robot perception and control technology, and more particularly to the fields of robot control, robot imitation learning, multimodal visual language strategy, diffusion model action generation, robotic arm teleoperation teaching acquisition, and robot real-machine closed-loop execution technology. Specifically, it relates to a robot low-step diffusion closed-loop execution method. Background Technology

[0002] In the field of robot perception and control technology, the mainstream technologies for current intelligent robot operation strategies include behavior cloning, action block generation, and diffusion strategies. These often combine Transformer models, state-space sequence models, and the DDIM sampling algorithm to achieve multimodal fusion, long sequence modeling, and inference acceleration, respectively. Behavior cloning can learn observation-action mappings to achieve imitation operations, action block generation can improve motion continuity, and diffusion strategies can model complex multi-peak action distributions. However, these technologies still have significant shortcomings in real-world robotic arm deployment scenarios: existing models cannot simultaneously achieve multimodal expressiveness, long sequence modeling effectiveness, and online low-latency inference performance. The traditional action block generation method suffers from high inference overhead and poor real-time performance, and the multimodal interaction of the state space sequence model is insufficient. At the same time, the traditional action block generation method has a contradiction between response flexibility and motion stability. The complete execution of the action block leads to environmental response lag, and high-frequency replanning will cause problems such as speed jumps, trajectory jitter, and motion loss. In addition, most existing real machine control systems are built with distributed scripts, which have poor integration, consistency and reproducibility. They lack a complete instruction conversion, motion limiting and safety arbitration mechanism. The discontinuous speed at the boundary of the action block can easily cause impact vibration of the robot arm and operation failure. Overall, it is difficult to meet the implementation requirements of multimodal understanding, low-latency inference, smooth and continuous motion and safe control of real machines at the same time. Summary of the Invention

[0003] To address the aforementioned issues, the present invention aims to provide a low-step diffusion closed-loop execution method for robots, which addresses problems in existing technologies such as high deployment delay of multimodal strategies, insufficient responsiveness of fixed action blocks, significant boundary jitter due to frequent replanning, discontinuous joint speed commands, and dispersed acquisition and execution systems.

[0004] To achieve the above technical objectives, this application provides a low-step diffusion closed-loop execution method for robots, comprising the following steps: Collect multimodal teaching data to form a dataset; The teaching trajectories are read from the dataset, and training samples are constructed according to time windows to perform offline training; Using the offline-trained model, and based on the current multimodal observations and task instructions, the joint velocity action blocks are generated using a few-step diffusion sampling, and joint velocity commands are issued. After executing a specified number of steps based on the joint velocity command, the action is re-observed and re-planned to form a new action block. After boundary timing fusion with the unexecuted part, a new joint velocity command is generated to continue executing the action.

[0005] Preferably, when constructing the dataset, the robot profile configuration is started and loaded, the remote operation input device is controlled to establish a connection and complete the input calibration, the remote operation control quantity is converted into joint speed commands and multimodal synchronous acquisition is performed to obtain the teaching trajectory in order to construct the dataset.

[0006] Preferably, when acquiring the joint velocity command, based on the end-effector spatial control input, the obtained end-effector linear velocity and angular velocity are solved by Jacobian matrix or inverse kinematic velocity to obtain the joint velocity vector, so as to form the joint velocity command; The joint space control input is directly converted into a joint velocity vector to form the joint velocity command, wherein the joint velocity command is constrained by a velocity upper limit and an acceleration upper limit before being sent.

[0007] Preferably, during multimodal synchronous acquisition, global view camera images, wrist view camera images, robot joint states, end effector states, and joint speed commands are acquired using a unified time reference to obtain the multimodal teaching data.

[0008] Preferably, when acquiring the teaching trajectory, the joint velocity motion is saved as a supervised learning target to the dataset, wherein each trajectory includes task instructions, image observations, robot state, gripper state, joint velocity motion, and timestamp.

[0009] Preferably, when constructing training samples, each training sample contains current or a period of historical observation, task instructions, robot state, and joint velocity action blocks, wherein each action in the action block is a joint velocity vector.

[0010] Preferably, during offline training, based on the training samples, the multimodal conditions are encoded, and after interactive modeling of the multimodal features is performed using a conditional encoder, the conditional memory is mapped to the action denoising representation space through an alignment adapter to train the action block denoising decoder. The model parameters are optimized according to the training objective so that the offline trained model can generate joint velocity action blocks for multiple future control cycles based on the current multimodal observations and task instructions.

[0011] Preferably, when generating joint velocity action blocks using low-step diffusion sampling, based on the offline trained model, and taking the current multimodal observations and task instructions as conditions, future joint velocity action blocks are generated through conditional feature construction, noisy action block initialization, and low-step reverse denoising.

[0012] Preferably, when generating future joint velocity action blocks, the current multimodal input is preprocessed and encoded separately to construct multimodal conditional memory; After aligning the conditional memory representation, the noisy future joint velocity action block is initialized and the few-step sampling time is determined to predict the noise component in the currently selected diffusion time. The noisy action block is then updated between adjacent selected sampling times based on the noise prediction value to form a schedulable future joint velocity action block.

[0013] Preferably, when executing a joint speed command, based on the total number of steps corresponding to the future joint speed action block, a specified number of steps is selected with the starting step as the reference to create a queue of actions to be executed in this round. Actions are submitted step by step according to the high-level control cycle. After converting the single-step action into an actual control signal, the execution of this round ends and the closed-loop connection information is retained. After executing the specified number of steps, the prefix of the newly generated future joint speed action block is used to perform boundary timing fusion with the tail joint speed sequence of the future joint speed action block that has not yet been executed. A safety state preemption mechanism is set up. When HomeZone is activated, the homing process is activated, the input device fails, the robot state is abnormal, or the control source is switched, the input running state is reset, teleoperation or model inference output is paused, and a preset safety joint speed or homing trajectory is executed. After the safety state is released, the current observation is reacquired and the planning is resumed.

[0014] The present invention discloses the following technical effects: This invention can improve the stability of robot closed-loop execution. By performing time-series fusion of the tail of the old action block and the prefix of the new action block at the boundary of the action block, the joint speed command will not change drastically during replanning, thereby reducing robot arm vibration, end effector jitter and contact action failure.

[0015] This invention can reduce online inference latency. By using fewer steps of diffusion sampling and heterogeneous model division of labor, it can generate future joint velocity motion blocks in fewer sampling steps, which is suitable for the latency requirements in real robotic arm control scenarios.

[0016] This invention can simultaneously take into account both action continuity and environmental responsiveness. By fixing the length L of the action block, it ensures that the model has a certain forward planning capability. By only executing the first r steps and then re-observing, it ensures that the system can adjust subsequent actions in a timely manner according to environmental changes.

[0017] This invention can improve the consistency between training and deployment. The actions recorded in the teaching data are joint velocity sequences, and the model training target is also the future joint velocity action block. Furthermore, the online execution output is still a joint velocity command, thus avoiding the problem of inconsistency between the training action space and the deployment control command format.

[0018] This invention improves engineering feasibility by integrating remote operation input, multimodal acquisition, HDF5 data recording, model inference, action block scheduling, joint speed control, gripper control, and safe return into a unified process, which facilitates reproduction, debugging, and actual deployment.

[0019] This invention can improve safety by controlling coordination and safety control, which can pause teleoperation or model output when HomeZone is active, homing, input abnormality, or control source switching, and prioritize safety actions to reduce the risks of misoperation and abnormal model output. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method described in this invention.

[0022] Figure 2 This is a structural diagram of the TAM heterogeneous diffusion strategy described in this invention.

[0023] Figure 3 This is a flowchart of the closed-loop replanning process for the fixed action block described in this invention.

[0024] Figure 4 This is a schematic diagram of the action block boundary timing fusion described in this invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] like Figure 1As shown, this invention provides a low-step diffusion closed-loop execution method for robots, which belongs to a closed-loop execution process for multimodal teaching acquisition, low-step diffusion motion block generation, joint velocity command output, fixed motion block closed-loop replanning, and timing fusion for robotic arms and end effectors.

[0027] Step S100 performs multimodal teaching acquisition.

[0028] In step S100 of one embodiment, step S101 starts and loads the robot profile configuration.

[0029] For example, by selecting the robot type, robot network address, input backend type, end effector type, and control mode in the graphical interface or command line entry, a robot profile configuration is loaded. This profile configuration stores parameters for the robotic arm, gripper, ROStopic, ROSservice, controller, Home pose, and HomeZone. By using the profile configuration as the source of truth for the hardware interfaces, the mixing of teleoperation parameters and underlying hardware interfaces can be avoided.

[0030] In step S100 of one embodiment, step S102 involves establishing a connection and completing input calibration for the remotely operated input device.

[0031] In one embodiment, the teleoperation input device is used to receive teaching input from an operator, wherein the device may be a VR controller, a handheld device, or a vision-based gesture input device.

[0032] For example, when the input device is a VR controller, a control method using relative pose plus clutch can be adopted. After the clutch is pressed, the current pose of the controller is recorded as a reference, and then the relative pose change of the controller is converted into the motion target of the robotic arm end effector or the joint speed control quantity. Frame reset, maximum relative pose angle limit, input dead zone, and key debouncing can also be set to reduce the risk of hand drift, accidental triggering, and pose flipping. When the input device is a gamepad or visual gesture input, corresponding motion control quantities can be generated based on axis values ​​or changes in hand key points.

[0033] In step S100 of one embodiment, step S103 converts the teleoperation control quantity into a joint speed command.

[0034] In one implementation, for end-effector spatial control input, the end-effector linear velocity and angular velocity can be obtained first, and then the joint velocity vector can be obtained by solving the Jacobian matrix or inverse kinematic velocity.

[0035] In one embodiment, joint velocity vectors can be directly generated from joint space control inputs.

[0036] For example, the joint speed command is constrained by a speed limit and an acceleration limit before being sent to prevent abrupt changes in commands between adjacent cycles.

[0037] In step S100 of one embodiment, step S104 performs multimodal synchronous acquisition.

[0038] In one embodiment, global view camera images, wrist view camera images, robot joint states, end effector states, and joint speed commands are acquired using a unified time reference.

[0039] For example, to ensure the availability of training data, images, states, and action sequences should be aligned according to timestamps, and nearest neighbor, linear interpolation, or fixed sampling period resampling may be used if necessary.

[0040] In step S100 of one embodiment, step S105 involves writing the teaching trajectory into the HDF5 dataset.

[0041] For example, each trajectory records at least the task instruction, image observation, robot state, gripper state, joint velocity motion, and timestamp. The joint velocity motion is saved as a supervised learning target, enabling the subsequent model to directly learn the mapping from the current observation to the future joint velocity motion block.

[0042] Step S200 performs offline training.

[0043] In step S200 of one implementation, step S201 reads the teaching trajectory from the HDF5 dataset and constructs training samples according to time windows.

[0044] For example, each training sample contains the current or a period of historical observations, task instructions, robot state, and a future joint velocity action block of length L, wherein each action in the action block is a joint velocity vector.

[0045] In step S200 of one embodiment, step S202 encodes the multimodal conditions.

[0046] For example, a visual encoder is used to extract global view image features and wrist view image features, a text encoder is used to extract task instruction features, and a state encoder is used to extract robot state features. All types of features are mapped to a unified dimension and then input into a conditional encoder.

[0047] In step S200 of one embodiment, step S203 performs interactive modeling of multimodal features using a conditional encoder.

[0048] For example, the conditional encoder can employ a Transformer encoding structure to correlate language tasks, image targets, local wrist viewpoints, and robot states to obtain conditional memories.

[0049] In step S200 of one embodiment, step S204 aligns the adapter to map the conditional memory to the action denoising representation space.

[0050] For example, since the features output by the conditional encoder are biased towards semantic and observational representations, while the action block denoising decoder needs to process noisy joint velocity sequences, an alignment adapter is set up to perform dimensional transformation, normalization, and representation bridging.

[0051] In step S200 of one embodiment, step S205 trains the action block denoising decoder.

[0052] In one embodiment, diffused noise is added to the real future joint velocity motion block to obtain a noisy motion block; the motion block denoising decoder predicts the noise or recovers the target motion block under the modulation of diffused step embedding and conditional memory.

[0053] For example, the action block denoising decoder preferably employs a state-space sequence model in order to efficiently model a joint velocity sequence of length L.

[0054] In step S200 of one embodiment, step S206 optimizes the model parameters according to the training objective.

[0055] For example, after training is completed, the model is enabled to generate joint velocity action blocks for multiple future control cycles based on the current multimodal observations and task instructions.

[0056] Step S300: Online inference and closed-loop execution process.

[0057] In step S300 of one embodiment, step S301 acquires the current observation.

[0058] For example, during online execution, global view images, wrist view images, robot status, and task instructions are acquired at the current moment, and the model inference stage is entered.

[0059] In step S300 of one embodiment, step S302 generates a joint velocity motion block by a few-step diffusion sampling. Based on the current multimodal observation and task instructions obtained in step S301, a future joint velocity motion block of length L is generated by constructing conditional features, initializing noisy motion blocks, and performing few-step reverse denoising.

[0060] In step S302 of one embodiment, step S3021 preprocesses and encodes the current multimodal input.

[0061] In one implementation, let the global perspective image at the start of the nth round of closed-loop programming be... Wrist view image The standard task instruction is l.

[0062] For example, the images are processed according to the size adjustment and pixel normalization methods used in the training phase, and the task instructions are tokenized according to the input rules of the text encoder.

[0063] For example, a frozen pre-trained ResNet is used to extract global scene features. Local features of the wrist .

[0064] For example, frozen CLIP text encoder task semantic features are employed. .

[0065] In one implementation, the robot state can also be obtained. State features are extracted through the robot's state coding unit. .

[0066] In step S302 of one embodiment, step S3022 constructs a multimodal conditional memory. , and Mapped to the same latent feature dimension; when robot state conditions are applied, it will also... Mapped to the latent feature dimension.

[0067] For example, such as Figure 2 As shown, modality coding and position coding are added to various features, and then they are concatenated into a conditional feature sequence in a predetermined order. Conditional encoders Global interactive modeling is performed to correlate task semantics, target objects, global scene, and local wrist observations to obtain conditional memory. When state features are included, It also includes the current robot state information. The condition encoder is preferably a Transformer encoding structure.

[0068] In step S302 of one embodiment, step S3023 involves aligning the representation of the conditional memory.

[0069] For example, the alignment adapter remembers conditions. Lightweight feedforward mapping and normalization are performed to adjust the feature dimension and numerical distribution of the conditional memory, resulting in an aligned conditional memory that matches the action block denoising decoder. . In subsequent reverse denoising steps, it serves as a conditional prefix to constrain the generated joint velocity sequence to conform to the current task and scenario; when the conditional memory contains the robot state, it is also used to constrain the action sequence to conform to the current robot state.

[0070] In step S302 of one embodiment, step S3024 initializes the noisy future joint velocity motion block and determines the few-step sampling time.

[0071] For example, let the number of joints in the robotic arm be J, and the gripper command dimension be... Then the dimension D of a single-step compound action is D = J + When the gripper command is not included in the action block, D=J. In a preferred embodiment, J=6. =1, therefore D=7. Let the number of online reverse denoising steps be... Selecting from the diffusion time in the training phase The sampling times are arranged from largest to smallest, with the termination time set to 0, as shown in the following formula: ; In the formula, This indicates the initial sampling time of the short-step reverse denoising process; Indicates the end time; This represents the noisy action block corresponding to the initial sampling time; This represents a standard Gaussian distribution with a mean of 0 and a covariance equal to the identity matrix. Represents the identity matrix.

[0072] For example, the sampling time typically corresponds to the terminal time T of the training diffusion process. The few-step diffusion sampler samples standard Gaussian noise in the L×D dimensional action block space as the initial noisy action block at the maximum sampling time. This initial noise block is only the starting point for inverse denoising and is not directly executed as a robot control command.

[0073] In step S302 of one embodiment, step S3025 predicts the noise component in the currently selected diffusion time.

[0074] For example, the L row vectors of the current noisy action block are mapped to L action feature labels through an action projection layer, the currently selected diffusion time is mapped to a diffusion step embedding, and the L row vectors of the current noisy action block are mapped to L action feature labels through an action projection layer ... The conditional prefix, concatenated with the action feature markers, is input into the action block denoising decoder. After updates through multiple Mamba decoding layers, the hidden states corresponding to the last L action markers are extracted from the joint sequence output and mapped back to the L×D dimensional action space via a linear prediction head to obtain the noise prediction block at the currently selected diffusion time. .

[0075] For example, the action block denoising decoder is preferably Mamba or other state-space sequence models.

[0076] In step S302 of one embodiment, step S3026 updates the noisy action block between adjacent selected sampling times based on the noise prediction value.

[0077] For example, the few-step diffusion sampler updates the current noisy action block to a less noisy action block based on the cumulative noise scheduling coefficients corresponding to the current selected time and the next selected time, as well as the noise prediction block.

[0078] For example, taking deterministic DDIM updates as an example, the noiseless action block is first estimated: Then obtain the action block for the next reverse sampling step: ; This represents a noise-free action block estimated based on the current noise prediction. This represents the noisy action block at the currently selected moment; This represents the cumulative retention coefficient corresponding to the currently selected time in the training spread noise scheduling; This represents the noise prediction block at the currently selected time. This represents the action block obtained from the next backsampling time after the update; This represents the cumulative retention coefficient corresponding to the next selected time. All the above coefficients are taken from the preset noise scheduling table during the training phase.

[0079] The cumulative noise scheduling coefficients in the above formula are taken from the current selected time and the next selected time, respectively, instead of directly using the number of online reverse updates as the training diffusion time. Steps S3025 and S3026 are repeated in descending order of the selected time sequence, for a total of [number missing] executions. Update the code until the termination time 0, and obtain the denoising action block. In other implementations, other low-step sampling methods may also be used.

[0080] In step S302 of one embodiment, step S3027 decodes the denoising result into a schedulable future joint velocity action block.

[0081] For example, The i-th row vector is decoded into the action of the i-th future high-level control cycle; if the action quantity was normalized during the training phase, it is then inversely normalized based on the action statistics parameters of the training data; if action normalization was not used, the action decoding result is used directly. The result is: ; in, Let J be the J-dimensional joint velocity vector for the i-th future high-level control cycle. For synchronized gripper commands; if the gripper is controlled by an independent strategy, a single-step action may only contain joint velocity vectors. After dimensionality verification, non-numerical checks, and gripper command format conversion, the candidate future joint velocity motion blocks for this round are obtained: .

[0082] For example, the individual actions in the action block are arranged in the order of their future execution times; the joint speed limit, acceleration limit, and safety status arbitration are completed by the subsequent control and coordination module.

[0083] Step S303 executes only the first r steps of the action block.

[0084] In one implementation, such as Figure 3 As shown, the action block scheduling does not send all L steps of actions generated in step S302 to the robot at once, but only schedules the first r steps in each round of closed-loop planning, where 1≤r≤L. Specifically, it includes the following steps.

[0085] In step S303 of one embodiment, step S3031 creates a queue of actions to be executed in the current round.

[0086] For example, action block scheduling retrieves action blocks from future joint velocity action blocks in chronological order: As the queue to be executed in this round, and Marked as an unexecuted tail in this round. Unexecuted tail R n Not issued directly in this round; when r=L, R n It is an empty sequence.

[0087] In step S303 of one embodiment, step S3032 submits actions step by step according to the high-level control cycle.

[0088] For example, suppose the execution frequency of high-level actions is... Then the high-level action update cycle =1 / The action block scheduling module sets the execution index i=0, and first submits the 0th action to the control coordination module; after each high-level control cycle, the execution index is incremented by 1, and the next action in the queue is submitted, until i=r-1. Joint velocity vectors and gripper commands are submitted synchronously using the same action sequence number and time stamp to keep the timing of the robotic arm movement consistent with the opening and closing of the gripper.

[0089] In step S303 of one embodiment, step S3033 converts the single-step action into an actual control signal.

[0090] For example, after receiving a single-step action, the speed limit, acceleration limit, and safety status judgment mentioned in step S304 are performed on the joint velocity vector. The arbitrated joint velocity command is sent to the lower-level joint velocity controller, and the gripper command is sent to the gripper controller. When the lower-level control frequency is higher than the higher-level action update frequency, the lower-level controller can maintain the joint velocity command within the current higher-level control cycle, or in one embodiment, interpolate adjacent higher-level commands and continuously output them according to the lower-level control cycle.

[0091] In step S303 of one embodiment, step S3034 ends the current round of execution and retains closed-loop connection information.

[0092] For example, as shown in Figure 4 , each time a single-step action is executed, the action serial number, issuing time, and the robot state after execution can be recorded. After the first r actions are executed, the action block scheduling module stops continuing to read the current action block, and retains the unexecuted tail part R n , then proceeds to step S305 to reacquire observations and generate the n+1-th round of future joint velocity action block. When r<L, R n is used in step S306 to establish an overlapping window with the prefix of the new action block for time sequence fusion; when r=L, this fusion may not be performed. If the safety state described in step S307 is triggered before the completion of the first r steps, the subsequent issuing of the current action queue will be stopped immediately, and the safety control module will seize control authority.

[0093] Step S304 issues joint velocity commands.

[0094] In one embodiment, the joint velocity vector of the k-th control cycle is received, subjected to velocity amplitude limiting, acceleration amplitude limiting and safety state judgment, and then sent to the underlying joint velocity controller. For a six-degree-of-freedom manipulator, the command format may be a six-dimensional joint velocity vector; for manipulators with other degrees of freedom, the command dimension is consistent with the number of joints. The gripper command may be sent synchronously as an independent channel, or form a composite action together with the joint velocity command.

[0095] Step S305 re-observes and re-plans after executing r steps.

[0096] In one embodiment, after completing the first r steps of actions, images and robot states are re-collected to form new conditional inputs, and the model generates a new future joint velocity action block. In this way, the robot can periodically respond to environmental changes while maintaining the continuity of action blocks.

[0097] Step S306 performs time sequence fusion at the action block boundary.

[0098] In one embodiment, after the new action block is generated, an overlapping window is established with the prefix of the new action block by extracting the unexecuted tail joint velocity sequence of the previous action block. For the i-th action in the overlapping window, the following standard formula is used for fusion: ; In the formula, represents the i-th fused joint velocity command; represents the fusion weight, with a value range of 0≤ ≤1; This indicates that the velocity of the i-th joint in the tail section was not executed in the previous action block; This represents the i-th joint velocity in the prefix of the new action block; where the joint velocities at the tail of the old action block are represented as old sequence terms, and the joint velocities in the prefix of the new action block are represented as new sequence terms, with weights decreasing over time. Weights can be expressed using linear, cosine, exponential, or other non-linear decreasing methods. The resulting fused joint velocity command retains the local continuity of the previous action block while gradually transitioning to the motion plan under the new observation conditions.

[0099] Step S307: Security state preemption.

[0100] In one implementation, when the HomeZone is activated, the homing process is activated, the input device fails, the robot's state is abnormal, or the control source is switched, a high-priority signal is sent to reset the input running state, suspend teleoperation or model inference output, and execute a preset safe joint speed or homing trajectory. After the safe state is lifted, the system reacquires the current observations and resumes planning.

[0101] In summary, this invention heterogeneously separates multimodal conditional modeling and motion denoising modeling. A conditional encoder is used to uniformly model the global view image, wrist view image, robot state, and task commands. An alignment adapter then maps the conditional memory to a suitable representation space for motion denoising. Finally, a state-space sequence model decoder generates future joint velocity motion blocks. This design avoids the representation conflicts and computational overhead caused by a single network simultaneously handling cross-modal interaction and long motion block denoising.

[0102] This invention combines low-step diffusion sampling with fixed action block closed-loop scheduling. Instead of generating single-step actions, it generates a future joint velocity action block of length L in each planning process. At the same time, the system does not mechanically execute the complete action block, but only executes the joint velocity commands for the first r steps, and then reacquires observations and plans again, thereby achieving a balance between continuity and responsiveness.

[0103] This invention implements temporal fusion between the tail of the old action block and the prefix of the new action block, establishes an overlapping window at the replanning boundary, and adopts a fusion weight that decreases over time, so that the robot can smoothly transition from the previous action block to the next action block, reducing joint speed jumps and end-effector trajectory jitter.

[0104] This invention forms a complete engineering closed loop from remote operation teaching, HDF5 data recording, model training, online inference to joint speed execution. By receiving input from VR controllers, handles or visual gestures, it collects dual-view images, robot status, gripper status and joint speed control sequences, and distributes joint speed commands and gripper commands uniformly through the control and coordination module during the online execution phase.

[0105] This invention introduces safety mechanisms such as Home, HomeZone, controller switching, and input state reset. When the safety return, home zone constraint, or home state is triggered, this invention can pause teleoperation or model inference output and let the safety action take over, avoiding conflicts between manual input, model output, and safety actions.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0108] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A low-step diffusion closed-loop execution method for robots, characterized in that, Includes the following steps: Collect multimodal teaching data to form a dataset; The teaching trajectories are read from the dataset, and training samples are constructed according to time windows to perform offline training; Using the offline-trained model, and based on the current multimodal observations and task instructions, the joint velocity action blocks are generated using a few-step diffusion sampling, and joint velocity commands are issued. After executing a specified number of steps based on the joint velocity command, the action is re-observed and re-planned to form a new action block. After boundary timing fusion with the unexecuted part, a new joint velocity command is generated to continue executing the action.

2. The robot low-step diffusion closed-loop execution method according to claim 1, characterized in that: When constructing the dataset, the robot profile configuration is started and loaded. After the remote operation input device establishes a connection and completes input calibration, the remote operation control quantity is converted into joint speed commands and multimodal synchronous acquisition is performed to obtain the teaching trajectory in order to construct the dataset.

3. The robot low-step diffusion closed-loop execution method according to claim 2, characterized in that: When acquiring the joint velocity command, based on the end-effector space control input, the obtained end-effector linear velocity and angular velocity are solved by Jacobian matrix or inverse kinematic velocity to obtain the joint velocity vector, so as to form the joint velocity command; The joint space control input is directly converted into a joint velocity vector to form the joint velocity command, wherein the joint velocity command is constrained by a velocity upper limit and an acceleration upper limit before being sent.

4. The robot low-step diffusion closed-loop execution method according to claim 2, characterized in that: During multimodal synchronous acquisition, global view camera images, wrist view camera images, robot joint states, end effector states, and joint speed commands are acquired using a unified time reference to obtain the multimodal teaching data.

5. The robot low-step diffusion closed-loop execution method according to claim 2, characterized in that: When acquiring the teaching trajectory, the joint velocity motion is saved as a supervised learning target to the dataset. Each trajectory includes task instructions, image observations, robot state, gripper state, joint velocity motion, and timestamp.

6. The robot low-step diffusion closed-loop execution method according to claim 1, characterized in that: When constructing training samples, each training sample contains the current or a period of historical observations, task instructions, robot state, and joint velocity action blocks, where each action in the action block is a joint velocity vector.

7. The robot low-step diffusion closed-loop execution method according to claim 1, characterized in that: During offline training, multimodal conditions are encoded based on the training samples, and after interactive modeling of multimodal features using a conditional encoder, the conditional memory is mapped to the action denoising representation space through an alignment adapter to train the action block denoising decoder. The model parameters are optimized according to the training objective to control the offline trained model to generate joint velocity action blocks for multiple future control cycles based on the current multimodal observations and task instructions.

8. The robot low-step diffusion closed-loop execution method according to claim 1, characterized in that: When generating joint velocity action blocks using low-step diffusion sampling, based on an offline-trained model, and taking current multimodal observations and task instructions as conditions, future joint velocity action blocks are generated through conditional feature construction, noisy action block initialization, and low-step reverse denoising.

9. The robot low-step diffusion closed-loop execution method according to claim 8, characterized in that: When generating future joint velocity action blocks, the current multimodal input is preprocessed and encoded separately to construct multimodal conditional memory; After aligning the conditional memory representation, the noisy future joint velocity action block is initialized and the few-step sampling time is determined to predict the noise component in the currently selected diffusion time. The noisy action block is then updated between adjacent selected sampling times based on the noise prediction value to form a schedulable future joint velocity action block.

10. The robot low-step diffusion closed-loop execution method according to claim 9, characterized in that: When executing joint velocity commands, based on the total number of steps corresponding to the future joint velocity action block, a specified number of steps is selected with the starting step as the reference, and a queue of actions to be executed in this round is created. Actions are submitted step by step according to the high-level control cycle. After converting the single-step action into an actual control signal, the current round of execution ends and the closed-loop connection information is retained. After executing the specified number of steps, the prefix of the newly generated future joint velocity action block is used to perform boundary timing fusion with the tail joint velocity sequence of the future joint velocity action block that has not yet been executed. A safety state preemption mechanism is set up. When HomeZone is activated, the homing process is activated, the input device fails, the robot state is abnormal, or the control source is switched, the input running state is reset, teleoperation or model inference output is paused, and the preset safety joint velocity or homing trajectory is executed. After the safety state is released, the current observation is reacquired and the planning is resumed.

Images