A robotic method of operation and system

Abstract

The present application relates to the technical field of robot intelligent control, and particularly relates to a robot operation method and system. The output subtask label of a high-level visual language semantic planner is relied on, and a low-level action strategy can continuously call the semantic label to output actions, without waiting for the end of single high-level reasoning, so that the continuity of action execution is improved. The present application injects subtask semantics into a visual feature space, so that the robot focuses on corresponding visual targets and spatial relationships in different operation stages, and the action matching accuracy of multi-stage tasks is improved. The present application sets a semantic perception timing integration mechanism, and when the semantic label is switched or semantic rollback is performed, the old and new action blocks are weighted and fused, so that the action mutation across subtasks is relieved, and the stability of contact operation tasks is improved. When the robot operation is detected to be abnormal or the subtask state is reversed, the present application can re-determine the recovery subtask stage planning action, and the abnormal autonomous recovery ability of long-time domain tasks is enhanced.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, and in particular to a robot operation method and system. Background Technology

[0002] With the continuous development and implementation of applications such as intelligent manufacturing, flexible assembly, warehousing and sorting, and service robots, the operation mode of robots is undergoing a significant transformation. It has gradually upgraded from single, fixed-path mechanical execution to an autonomous intelligent operation mode adapted to natural language commands, multi-stage task processes, and unstructured environments. Traditional industrial robots rely heavily on manually taught trajectories or pre-programmed instructions, making them only suitable for standardized production scenarios where the work object's position is fixed and environmental fluctuations are minimal. However, in new and complex tasks involving multiple objects, containers, drawers, and other complex interactive objects, such as desktop operations and flexible assembly, the demands on robots have increased dramatically. They need to rely on real-time visual observation to determine the current task progress and dynamically switch action execution strategies for different operation stages according to the work scenario.

[0003] To meet the evolving needs of autonomous robot operation, the industry has gradually developed various technical methods tailored to robot operation tasks. Among these, imitation learning and action segmentation methods have been widely applied. These methods rely on expert demonstration training to learn and construct a mapping relationship from visual images and ontological states to continuous actions, effectively completing various delicate robot operation tasks such as grasping, placing, and opening / closing drawers, with excellent performance. Meanwhile, with the rapid iterative development of Vision-Language Models (VLM) and Vision-Language-Action Models (VLA), robots have acquired the ability to perform semantic understanding based on natural language and visual observation, opening up new technical pathways for the implementation of long-term autonomous robot operation tasks. Building upon this foundation, the industry has successively proposed several targeted optimization algorithms and models: the Action Chunking with Transformers (ACT) method, relying on the Transformer structure, can predict multiple future actions at once, effectively alleviating the error accumulation problem caused by stepwise action prediction, and demonstrating good performance in dual-arm fine manipulation tasks; the Diffusion Policy method models the robot action generation process as a conditional denoising diffusion process, which can adapt to multimodal action distributions and achieves excellent performance in various robot operation benchmark tests; OpenVLA, as an open-source visual language action model, is trained based on massive robot demonstration data, and can accurately generate robot actions by combining image information and language instructions, possessing extremely strong semantic generalization capabilities; the subsequently proposed SmolVLA reduces deployment costs through a lightweight visual language action model and asynchronous inference stack, achieving decoupling of perception, action prediction, and action execution, effectively improving the responsiveness of robot operations.

[0004] While the aforementioned techniques optimize robot performance in terms of action generation, semantic understanding, and deployment efficiency, existing solutions still have significant shortcomings and limitations, failing to fully adapt to complex, long-duration robot operation scenarios. First, although the ACT method optimizes action prediction errors, its core remains an end-to-end mapping from vision and ontology state to action sequences. Task stage information is only implicitly contained in network features, lacking explicit semantic sub-task state variables, thus failing to achieve accurate explicit determination of task stages and physical states. Second, while the Diffusion Policy method enhances action generation capabilities, its inference process requires multiple denoising steps and lacks an explicit recovery mechanism for physical state regression in long-duration tasks, resulting in insufficient fault tolerance. Third, while the OpenVLA model excels in semantic generalization, its massive size significantly increases deployment costs and introduces response uncertainty when directly applied to continuous action loop operations. While SmolVLA solved the problem of efficient deployment of VLA models and improved the robot's response speed, it did not specifically address the core challenge of "actively reverting to the previous subtask after physical state regression" in long-term operations.

[0005] In summary, while existing motion segmentation, diffusion strategies, and VLA series models have achieved individual improvements in robot motion generation capabilities, semantic understanding capabilities, and system deployment efficiency, in real-world complex work scenarios, if unexpected situations such as object slippage, manual intervention in movement, grasping failure, or container state changes occur during task execution, existing strategies cannot accurately determine that the current physical state has regressed. They often continue to advance subsequent sub-tasks, leading to the continuous accumulation of errors. At the same time, the fixed forward execution flow cannot adapt to special scenarios such as task state reversal or failure of pre-execution conditions, making it difficult to meet the operational needs of complex long-term autonomous robot operation. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the defect that existing long-time-domain robot operation methods usually embed the task phase within the continuous action strategy, making it difficult to re-determine the sub-task to be executed when the physical task state regresses or the prerequisite completion conditions fail.

[0007] To address the aforementioned technical problems, the present invention provides a robot operation method, comprising: Real-time acquisition of visual observation images of each frame at each moment during the robot's execution of the target operation task; The target operation task instruction, the first The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. based on The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. A future multi-step robot action sequence starting from a given time point, and robot control instructions are generated based on the future multi-step robot action sequence; Repeat the above steps until the target operation task is completed.

[0008] Preferably, the following steps are performed for each moment within each first update cycle: based on the subtask label corresponding to the end moment of the first update cycle, combined with the visual observation image and robot body state at that moment, a future multi-step robot action sequence starting from the next moment of that moment is generated.

[0009] Preferably, the method for generating robot control instructions based on the future multi-step robot action sequence includes: Traversal For each future time step starting from a given time, summarize all candidate robot actions for that future time step; for all candidate actions at that future time step, based on the generation time of each candidate action and... The time interval is used to obtain the temporal decay weight of each candidate action at the future time step; Based on the consistency of the subtask labels used when generating all candidate actions at each future time step, and the temporal decay weights of all candidate actions, the final execution action at each future time step is obtained, and robot control instructions are generated.

[0010] Preferably, the method for obtaining the final execution action for each future time step, based on the consistency of the subtask labels used when generating all candidate actions at each future time step and the temporal decay weights of all candidate actions, includes: Traversal Each future time step, starting from a specific moment; If all candidate actions in the current future time step use the same subtask label when they are generated, then the final action to be executed in the future time step is obtained based on all candidate actions in the current future time step and their corresponding temporal decay weights. If there are differences in the subtask labels used when generating candidate actions at the current future time step, then the subtask label switching boundary is used as the dividing line to divide all candidate actions at that future time step into candidate actions before label switching and candidate actions after label switching; for candidate actions before label switching, their temporal decay weight is multiplied by the semantic decay factor to obtain the target weight; for candidate actions after label switching, their temporal decay weight is used as the target weight. Based on all candidate actions within the current future time step and their corresponding target weights, obtain the final action to be executed in that future time step.

[0011] Preferably, the high-level visual language semantic planner is the Qwen3-VL-2B visual language model.

[0012] Preferably, based on The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. Future multi-step robot action sequences starting from a specific time point include: Will The corresponding subtask tag at any time The corresponding visual observation image and robot body state at any given time are used as inputs for the low-level action strategy, and the output is... A sequence of future multi-step robot actions, starting from a specific time point, includes: Will The visual observation image corresponding to a given time is encoded as a visual feature; Will The robot's body state at any given moment is encoded as a body state feature; Will The subtask label corresponding to each time step is encoded as a semantic feature; Scaling and offset parameters are generated based on semantic features. The scaling and offset parameters are then used to perform feature-level modulation on the visual features to obtain the target visual features. Based on the target's visual features and ontological state features, generate... A sequence of future multi-step robot actions, starting from a specific moment.

[0013] Preferably, the training method for the high-level visual language semantic planner includes: Obtain an expert operation demonstration dataset, which includes at least visual observation image data, robot body state data, and robot motion data; The expert operation demonstration dataset is divided into subtasks, and subtask labels are generated for each time step. The visual history window size is set to N. The first training sample is constructed based on the target operation task instruction and the visual observation images of each frame in the visual history window. The subtask label corresponding to the next moment after the end moment of the visual history window is used as the supervision label of the first training sample to train the high-level visual language semantic planner.

[0014] Preferably, the training method for low-level action strategies includes: The visual observation image at the end of the visual history window, the robot's body state, and the subtask label corresponding to the next moment after the end of the visual history window are used as the second training samples; the future multi-step robot action sequence starting from the next moment after the end of the visual history window is used as the supervision label for the second training samples to train the low-level action strategy.

[0015] Preferably, the method for dividing the expert operation demonstration dataset into subtasks includes: Based on the gripper opening and closing states at various times in the expert operation demonstration dataset, identify all interactive events in the expert operation demonstration dataset. Using each interaction event as a dividing boundary, the demonstration trajectory in the expert operation demonstration dataset is divided into several continuous segments, and each segment is assigned a unique subtask label for its corresponding time.

[0016] The present invention also provides a robot operating system, comprising: The data acquisition module is used to acquire visual observation images of each frame at each moment during the robot's execution of the target operation task in real time; The subtask tag generation module is used to generate subtask tags for target operation tasks, including the first subtask tag. The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. Control command generation module, used for... The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. The system generates a sequence of future multi-step robot actions starting from a given time point. Based on this sequence, robot control instructions are generated to control the robot to perform actions. The system then returns to the execution data acquisition module to complete the target operation task.

[0017] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The robot operation method and system described in this invention utilize a high-level visual language semantic planner to perform operations based on the target operation task instructions and the... The end of the first update cycle And its multiple frames of visual observation images from the previous N consecutive time steps, to determine the Nth time step. The end of the first update cycle The corresponding subtask labels enable explicit semantic representation of task stages during robot operation. Continuous multi-frame visual observations can characterize the current scene state and recent physical state changes, which helps reduce stage confusion in multi-stage tasks compared to embedding task stages within continuous action strategies. Low-level action strategies are based on... The corresponding subtask tag at any time Generate the corresponding visual observation image and robot body state at each moment. The system generates a sequence of future multi-step robot actions starting from a specific moment, providing corresponding action generation conditions for different task stages and reducing short-term jitter and error accumulation that may occur during stepwise single-action prediction. When the predicted physical task state is inconsistent with the predetermined task progress, or when the completion conditions of a sub-task are no longer met, the high-level visual language semantic planner re-determines a recovery sub-task label that matches the current physical task state, causing the low-level action strategy to re-execute the corresponding sub-task. This improves the robot's recovery capability in situations such as grasping failure, target object slippage, placed object being moved back, or task preconditions failing, thereby enhancing the accuracy of the robot when performing long-term tasks.

[0018] To address the issue that while existing VLA models possess visual language semantic understanding capabilities, directly using them as continuous action generators increases deployment costs and may limit action execution due to the time-consuming inference of large models, a non-blocking interface is lacking between high-level semantic inference and low-level action execution, leading to problems such as action waiting, unstable responses, or untimely semantic updates. This invention addresses this issue by allowing low-level action strategies to continue generating future multi-step robot action sequences based on the most recently valid subtask labels while the high-level visual language semantic planner has not yet completed a new round of inference. This reduces the blocking effect of high-level semantic inference on continuous action output. Furthermore, this invention obtains the final execution action for each future time step based on the consistency of subtask labels used in generating all candidate actions at each future time step, as well as the temporal decay weights of all candidate actions. When subtask labels switch or revert, the influence of old semantic action candidates is reduced, allowing action candidates generated under new semantic conditions to gradually dominate the current execution action. This reduces action abrupt changes during cross-subtask switching and improves action continuity during long-term robot operations. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a robot operation method according to the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0021] Reference Figure 1 As shown, this embodiment provides a robot operation method, including: Real-time acquisition of visual observation images of each frame at each moment during the robot's execution of the target operation task; The target operation task instruction, the first The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. In this embodiment, optionally, the high-level visual language semantic planner can adopt any visual language model with the ability to perform multi-frame visual input, text instruction understanding and limited subtask label output, such as any one of the Qwen3-VL-2B visual language model, Qwen2-7B visual language model and Doubao-Seed-2.1-Pro visual language model.

[0022] For example, when a high-level visual language semantic planner uses the Qwen3-VL-2B visual language model, the Qwen3-VL-2B visual language model consists of a visual encoder, a connector, and a Qwen3-LM language decoder. The specific prediction process is as follows: The first The end of the first update cycle The visual feature sequence is obtained by passing multiple frames of visual observation images from the preceding N consecutive time steps through a visual encoder. The target operation task instruction is generated into text embedding features by the word embedding unit of the Qwen3-LM language decoder, and then converted into fixed-dimensional text embedding features by the embedding layer of the Qwen3-LM language decoder. By connecting the visual feature sequence and the text embedding features, a fused multimodal feature sequence is obtained; The fused multimodal feature sequence is then passed through the Qwen3-LM language decoder to obtain the first... The end of the first update cycle The corresponding subtask tag.

[0023] Let the high-level visual language semantic planner be... Its input is and its multiple frames of visual observation images from the previous N consecutive time steps and target operation task instructions Output probability distribution of subtask labels at time step ; The training objective of the high-level visual language semantic planner is to maximize the conditional probability of the true subtask labels, and the loss function can be expressed as: , in, This represents the training loss of the high-level visual language semantic planner.

[0024] In this embodiment, specifically, the training method for the high-level visual language semantic planner includes: Construct a long-term, multi-stage robot operation task scenario, and define the target object, target region, and sub-task sequence of the long-term, multi-stage robot operation task; The long-term, multi-stage robotic operation platform consists of a Songling Piper six-axis robotic arm equipped with a two-finger gripper and three Intel RealSense D435 depth cameras. Scenarios include, but are not limited to, object pickup, object placement, drawer opening and closing, object organization, container placement, and multi-object sequential operations. Each task comprises several sub-tasks with clear physical meanings. These sub-tasks describe the phased semantic states during the robot's operation, such as approaching a target object, grasping a target object, moving to a target area, placing a target object, opening a drawer, and closing a drawer. By breaking down long-term tasks into multiple sub-tasks, a high-level visual-language semantic planner can explicitly model the task progress and provide phased conditional information for low-level action strategies.

[0025] Obtain an expert operation demonstration dataset, which includes at least visual observation image data, robot body state data, and robot motion data; The expert demonstration dataset was collected using a teach pendant control method. Each expert demonstration trajectory in the dataset is denoted as... ,in, This indicates the expert demonstration trajectory. Indicates time, Indicates the length of the trajectory. Indicates the first Visual observation images at each moment, Indicates the first The robot's actions at each time step. The visual observation image. Includes RGB images from a fixed camera and a wrist camera; the robot's movements. This can include the joint angle of the robotic arm and the opening and closing state of the gripper.

[0026] The expert operation demonstration dataset is divided into subtasks, and subtask labels are generated for each time step. The visual history window size is set to N. The first training sample is constructed based on the target operation task instruction and the visual observation images of each frame in the visual history window. The subtask label corresponding to the next moment after the end moment of the visual history window is used as the supervision label of the first training sample to train the high-level visual language semantic planner. N is less than the total number of frames of the expert demonstration trajectory.

[0027] In this embodiment, optionally, the method for dividing the expert operation demonstration dataset into subtasks includes: Based on the gripper opening and closing states at various times in the expert operation demonstration dataset, identify all interactive events in the expert operation demonstration dataset. Using each interaction event as a dividing boundary, the demonstration trajectory in the expert operation demonstration dataset is divided into several continuous segments, and each segment is assigned a unique subtask label for its corresponding time.

[0028] The formula for identifying all interaction events in the expert operation demonstration dataset is as follows: , in, Indicates the first The opening and closing state of the grippers at any given moment. Indicates the first Interaction events at a given moment, Indicates the first The opening and closing state of the grippers at any given moment. For time intervals, For indicator functions, This is the threshold for changes in the gripper state. At that time, it was determined that in the first Grab, release, or other gripper-related interaction events occur near a certain time step.

[0029] Based on the detected interaction events, the demonstration trajectories in the expert operation demonstration dataset will be used. Divide into several continuous segments: , Indicates the first A series of segments, Indicates the number of segments, each segment A unique subtask label is assigned to each corresponding time point. , Indicates the first A series of segments, Examples include "grab a banana," "place a banana," "approach the drawer," and "close the drawer." The segment-level subtask labels are then expanded to frame-level subtask labels, resulting in a fully annotated labeled example dataset. : , in, For the first Visual observation images at each moment, For the first The opening and closing state of the grippers at any given moment. For the first The robot's actions at any given moment For the first Subtask tags for each moment.

[0030] Pick As the endpoint index of the visual history window, select the index. and its preceding successive The visual observation images corresponding to each historical index constitute a visual history window. : , in, For the length of the visual history window, For training time series index Visual observation images acquired at the corresponding time.

[0031] Visual History Window With target operation task instructions Together, they serve as the first training samples input to the high-level visual language semantic planner. And the next moment after the end of the visual history window. Corresponding subtask tags Supervised labels for the first training sample Conduct training, This forms the training dataset for the visual language model. :

[0032] in, This indicates the number of the first training samples in the dataset. Used to train a high-level visual language semantic planner to identify the current subtask label based on multi-frame visual context and task instructions.

[0033] Will The corresponding subtask tag at any time The corresponding visual observation image at each moment, the robot's body state input, and the low-level action policy (Semantic-Conditioned Action Chunking with Transformers, SC-ACT) are used to generate... The robot generates a sequence of future multi-step robot actions starting from a given time, and generates robot control instructions based on this sequence to control the robotic arm and gripper to perform the current sub-task. The robot actions may include: robotic arm joint angle increments, end effector pose increments, gripper opening and closing instructions, or combinations thereof.

[0034] In this embodiment, specifically, the low-level action strategy operates according to a second update cycle, which is shorter than the first update cycle.

[0035] When a state reversal or execution exception occurs, the high-level visual language semantic planner adjusts its strategy based on the current time frame. The subtask labels are re-evaluated based on the multi-frame visual observation images from the previous N time points. The subtask state reversal includes, but is not limited to: the placed object being moved back to the initial area, the target object sliding off the target area, the object remaining in its original position due to a failed grabbing attempt, and the drawer not being opened or closed as expected.

[0036] When the subtask label output by the high-level visual-language semantic planner regresses relative to the current task progress, the low-level action policy re-evaluates the subtask label and generates a new sequence of future multi-step robot actions. For example, when the next object manipulation stage has been entered, but visual observation shows that the previous object is not in the target area, the high-level visual-language semantic planner can regress the current subtask to the object grasping or placing stage, and the low-level policy will then re-execute the corresponding action.

[0037] In this embodiment, specifically, the training method for the low-level action strategy includes: The visual observation image at the end of the visual history window, the robot's body state, and the subtask label corresponding to the next moment after the end of the visual history window are used as the second training samples; the future multi-step robot action sequence starting from the next moment after the end of the visual history window is used as the supervision label for the second training samples to train the low-level action strategy.

[0038] Repeat the above steps until the target operation task is completed.

[0039] If all target objects have reached the preset target area and all predefined subtasks have been completed, a task completion signal is output to determine that the target operation task has been completed. If the task has not been completed, the high-level visual language semantic planner and low-level action strategy are used to perform semantic updates, action block (future multi-step robot action sequence) generation, action fusion and anomaly recovery until the task is completed or the preset termination condition is met.

[0040] In this embodiment, preferably, the low-level action strategy includes a visual encoding module, an ontology state encoding module, a semantic encoding module, a feature-level semantic modulation module, and an action prediction module; Will The corresponding subtask tag at any time The corresponding visual observation image at each moment, the robot's body state input, and the low-level action strategy are used to generate... Methods for sequencing future multi-step robot actions based on a given time point include: Using the visual encoding module, The visual observation image corresponding to a given time is encoded as a visual feature. ; Using the ontology state coding module, The robot's body state at any given moment is encoded as a body state feature; Using the semantic encoding module, The subtask label corresponding to each time step is encoded as a semantic feature. ; Using a feature-level semantic modulation module, scaling and offset parameters are generated based on semantic features. and The formula is: , , in, Scaling parameters that match the visual feature channel dimensions. An offset parameter that matches the visual feature channel dimension. For a multilayer perceptron used to map semantic features to scaling parameters, This is a multilayer perceptron used to map semantic features to offset parameters.

[0041] Scaling and offset parameters are generated based on semantic features. These parameters are then used to perform feature-level modulation on the visual features to obtain the target visual features. The formula is: , in, This indicates channel-level multiplication.

[0042] Using the action prediction module, based on the target's visual features and ontological state features, generate Future multi-step robot action sequences starting from a specific time point , Indicated by The future starting from a certain moment Robotic movements at each time step Indicates the length of a future multi-step robot action sequence. .

[0043] In this way, different subtask labels can guide the action strategy to focus on different visual regions and operational cues. For example, in the grasping phase, the focus is on the relative position of the target object and the gripper, while in the placement phase, the focus is on the relative position of the target area and the end effector. By segmenting actions, the low-level strategy can generate continuous action sequences instead of just outputting single-step actions, thereby reducing short-term motion jitter and gradual error accumulation in long-term operations.

[0044] The loss function for low-level action policies includes behavior clone loss. , is represented as: , in, Indicated by The future starting from a certain moment Realistic labeling of robot movements at each time step.

[0045] When the low-level action policy adopts a conditional variational autoencoder structure, the total loss function is... This can be further expressed as: , in, Indicates behavioral cloning loss. Indicates the weighting coefficient. This represents the regularization loss of latent variables.

[0046] In this embodiment, preferably, a zero-order hold semantic buffer is used to realize the asynchronous connection between the high-level visual language semantic planner and the low-level action strategy.

[0047] For each moment within each first update cycle, perform the following steps: Based on the subtask label corresponding to the end moment of the first update cycle, combined with the visual observation image and robot body state at that moment, generate a future multi-step robot action sequence starting from the next moment of that moment through a low-level action strategy.

[0048] The zero-order hold semantic buffer stores the latest subtask label output by the high-level visual-language semantic planner. When the low-level action policy does not receive a subtask label from the high-level visual-language semantic planner, it reads the latest subtask label stored in the zero-order hold semantic buffer and generates a sequence of future multi-step robot actions. If the high-level visual-language semantic planner has not yet completed new reasoning, the low-level action policy continues to use the most recently valid subtask label in the buffer. In this way, there is no need for synchronization and waiting between high-level semantic updates and low-level action execution.

[0049] In this embodiment, preferably, based on Methods for generating robot control instructions from a future multi-step robot motion sequence starting at a specific time include: Because the low-level execution strategy continuously iterates and predicts over time, the action corresponding to the same future time step can be repeatedly deduced at different low-level execution strategy generation times.

[0050] This invention traverses with For each future time step starting from a given time, summarize all candidate robot actions for that future time step to obtain a set of candidate robot actions for that future time step. For all candidate actions at this future time step, the set Any candidate action ,remember The generation time corresponding to this candidate action and The time interval is used to obtain the temporal decay weight of the candidate action at the future time step: ;in, Indicated by The first time starting from time The set of all candidate actions for a given future time step. For The first time starting from time In the next future time step, candidate actions The time-decay weight, The attenuation coefficient is... This represents an exponential function with the natural constant as its base. Candidate actions Corresponding generation time and The time interval between moments. Based on the consistency of the subtask labels used when generating all candidate actions at each future time step, and the temporal decay weights of all candidate actions, the final execution action for each future time step is obtained, and robot control instructions are generated.

[0051] Existing methods lack a smooth action fusion mechanism during cross-semantic switching. If a high-level semantic tag suddenly switches from one subtask or reverts to another, the low-level action strategy may experience abrupt changes in action, affecting the stability of contact processes such as grasping, placing, and drawer operations.

[0052] In this embodiment, preferably, the method for obtaining the final execution action for each future time step based on the consistency of the subtask labels used when generating all candidate actions at each future time step, and the temporal decay weights of all candidate actions, includes: Traversal Each future time step, starting from a specific moment; If all candidate actions in the current future time step use the same subtask labels when generating them, then the final execution action for that future time step is obtained based on all candidate actions and their corresponding temporal decay weights. The formula is: ,in, For The first time starting from time The final action to be performed in a future time step.

[0053] If the subtask labels used when generating candidate actions at the current future time step differ, then the subtask label switching boundary will be used as the dividing line for that future time step. All candidate actions are divided into a set of candidate actions before tag switching. Candidate action set after tag switching For candidate actions before label switching, multiply their temporal decay weights by the semantic decay factor. , The target weight is used as the target weight; for the candidate action after the label switch, its temporal decay weight is used as the target weight; based on all candidate actions in the current future time step and their corresponding target weights, the final execution action of the future time step is obtained.

[0054] Let the subtask label switching time be... ,but: , , in, Indicated by The first time starting from time A set of candidate actions for tag switching at a future time step. Indicated by The first time starting from time A set of candidate actions after tag switching at a future time step. Indicated by The first time starting from time The set of all candidate actions for a given future time step. express Candidate actions within, Indicates candidate actions The corresponding generation time.

[0055] Based on all candidate actions within the current future time step and their corresponding target weights, obtain the final action to be executed in that future time step. Further expressed as: .

[0056] When a subtask semantic switch or semantic rollback occurs, the candidate actions generated under the old semantic conditions satisfy... Its target weight is determined by Become Therefore, the old semantic action candidates are additionally attenuated; the action candidates generated under the new semantic conditions satisfy... Its target weight remains unchanged. In this way, action blocks generated under new semantic conditions can gradually dominate the currently executed action, while old semantic action blocks can still provide a smooth transition in a short period of time, thereby reducing abrupt action changes during subtask semantic switching or semantic rollback. The time for generating candidate actions is The corresponding subtask tag at that time, The time for generating candidate actions is The subtask labels correspond to each moment. Through this temporal integration method, multiple historical action blocks can jointly participate in the generation of the current action, thereby reducing the impact of single prediction errors on the executed action. When the high-level visual language semantic planner outputs new subtask labels, the semantic-aware temporal integration divides the action candidates into a set of candidate actions before the label switch and a set of candidate actions after the label switch according to the semantic update time.

[0057] This invention aims to address the challenge of coordinating high-level semantic reasoning with low-level continuous action execution in long-term robotic operations. Specifically, it addresses the following: first, the lack of explicit subtask state representation in existing end-to-end action strategies, hindering proactive recovery during physical state rollback; second, the mismatch between the inference speed of large visual language models and the demands of continuous action execution, potentially causing action output to be blocked by high-level semantic reasoning; third, the discontinuity and lack of smoothness in action execution during semantic label switching or rollback in long-term multi-subtask operations; and fourth, the inability of fixed-sequence execution methods in multi-stage tasks to adjust task stages in real-time based on visual state. To this end, this invention proposes a robot operation method that decouples the high-level visual language semantic planner from the low-level action strategy. It uses a zero-order semantic buffer to pass the latest subtask labels and achieves smooth cross-subtask action through semantically aware temporal integration, enabling the robot to perform semantic rollback and recovery operations based on visual state during continuous action execution.

[0058] Based on Embodiment 1, this Embodiment 2 further elaborates on the robot autonomous operation method proposed in this invention by combining a specific long-term robot operation task. It should be understood that this embodiment is only used to illustrate the complete technical solution of this invention and is not intended to limit the scope of protection of this invention. Without departing from the core idea of ​​this invention, those skilled in the art can apply this method to other long-term robot operation tasks, such as drawer operations, object sorting, assembly operations, multi-target grasping and placement, etc.

[0059] This second embodiment selects a desktop multi-object orderly transfer task as a practical case. The complete task objective is to sequentially place bananas and mangoes into a plate container, and then place toys and pen holders into a basket container. This task is a multi-stage, long-time domain operation, which, after being broken down, contains multiple continuous semantic sub-tasks. Throughout the operation, the robot needs to rely on visual observation information to determine the current task progress stage in real time and match the currently valid sub-task label to output the corresponding robot control action.

[0060] First, the operational scenario is constructed, including a robotic arm, grippers, a multi-view camera, a desktop, a target object, and a target container. Target objects include, but are not limited to, bananas, mangoes, toys, and pen holders; target containers include, but are not limited to, plates and baskets. In this embodiment, the robotic arm is used to perform actions such as grasping, moving, and placing; the grippers are used to grasp and release the target object; and the multi-view camera is used to collect visual observations during the task execution process. The camera may include a fixed external camera and a wrist camera, where the fixed external camera provides overall desktop scene information, and the wrist camera provides local visual information near the grippers.

[0061] Robot operation tasks can be divided into multiple sub-tasks with clear semantic meanings, such as: (1) approaching a banana; (2) grasping and moving a banana; (3) placing a banana; (4) approaching a mango; (5) grasping and moving a mango; (6) placing a mango; (7) approaching a toy; (8) grasping and moving a toy; (9) placing a toy; (10) approaching a pen holder; (11) grasping and moving a pen holder; (12) placing a pen holder. Collecting expert demonstration datasets: In the constructed operation scenario, 50 expert demonstration datasets were collected using a teaching method. Each expert demonstration dataset contains 1200 frames of data representing the complete execution process of the task from start to finish. Each frame contains image data from three cameras and data on the joint angles of the robotic arm and the opening and closing of the gripper.

[0062] The system divides subtasks and generates frame-level subtask labels. After collecting expert demonstration trajectories, subtasks are divided based on the interaction events between the robot and the target object. The high-level visual language semantic planner does not directly generate robot actions but only determines the semantic stage of the current task. For example, when the robot has placed the banana and is preparing to handle the mango, the high-level visual language semantic planner outputs a subtask label related to the mango operation; when it detects that the banana has been moved back to the initial area, the high-level visual language semantic planner can output a backtracking subtask label related to the banana operation.

[0063] By using low-level action strategies, different sub-task labels can guide the low-level strategy to focus on different visual cues. For example, in the "grabbing banana" stage, the strategy focuses more on the relative position of the gripper and the banana; in the "placing banana" stage, the strategy focuses more on the relative position of the banana and the plate.

[0064] Since the reasoning process of the high-level visual language semantic planner and the low-level action execution process are not necessarily synchronized, this embodiment sets a zero-order hold semantic buffer to connect the two. This design allows the high-level visual language semantic planner to act as a low-rate semantic state monitoring module, while the low-level action strategy can continuously output action blocks and drive the robot to perform operations. When generating decay weights based on the inference time corresponding to each robot action at each future execution time step, and fusing them with the corresponding robot actions to obtain the final execution action for that time step, , .

[0065] The robot performs a "desk tidying" task. During this task, it first acquires current multi-view visual observations and inputs the visual history window into a high-level visual-language semantic planner. The high-level planner outputs the current activity subtask label, such as "approaching the banana" or "grabbing the banana." This subtask label is written to a zero-order hold semantic buffer. The low-level SC-ACT policy reads the latest valid label from the buffer and generates the corresponding action block. The robot controls the robotic arm and gripper to complete the current subtask based on the fused action. During execution, new visual observations are continuously acquired, and the visual history window is constantly updated. If the high-level planner detects a change in the current task stage, it updates the subtask label in the buffer; the low-level action policy then generates new action blocks based on the new semantic label.

[0066] During task execution, state reversals or execution anomalies may occur. For example, after a banana has been placed on a plate, external intervention may move it back to its initial position on the table; or the robot may fail to grasp the object, leaving it still in its original position. In such cases, if the system continues to perform subsequent object operations according to the fixed task schedule, the task may fail.

[0067] In this embodiment, the high-level visual language semantic planner re-evaluates the activity subtask based on the current visual history. When visual observations show that the physical prerequisites of a previous subtask are no longer met, the high-level visual language semantic planner can output the correct subtask label. For example, when the system is ready to operate on the mango, but visual observations show that the banana is not in the plate, the high-level visual language semantic planner can revert the current activity subtask to "approach the banana" or "grab the banana".

[0068] After the reverted subtask label is written to the zero-order hold semantic buffer, the low-level SC-ACT policy regenerates the action block based on the label, enabling the robot to re-execute the corresponding subtask. Once the state is restored, the high-level visual-linguistic-semantic planner outputs subsequent subtask labels, allowing the robot to continue completing the remaining tasks.

[0069] To verify the effectiveness of the proposed method in long-term robot operation tasks, this embodiment uses five tasks as test scenarios: Task 1: "Put the toy into the box", Task 2: "Open the drawer", Task 3: "Stack paper cups", Task 4: "Tidy up the table", and Task 5: "Put the bottle into the drawer". The robot's task completion status during the multi-stage operation process is observed and recorded.

[0070] During testing, the method of this invention was compared with methods such as ACT, Diffusion Policy, RDT, and SmolVLA. Evaluation metrics included task success rate and subtask success rate. Task success rate represents the proportion of times the robot completes a task without human intervention; subtask success rate represents the ratio between the number of completed subtasks and the total number of subtasks. Each subtask is independent. To comprehensively evaluate the capability of each baseline, evaluators were allowed to intervene and assist the strategy in completing failed subtasks so that execution could continue.

[0071] The experimental results are shown in Table 1, which compares the performance of different methods in the five tasks.

[0072] Table 1

[0073] In short-time tasks, the method of this invention can stably complete the grasping, moving, and placement of target objects. In long-time tasks containing multiple sub-task stages, the method of this invention can determine the current active sub-task based on visual observation and generate corresponding continuous actions through a semantic conditional action segmentation strategy. Compared with methods that rely solely on fixed action strategies or end-to-end action generation, the method of this invention exhibits better task continuity and sub-task completion capabilities in long-time tasks.

[0074] To verify the function of each module, ablation experiments were set up for comparison. The complete method includes a high-level visual language semantic planner and a zero-order hold semantic buffer; Ablation ①: After removing the high-level visual language semantic planner, subtasks are switched in a fixed time sequence; Ablation ②: After removing the zero-order hold semantic buffer, the low-level action strategy needs to wait for the high-level semantic update before continuing to execute.

[0075] As shown in Table 2, Table 2 compares the results of the ablation experiment.

[0076] Table 2

[0077] The comparison results show that after removing the high-level visual language semantic planner, the system has difficulty correcting the current subtask according to the actual visual state; after removing the zero-order hold semantic buffer, the continuity of action execution decreases and the performance of long-term tasks deteriorates.

[0078] The above verification results show that by combining a high-level visual language semantic planner, a low-level action strategy, a zero-order hold semantic buffer, and semantic perception timing integration, the present invention enables robots to maintain good task progress judgment and continuous action execution capabilities in long-term multi-stage operation tasks, thereby improving the completion effect of multi-stage robot operation tasks.

[0079] This third embodiment provides a robot operating system, including: The data acquisition module is used to acquire visual observation images of each frame at each moment during the robot's execution of the target operation task in real time; The subtask tag generation module is used to generate subtask tags for target operation tasks, including the first subtask tag. The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. Control command generation module, used for... The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. The system generates a sequence of future multi-step robot actions starting from a given time point. Based on this sequence, robot control instructions are generated to control the robot to perform actions. The system then returns to the execution data acquisition module to complete the target operation task.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A robot operation method, characterized in that, include: Real-time acquisition of visual observation images of each frame at each moment during the robot's execution of the target operation task; The target operation task instruction, the first The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. For each moment within each first update cycle, perform the following steps: Based on the subtask label corresponding to the end moment of the first update cycle, and combined with the visual observation image and robot body state at that moment, generate a future multi-step robot action sequence starting from the next moment of that moment; based on The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. The method for sequencing future multi-step robot actions based on a given time point is as follows: The corresponding subtask tag at any time The corresponding visual observation image and robot body state at any given time are used as inputs for the low-level action strategy, and the output is... A sequence of future multi-step robot actions, starting from a specific time point, includes: Will The visual observation image corresponding to a given time is encoded as a visual feature; Will The robot's body state at any given moment is encoded as a body state feature; Will The subtask label corresponding to each time step is encoded as a semantic feature; Scaling and offset parameters are generated based on semantic features. The scaling and offset parameters are then used to perform feature-level modulation on the visual features to obtain the target visual features. Based on the target's visual features and ontological state features, generate... Future multi-step robot action sequences starting from a specific moment; Based on the aforementioned future multi-step robot action sequence, robot control instructions are generated, including: Traversal For each future time step starting from a given time, summarize all candidate robot actions for that future time step; for all candidate actions at that future time step, based on the generation time of each candidate action and... The time interval is used to obtain the temporal decay weight of each candidate action at the future time step; Based on the consistency of the subtask labels used when generating all candidate actions at each future time step, and the temporal decay weights of all candidate actions, the final execution action for each future time step is obtained, and robot control instructions are generated; wherein, the method for obtaining the final execution action for each future time step includes: Traversal Each future time step, starting from a specific moment; If all candidate actions in the current future time step use the same subtask label when they are generated, then the final action to be executed in the future time step is obtained based on all candidate actions in the current future time step and their corresponding temporal decay weights. If there are differences in the subtask labels used when generating candidate actions at the current future time step, then the subtask label switching boundary is used as the dividing line to divide all candidate actions at that future time step into candidate actions before label switching and candidate actions after label switching; for candidate actions before label switching, their temporal decay weight is multiplied by the semantic decay factor to obtain the target weight; for candidate actions after label switching, their temporal decay weight is used as the target weight. Based on all candidate actions within the current future time step and their corresponding target weights, obtain the final action to be executed in that future time step; Repeat the above steps until the target operation task is completed.

2. The robot operation method according to claim 1, characterized in that, The high-level visual language semantic planner is the Qwen3-VL-2B visual language model.

3. The robot operation method according to claim 1, characterized in that, Training methods for high-level visual language semantic planners include: Obtain an expert operation demonstration dataset, which includes at least visual observation image data, robot body state data, and robot motion data; The expert operation demonstration dataset is divided into subtasks, and subtask labels are generated for each time step. The visual history window size is set to N. The first training sample is constructed based on the target operation task instruction and the visual observation images of each frame in the visual history window. The subtask label corresponding to the next moment after the end moment of the visual history window is used as the supervision label of the first training sample to train the high-level visual language semantic planner.

4. A robot operation method according to claim 3, characterized in that, Training methods for low-level action strategies include: The visual observation image at the end of the visual history window, the robot's body state, and the subtask label corresponding to the next moment after the end of the visual history window are used as the second training samples; the future multi-step robot action sequence starting from the next moment after the end of the visual history window is used as the supervision label for the second training samples to train the low-level action strategy.

5. A robot operation method according to claim 3, characterized in that, Methods for subtask partitioning of expert demonstration datasets include: Based on the gripper opening and closing states at various times in the expert operation demonstration dataset, identify all interactive events in the expert operation demonstration dataset. Using each interaction event as a boundary, the demonstration trajectory in the expert operation demonstration dataset is divided into several continuous segments, and each segment is assigned a unique subtask label for its corresponding time.

6. A robot operating system based on the robot operation method as described in any one of claims 1 to 5, characterized in that, include: The data acquisition module is used to acquire visual observation images of each frame at each moment during the robot's execution of the target operation task in real time; The subtask tag generation module is used to generate subtask tags for target operation tasks, including the first subtask tag, the second subtask tag, and the third subtask tag. The end of the first update cycle The system inputs multiple frames of visual observation images from the preceding N consecutive time steps into a high-level visual language semantic planner, and outputs the first N frames. The end of the first update cycle The corresponding subtask tags; among them, Here, N is the index for the first update cycle, and N is the size of the visual history window. Control command generation module, used for... The corresponding subtask tag at any time The corresponding visual observation image and robot body state at each moment are used to generate a sequence of data. The system generates a sequence of future multi-step robot actions starting from a given time point. Based on this sequence, robot control instructions are generated to control the robot to perform actions. The system then returns to the execution data acquisition module to complete the target operation task.

Images