Robot long sequence control method and device based on graph thought chain and double channels

Abstract

The application belongs to the technical field of embodied intelligent robots, and particularly relates to a robot long-sequence control method and device based on graph thinking chain and double channels. The method comprises the following steps: maintaining an explicit task state and providing a deterministic logical constraint through a graph executor; driving a vision-language model to generate a structured text channel intermediate product; decoding a spatial channel intermediate product in the form of a Gaussian heat map through a heat map decoding head; extracting spatial guidance features based on the spatial channel intermediate product; and finally fusing the text instruction and the spatial guidance features by an action specialist, and generating a continuous action sequence based on flow matching denoising sampling. The application realizes long-sequence robot control with logical reliability, spatial accuracy and online error correction capability with low computing overhead through graph structure priori and extremely simple double channel design.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligent robot technology, specifically relating to a long sequence control method and device for robots based on graph thinking chain and dual channels. Background Technology

[0002] In the field of robot control, the precise execution of long sequence operations such as multi-object grasping and placement is a core technological challenge. These tasks require systems to have long-range planning capabilities, strictly adhere to causal logic in each step, and provide high-precision spatial control for each operation. However, current mainstream Visual-Language-Action Models (VLA) are based on the approximate Markov assumption of "predicting the next action from the current observation" for decision-making. This makes it difficult to effectively manage the task's historical state and reasonably decompose complex steps, easily leading to error accumulation, logical confusion, and state drift, which seriously affects execution stability.

[0003] To improve the long sequence task processing capabilities of VLA, existing research has proposed various optimization schemes, but all have significant limitations: Phased modeling schemes (such as Long-VLA) decompose long tasks into multiple sub-stages for sequential execution and management, but because the decision-making process is an implicit black box, lacking explicit state tracking and step logic constraints, they are prone to logical inversion and repetitive operations, making it difficult to guarantee execution continuity; Text-generating sub-task schemes (such as PI0.5) complete the decomposition of long sequence steps by generating descriptive text, but pure text can only express semantic logic and cannot connect the visual positioning and spatial coordinate information required for robot operation, lacking precise spatial guidance, ultimately leading to spatial drift in action execution; Next-stage visual state generation schemes (such as CoT-VLA) assist planning by predicting future scenes, but lack structural constraints from the external task graph, resulting in insufficient planning stability; One-time sub-target image chain generation schemes (such as CoTDiffusion) transform long sequences into short-term tracking, but due to the huge computational overhead of image chain generation, they cannot meet real-time control requirements.

[0004] In addition, most existing solutions rely on independent external state detectors to determine the success or failure of subtasks and drive stage transitions, which makes the system lack self-consistent closed-loop online correction capabilities and susceptible to detection noise interference, leading to task interruption and further reducing the reliability of long sequence operations.

[0005] In summary, existing technologies have several drawbacks when dealing with long-sequence robot operation tasks, including difficulty in resolving logical inversion and repetitive operations, lack of built-in closed-loop correction capabilities, lack of precise spatial guidance to suppress error accumulation, high computational overhead, and poor real-time performance in some solutions. Summary of the Invention

[0006] This application proposes a robot long sequence control scheme based on graph thinking chain and dual channels, aiming to solve the problems of logical inversion and repetitive operation, lack of built-in closed-loop online correction capability, inability to provide accurate spatial guidance to suppress error accumulation, and high computational overhead and poor real-time performance of some schemes in the robot long sequence control task in the prior art.

[0007] The first aspect of this application provides a long sequence control method for robots based on graph thinking chains and dual channels, including: Based on a predefined task graph template, the graph executor maintains the explicit state of the current task and outputs the state context information. The state context information, the predefined task graph, the robot's observation information, and the overall task instructions are input into the visual-language model. The visual-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the visual-language model generating the object identifier to be operated through the state context information. Based on the state context information and the last hidden state of the visual-language model, a spatial channel intermediate product is generated through a heatmap decoding head. The spatial channel intermediate product includes a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and operation target region. Based on the first Gaussian heatmap and the second Gaussian heatmap, a spatial guidance feature is generated to represent the spatial relationship from the operation source region to the operation target region. The intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot are input into the motion expert module, which then generates a continuous sequence of actions to control the robot.

[0008] In some embodiments of this application, the graph executor is configured to perform the following operations: Maintain the explicit state including the current node state, the identifier of the currently operated object, and the set of objects whose operations have been completed; The robot's single operation primitive executed based on the current text channel intermediate product and the spatial channel intermediate product is determined as the current subtask, wherein the operation primitive includes grasping and placing; Based on the success or failure result of the current subtask, the explicit state is updated according to a predefined deterministic state transition function T, i.e. ,in, For a moment The explicit state. For a moment The explicit state; The output is a sequence of tokens as state context information, wherein the token sequence includes <current> 、 <next> 、 <prim> 、 <completed> 、 <remaining>At least one of them, and used to inject into the vision-language model.

[0009] In some embodiments of this application, the graph executor transforms the set of objects that have been operated on into a logical mask decoding constraint and applies it to the object identifier decoding process of the visual-language model, so that when generating the object identifier, selection can only be made from outside the set of objects that have been operated on.

[0010] In some embodiments of this application, the method further includes: When the current subtask is determined to have failed, the graph executor maintains its current state and triggers the vision-language model to regenerate scene-adaptive semantic instructions based on the latest observation information. When the current subtask is determined to be successful, the graph executor completes the update and transition of the explicit state according to the deterministic state transition function.

[0011] In some embodiments of this application, the determination is based on at least one of the following conditions: abnormal force after the gripper closes, abnormal displacement after the gripper closes, operation timeout, or target not achieved.

[0012] In some embodiments of this application, the intermediate product of the text channel is a structured cue word.<OBJ,INST> The object to be operated is identified <obj>The scene-adaptive semantic instructions are abstract object identifiers. <inst>Includes at least one of the following: contact surface, force, path constraint, or correction strategy.

[0013] In some embodiments of this application, the heatmap decoding head is a new lightweight module of the visual-language model, and the number of parameters is less than the preset threshold of the visual-language model backbone network; The semantics of the spatial channel intermediate product generated by the heatmap decoding head are jointly determined by the primitive type and object identifier in the state context information. When the primitive type is grasping, the first Gaussian heatmap represents the current position of the end effector and the second Gaussian heatmap represents the grasping point area of ​​the target object. When the primitive type is placement, the first Gaussian heatmap represents the position of the grasped object and the second Gaussian heatmap represents the position of the target placement area. The dimensions of the first Gaussian heatmap and the second Gaussian heatmap are both 56×56. The process by which the heatmap decoding head generates the aforementioned Gaussian heatmap is as follows: using two sets of learnable query vectors, cross-attention calculation is performed on the last layer of the hidden state of the visual-language model, which contains the state context information, to obtain the low-resolution hidden representation in the source space and the low-resolution hidden representation in the target space. Then, the low-resolution hidden representation in the source space and the low-resolution hidden representation in the target space are adjusted to a low-resolution two-dimensional matrix format, and then decoded to 56×56 through transposed convolution or upsampling, finally generating the first Gaussian heatmap and the second Gaussian heatmap. The text channel and the spatial channel maintain orthogonal division of labor by adopting an attention isolation strategy. That is, the query vector of the heatmap decoder only reads the hidden state of the visual-language model and does not write back the token sequence of the visual-language model. In addition, the action expert module only reads the prefix token and does not directly perform attention calculation on the tokens of fast actions.

[0014] In some embodiments of this application, the step of fusing the first Gaussian heatmap and the second Gaussian heatmap to generate a spatial guidance feature characterizing the spatial relationship from the operation source region to the operation target region includes: Using the first Gaussian heatmap respectively and the second Gaussian heatmap As weights, the visual feature maps aligned with the Gaussian heatmap space are... Perform weighted pooling to obtain the source visual vector. and target visual vector The formula is: , , in, These are the pixel coordinates of the feature map. Indicates that it is located at The feature vector of the location; For the source visual vector The target visual vector and the difference between the two The concatenated combined vector is encoded to obtain the spatial guidance feature. The formula is: , in, This represents a multilayer perceptron. This indicates a vector concatenation operation.

[0015] In some embodiments of this application, the step of inputting the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot into the motion expert module, and having the motion expert module generate a continuous sequence of actions for controlling the robot, includes: The motion expert module uses the robot's current observation information, the intermediate products of the text channel, and the spatial guidance features. As a condition, the distribution of conditional actions is modeled by flow matching, and a continuous sequence of actions for controlling the robot is generated through multi-step denoising sampling.

[0016] A second aspect of this application provides a robot long sequence control device based on graph thinking chain and dual channels, comprising: The maintenance module is used to maintain the explicit state of the current task based on a predefined task graph template through the graph executor, and output the state context information. The first generation module is used to input the state context information, the predefined task graph, the robot's observation information, and the overall task instructions into the vision-language model. The vision-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the vision-language model generating the object identifier to be operated through the state context information. The second generation module is used to generate spatial channel intermediate products through a heatmap decoding head based on the state context information and the last hidden state of the visual-language model. The spatial channel intermediate products include a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and the operation target region. The extraction module is used to fuse the first Gaussian heatmap and the second Gaussian heatmap to generate a spatial guidance feature that represents the spatial relationship from the operation source region to the operation target region. The third generation module is used to input the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot into the motion expert module, which then generates a continuous motion sequence for controlling the robot.

[0017] In summary, the robot long sequence control method and apparatus based on graph thinking chain and dual-channel proposed in the embodiments of this application firstly, by introducing a graph actuator to maintain explicit state and drive a constrained decoding mechanism, the visual-language model is constrained to select objects to be operated on only from unfinished objects. This transforms the logical decision of object selection in long sequences into deterministic constraints imposed by an external state machine, thereby eliminating logical inversion and repetitive operations, and achieving reliable and stable tracking of the task causal chain. Secondly, by constructing a text channel intermediate product bound to the graph state and a closed-loop correction mechanism, when a subtask is determined to have failed based on physical signals, the graph actuator maintains its state and triggers the visual-language model to regenerate scene-adaptive instructions. This allows the system to achieve adaptive anomaly recovery without relying on susceptible external detectors, thus obtaining highly robust built-in closed-loop correction capabilities and significantly reducing the risk of task interruption. Simultaneously, by designing a spatial channel intermediate product, a Gaussian heatmap is generated by a heatmap decoding head and encoded into compact spatial guidance features through a spatial feature extraction step, which is then injected into the action expert. This mechanism effectively solves the error accumulation problem caused by the lack of precise spatial guidance by providing pixel-level precision heatmap guidance, focusing motion generation on key operational areas and improving the spatial accuracy of long sequence operations. Ultimately, these effects are achieved through a minimalist dual-channel intermediate product and a lightweight heatmap decoder. This design avoids the enormous overhead of generating high-dimensional image chains without significantly increasing the computational burden on the backbone network, thus supporting high-frequency generation of continuous actions by motion experts while maintaining extremely low computational and parameter overhead. This effectively resolves the contradiction between high computational costs and the difficulty in balancing real-time requirements in existing technologies. Through the deep coupling of the above technical features, this application achieves systematic improvements in four aspects: logical reliability, system robustness, spatial accuracy, and computational efficiency, providing an innovative high-performance solution for long sequence complex robotic operations. Attached Figure Description

[0018] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 A schematic diagram of the layered modeling of the original PI0.5; Figure 2 This is a system architecture diagram of a VLA control method based on graph thinking chain and dual-channel intermediate products according to some embodiments of this application; Figure 3 is a schematic diagram of a dual-channel explicit intermediate product generation process according to some embodiments of this application; Figure 4 This is a schematic diagram of a spatial feature extraction and action generation network structure according to some embodiments of this application; Figure 5 This is an exemplary computer system architecture diagram to which the VLA control method based on graph thinking chain and dual-channel intermediate product of this application applies; Figure 6 This is a flowchart illustrating a long sequence control method for robots based on graph thinking chain and dual channels, according to some embodiments of this application; Figure 7 This is a schematic diagram of a long sequence control device for a robot based on graph thinking chain and dual channels, according to some embodiments of this application. Detailed Implementation

[0019] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the use of the terms "system," "apparatus," "unit," and / or "module" in this application is one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0020] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0021] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0022] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0023] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0024] In the field of embodied intelligent robots, the ability to perform long sequences of operations such as grasping and placing multiple objects is a key indicator of their intelligence level. Currently, the mainstream technological approach relies on vision. language Action Model (Visual) Language The Action Model (VLA) is implemented using a basic paradigm that leverages a vision system pre-trained on large-scale data. Language Model (Visual) The Language Model (VLM) serves as a core cognitive component for understanding the environment and task instructions, and drives motion experts to generate robot control signals.

[0025] However, the standard VLA framework has inherent structural flaws when dealing with long-sequence tasks. Its decision-making process is usually modeled as an approximate Markov decision process, which is highly dependent on the current instantaneous observation and lacks explicit management of the overall task structure and historical execution states. This makes it prone to logical confusion, error accumulation, and state drift in long-sequence execution.

[0026] To overcome these limitations, the industry has proposed various long sequence enhancement schemes, but all of them have significant drawbacks that are difficult to overcome: Defects of Phased Modeling Path and its Logical Management: With Long VLA-based solutions break down long-sequence tasks into multiple sub-stages for execution. However, the division and transition of these stages rely entirely on implicit reasoning from VLM, lacking explicit tracing and constraints from an external deterministic state machine. This makes the system prone to logical inversion and repetitive operations during long-sequence execution, such as attempting to place ungrabbed objects or repeatedly processing completed objects, resulting in low logical reliability.

[0027] Defects in generating intermediate text representation paths and their spatial guidance: Schemes exemplified by PI0.5 utilize VLM to generate text-based subtask instructions to achieve step decomposition. For example... Figure 1 As shown in the original PI0.5 layered modeling diagram, the high-level text output and the low-level action generation are layered independently. However, plain text instructions can only express semantic information and cannot provide pixel-level spatial positioning information that is aligned with the current visual observation. This results in a lack of precise spatial guidance when the action expert executes the instructions, ultimately leading to spatial attention drift and error accumulation.

[0028] The limitations of generating visual state paths and their computational efficiency: With CoT Solutions such as VLA and CoTDiffusion predict future intermediate visual states or generate sub-target image chains to assist planning. While they have a certain visual foresight capability, the computational overhead of generating high-dimensional images or image chains is huge, which is significantly contradictory to the real-time control requirements of robots. This makes it difficult for some solutions to meet the real-time requirements of actual deployment.

[0029] Furthermore, existing solutions generally rely on independent external state detectors to determine the success or failure of subtasks and drive stage transitions, resulting in an open-loop execution mode for the entire system. Once the detector is affected by observation noise and makes a misjudgment, task execution is easily interrupted. The system lacks the intrinsic capability for online diagnosis and autonomous recovery based on native sensor signals, i.e., it does not have built-in closed-loop online correction capability.

[0030] In summary, existing technologies, while improving the capabilities of VLA long sequence tasks, collectively expose four core defects: easy occurrence of logical inversion and repetitive operations, lack of built-in closed-loop online correction capability, inability to provide accurate spatial guidance to suppress error accumulation, and high computational overhead and poor real-time performance in some solutions.

[0031] To systematically address the four major shortcomings mentioned above, this application proposes a VLA control method based on graph thinking chains and dual-channel intermediate products. For example... Figure 2 As shown in the system architecture diagram, the core of this application's solution lies in introducing a graph executor to achieve deterministic state management and driving the generation of explicit intermediate products with semantic and spatial orthogonality. Figure 3 (Dual-channel explicit intermediate product generation process diagram) and Figure 4 (A schematic diagram of the spatial feature extraction and action generation network structure is shown.) The text channel outputs structured semantic instructions to ensure logical constraints, while the spatial channel provides pixel-level spatial guidance through a Gaussian heatmap. The two are fused with extremely low computational overhead, and closed-loop online correction is achieved through built-in physical signal monitoring. The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0032] This application's system, based on the PI0.5 two-layer architecture, introduces a task graph prior and dual-channel explicit intermediate products. Without altering the VLM backbone and the internal structure of the action experts, it significantly improves the stability and success rate of long-sequence tasks with extremely low parameter and computational costs. As shown in Figure 2, the system mainly includes the following core components: Graphical actuator: Maintain the state of long-sequence explicit tasks (idle / fetch, etc.), generate state context tokens and apply constraints to decode them, driving state transitions; Visual-Language-Motion Large Model (VLM Backbone): Receive multimodal input (visual information, text instructions, graph structures), and output structured subtask instructions ( <obj> + <inst>); Heatmap Decoding Head: Gaussian heatmaps are generated by decoding the hidden features of VLM as intermediate products of spatial channels; Spatial Feature Extractor: Integrate Gaussian heatmaps with visual observation features to create spatial guidance tokens; Action Expert: Based on visual observation, text commands, and spatial guidance features, a continuous action sequence is generated by combining noise input. Execution monitoring and correction module: The success or failure of a subtask is determined and fed back to the graph executor to drive state updates, forming a closed-loop control.

[0033] After the multimodal input is processed by the VLM backbone, the intermediate products of the text channel (subtask instructions) and the intermediate products of the spatial channel (Gaussian heatmap) are output respectively. The state context output by the graph actuator is simultaneously injected into the VLM backbone and the heatmap decoding head to achieve the coordination of logical constraints and spatial guidance. Finally, the motion expert fuses the multimodal conditions to generate continuous actions, driving the robot body to execute.

[0034] Figure 5 This is an exemplary computer system architecture diagram to which the VLA control method based on graph thinking chain and dual-channel intermediate products of this application applies. For example... Figure 5 As shown, the system mainly includes an execution unit (robot body), a sensing unit, and a computing unit (core computing unit). Each unit interacts with other units via a communication link. Specifically: The execution unit (robot body) is the final execution carrier of actions, such as a robotic arm or mobile robot platform. It receives action sequence instructions from the computing unit and drives mechanical parts to complete physical operations through its underlying controller (such as joint motor controller), realizing real-time closed-loop control of the robot.

[0035] Sensing Unit: Used to collect real-time data on the working environment and its own status. The core includes a vision sensor (such as an RGB camera), responsible for providing the current visual observation image required for motion generation; optional configurations include force / torque sensors, position encoders, etc., for sub-task success / failure determination and closed-loop correction. The sensed data is sent to the computing unit as status input and monitoring data.

[0036] The computing unit (core computing unit) is the core computing device and execution entity that loads and runs the "Robot Long Sequence Control Method Based on Graph Thinking Chain and Dual-Channel Intermediate Product" described in this application. As the execution entity of the method in this application, this unit sequentially runs the following core logic: the graph executor maintains the explicit task state and generates state context information; and generates structured text channel intermediate products (containing...) through the visual-language model (VLM). <obj>and <inst>The field instructions are decoded under logical mask constraints; a Gaussian heatmap-form spatial channel intermediate product is generated through the heatmap decoding head; spatial guidance features are obtained through the spatial feature extractor based on heatmap weighted pooling and encoding; and the final continuous action sequence is generated through the action expert module based on flow matching denoising sampling, using observation, text channel intermediate product and spatial guidance features as conditions, and sent to the execution unit.

[0037] The computing unit can be flexibly deployed on the robot body, nearby edge servers, or cloud servers according to computing resources and latency requirements; the sensing unit and the execution unit (robot body) are usually deployed together to ensure low-latency synchronization of perception and execution.

[0038] Figure 6 This is a flowchart illustrating a long sequence control method for robots based on graph thinking chains and dual channels, according to some embodiments of this application. In some embodiments, the long sequence control method for robots based on graph thinking chains and dual channels can be... Figure 5 The core computing unit in the system shown executes the following steps: S610, based on a predefined task graph template, maintains the explicit state of the current task through the graph executor and outputs state context information.

[0039] To achieve structured management of long-sequence task flows, this application encodes the prior structure of relevant long-sequence tasks into a task graph format: , in This represents the overall structure of the Task Graph. It is a set of template nodes, suitable for representing the stage states in a task flow, such as abstract states like IDLE or GRASPED; This is a set of template edges used to represent the legal transition relationships between states and their corresponding primitives. This is a collection of objects to be operated on, where each element is an abstract object identifier (e.g., A, B, C, etc.) used to dynamically bind specific instances during execution.

[0040] To compress VLM overhead and simplify state determination in long sequences, this application employs a subgraph template with fewer states. For example, in typical multi-object long sequence grasping tasks, a 3-state subgraph template (IDLE) is used for each object. GRASPED), corresponding to two operation primitives: PICK and PLACE:

[0041] in, This indicates a state where the object is not currently held; This indicates that the robot has successfully grasped the object; This indicates that the primitives are being retrieved. Indicates the placement of primitives.

[0042] This template requires only two VLM calls per object (one for PICK and one for PLACE), significantly reducing the number of long sequence inference calls, and delegating state judgments to the Graph Executor instead of relying on the implicit memory of the VLM.

[0043] The graph executor maintains an explicit state at runtime, which is represented by a triplet structure, including the current graph node state, the currently processed object, and the set of objects whose operations have been completed: , in, Indicates time System state variables; Represents the current state of a graph node, belonging to the set. This is used to indicate whether the current stage is the grabbing stage or the placing stage; This represents the object currently being processed or held, whose value belongs to a set. ,in This indicates that no specific object is currently bound to it; Represents a collection of objects that have completed operations, satisfying... This triplet structure can fully characterize the current execution stage, the operation object, and the progress already completed, thus providing an explicit memory carrier for long sequence consistency.

[0044] Based on this, the graph executor defines a deterministic state transition function to update the state at the next time step according to the current state and the result (success or failure) of the subtask execution:

[0045] in, This represents a state transition mapping function; the input parameters include the current state. And the execution result flag (success or failure). Indicates time System state variables.

[0046] The graph executor generates state context tokens for injecting the VLM prefix before each VLM call: <current> 、 <next> 、 <prim> 、 <completed> 、 <remaining>;in <current> / <next> / <prim>Depend on The only certainty, <remaining>Depend on Determined; when (PLACE phase) <obj>Depend on Direct injection avoids repeated generation. This design decouples the "graph traversal decision" from the autoregressive generation of the VLM, transforming the state context from "letting the VLM make a decision" to "state declaration," which also serves as an explicit conditional signal for the HeatmapHead.

[0047] If the monitoring module determines that the operation primitive failed (e.g., abnormal force / displacement after gripper closure, timeout, target not achieved, etc.), the actuator will remain in place. It remains unchanged and triggers the next VLM call. This is because the state context explicitly declares the current state. <prim> / <obj>VLM can autoregressively generate corrective functions based on the latest visual feedback on the same operating primitives. <inst>(e.g., "re-approach", "adjust gripping angle", "decelerate and lower", etc.), thereby achieving closed-loop online correction without the need for additional primitive stage detectors or explicit backtracking edge modeling.

[0048] S620, the state context information, the predefined task graph, the robot's observation information, and the overall task instructions are input into the vision-language model. The vision-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the vision-language model generating the object identifier to be operated through the state context information.

[0049] See Figure 3 This illustrates the generation process of the dual-channel explicit intermediate in this embodiment.

[0050] The method in this application is an improvement on the two-layer architecture of PI0.5.

[0051] Let the robot's observation at time t be... ,in For multi-camera images, This refers to the robot's physical state; the overall task language instructions are: The hierarchical modeling of PI0.5 jointly models the "high-level text output" and the "low-level continuous action chunks," and decomposes them into: , in It is the set of all learnable parameters of the model. The parameter is The conditional probability distribution represented by the strategy model, with symbols Indicates the current decision-making moment. Indicates the time span (horizon) of the action block. Indicates from time to A continuous control sequence, typically corresponding to a length of Action vector sequence, symbol For overall mission instructions, This represents a sequence of high-level text output tokens generated by the model, whose semantics are "a description of the subtasks to be executed in the current stage", such as "pick up the plate".

[0052] To further enhance structural controllability and long sequence stability, this application incorporates the native natural language subtasks of PI0.5. Reparameterize into structured, auditable, low-token subtask prompts: , in, This represents the high-level text output token sequence generated by the model, with subscripts... This indicates that the output of this subtask corresponds to the current decision-making moment. (Symbol) This indicates a structured subtask hint; This represents an ordered structure consisting of two fields. (Field) Select a field for the object to specify the target object or instance identifier to be operated on in this round; field This is the instruction field, used to describe the execution semantic constraints generated for the current scenario, such as the capture location strategy, path offset, or contact method description.

[0053] Given observations General instructions Task Map With state The VLM autoregressive generation distribution is: , in It is the set of all learnable parameters of the model. The parameter is The conditional probability distribution represented by the strategy model. Indicates the first Generate a token. Indicates the first Step on the previously generated token sequence, and These are the current observation, overall mission instructions, mission graph structure, and current state context, respectively.

[0054] This formula represents an autoregressive generation process, where the probability of the current token depends on all previous tokens and the conditional input. In a concrete implementation, <obj>The generation step is subject to the aforementioned constraints on decoding, while <inst>The steps are open-ended, but their conditional distribution explicitly depends on the state context, thus maintaining stage consistency at the semantic level.

[0055] when At that time, VLM needs to generate <obj>Therefore, the graph executor defines the set of legal objects that can be selected in the current state. This set consists of the remaining set of all objects to be operated on, excluding those that have already been operated on.

[0056] in Indicates the current state The set of objects that can be selected is below. For the set of all objects to be operated on, The set of objects for which operations have been completed, symbol Represents a single element in a collection of objects.

[0057] Furthermore, this set of legal objects is transformed into a logarithmic probability mask, which is then applied to the VLM. <obj>In terms of the output distribution of the decoding step, the repeated selection of completed objects is fundamentally eliminated, avoiding the "logical loop / repetitive operation" problem common in long sequences.

[0058] S630, based on the state context information and the last hidden state of the visual-language model, a spatial channel intermediate product is generated through a heatmap decoding head. The spatial channel intermediate product includes a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and the operation target region.

[0059] The intermediate products of the spatial channel in this application are two Gaussian heat maps: , in, Indices for spatial coordinates on the image plane; Heatmap showing the source location This represents a heat map of the target location. This indicates that the heatmap is a real-valued matrix with a resolution of 56×56.

[0060] The semantics of the two heatmaps are derived from the state context. <prim>and <obj>Decision: When <prim>=PICK, Identify the current position of the end effector. Identify the grabbing point area of ​​the target object; when <prim>When =PLACE, Identify the location (approximately the end) of the object being grabbed. Identify the target placement area. The two images, as displayable spatial intermediates, can be overlaid on the camera image at runtime to audit where the model is "looking," thereby reducing the risk of spatial attention drift in long sequences.

[0061] This application adds a lightweight heatmap decoding head after the visual-language model. After the visual-language model generates subtask prompts, the heatmap decoding head reads the features of its last hidden layer: , in, This represents the hidden states output by the last layer of the multimodal Transformer. The current sequence length (including visual tokens, task instruction tokens, graph definition tokens, state context tokens, and generated tokens). <obj>and <inst>wait), To hide the feature dimension.

[0062] The prefix contains state context tokens (especially) <prim> 、 <obj>Therefore, the hidden state at the corresponding position carries explicit conditional information. Two sets of learnable query vectors (e.g., 8×8=64 query vectors) are used to perform cross-attention computation on the above hidden layer features, yielding low-resolution implicit representations of the source and target spaces, respectively. , in This represents the cross-attention operation; and They are respectively from Key-value representations obtained from linear mapping; and These are the source learnable query matrix and the target learnable query matrix, respectively (e.g., set to...). (query vectors).

[0063] Cross attention can directly focus on <prim>and <obj>The corresponding hidden state is used to obtain a stable "primitive-object" conditionalization. Then... , Dimensional reshaping Then, after transpose convolution / upsampling, it is decoded to... Output , .

[0064] S640, based on the first Gaussian heatmap and the second Gaussian heatmap, a spatial guidance feature is fused to generate a spatial relationship representing the spatial relationship from the operation source region to the operation target region.

[0065] Referring to Figure 4, it shows the network structure for spatial feature extraction and action generation in this embodiment.

[0066] To enable spatial intermediate products to enter the motion expert in a low-bandwidth manner, this application designs a spatial feature extractor. , Align with visual feature maps and extract as a single spatial token .

[0067] Let the visual feature map be aligned with the heatmap space. Then, the source / target visual vector is obtained by weighted pooling using the heatmap as the weight: , , in, and These are the visual vectors of the source region and the visual vector of the target region. Image plane coordinate index; A C-dimensional visual feature vector representing the corresponding spatial location; , These represent the thermal weights for the corresponding locations. The above operation is equivalent to performing a spatially weighted summation on the visual feature map, thereby extracting a global vector representation focused on the source and target regions.

[0068] Furthermore, the displacement direction and amplitude are explicitly expressed through differential coding: , in, Represents the spatial difference token. This represents a vector concatenation operation; Represents the explicit difference vector from the source to the target; Represents a multilayer perceptron mapping; This refers to the dimension of the final spatial token. The design compresses the spatial relationship of "from where to where" into a single low-dimensional token, which is used to guide the generation of subsequent actions.

[0069] Then This is added as an extra token to the action expert's policy prefix. This design forces "spatial attention" into the action generation process as a low-dimensional token, suppressing action distortion and drift caused by global background interference in long sequences.

[0070] S650, the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot are input to the motion expert module, and the motion expert module generates a continuous motion sequence for controlling the robot.

[0071] This application adopts the π0.5 action expert algorithm, uses flow matching to model the continuous action distribution, and employs 10 denoising steps during inference to output continuous action chunks. The differences from the baseline are: The requirements for a motion expert are as follows: Enhanced to To simultaneously consume intermediate products from both text and spatial channels: , in For parameters Action expert model; This is a hint for the current subtask; Indicates time Spatial difference feature token; Indicates from time to A continuous control sequence.

[0072] The corresponding flow matching algorithm loss form is: , in Represents the relationship between random variables (Time interpolation coefficients) and The expectation of (Gaussian noise); This represents the noise interpolation action within the framework of the stream matching algorithm. This represents the flow vector field predicted by motion experts; It is the L2 norm. This loss is used to train the model to predict the direction of evolution from noisy states to the true action distribution, and its sign is... This is based on current observations.

[0073] During deployment, the conditional distribution is denoised 10 times to obtain a continuous action sequence, which is then sent to the controller for execution.

[0074] To avoid shortcut learning (e.g., attention leakage when discrete FAST action supervision is present), this application maintains the isolation of the two-channel paths: the heatmap decoder query vector only reads the VLM hidden state and does not write back the VLM token sequence; the action expert only reads prefix tokens (state / instruction / spatial difference / visual) and does not pay attention to FAST (Fast Action Sequence) action tokens. This strategy ensures that the orthogonal division of labor, where "the text channel is responsible for what to do and how to do it, and the spatial channel is responsible for where to do it," can be structurally maintained, and that intermediate products can be stably displayed and audited.

[0075] Figure 7 This is a schematic diagram of a long sequence control device for a robot based on graph thinking chain and dual channels, according to some embodiments of this application. Figure 7 As shown, the robot long sequence control device 700 based on graph thinking chain and dual-channel includes a maintenance module 710, a first generation module 720, a second generation module 730, an extraction module 740, and a third generation module 750. The robot long sequence control function based on graph thinking chain and dual-channel can be... Figure 6 The core computing unit in the system shown executes the commands. Wherein: The maintenance module 710 is used to maintain the explicit state of the current task based on a predefined task graph template through the graph executor and output the state context information. The first generation module 720 is used to input the state context information, the predefined task graph, the robot's observation information, and the overall task instructions into the vision-language model. The vision-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the vision-language model generating the object identifier to be operated through the state context information. The second generation module 730 is used to generate spatial channel intermediate products through a heatmap decoding head based on the state context information and the last hidden state of the visual-language model. The spatial channel intermediate products include a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and the operation target region. Extraction module 740 is used to generate a spatial guidance feature that represents the spatial relationship from the operation source region to the operation target region based on the first Gaussian heat map and the second Gaussian heat map; The third generation module 750 is used to input the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot into the motion expert module, so that the motion expert module can generate a continuous motion sequence for controlling the robot.

[0076] In summary, the robot long sequence control method and apparatus based on graph thinking chain and dual-channel proposed in the embodiments of this application firstly, by introducing a graph actuator to maintain explicit state and drive a constrained decoding mechanism, the visual-language model is constrained to select objects to be operated on only from unfinished objects. This transforms the logical decision of object selection in long sequences into deterministic constraints imposed by an external state machine, thereby eliminating logical inversion and repetitive operations, and achieving reliable and stable tracking of the task causal chain. Secondly, by constructing a text channel intermediate product bound to the graph state and a closed-loop correction mechanism, when a subtask is determined to have failed based on physical signals, the graph actuator maintains its state and triggers the visual-language model to regenerate scene-adaptive instructions. This allows the system to achieve adaptive anomaly recovery without relying on susceptible external detectors, thus obtaining highly robust built-in closed-loop correction capabilities and significantly reducing the risk of task interruption. Simultaneously, by designing a spatial channel intermediate product, a Gaussian heatmap is generated by a heatmap decoding head and encoded into compact spatial guidance features through a spatial feature extraction step, which is then injected into the action expert. This mechanism effectively solves the error accumulation problem caused by the lack of precise spatial guidance by providing pixel-level precision heatmap guidance, focusing motion generation on key operational areas and improving the spatial accuracy of long sequence operations. Ultimately, these effects are achieved through a minimalist dual-channel intermediate product and a lightweight heatmap decoder. This design avoids the enormous overhead of generating high-dimensional image chains without significantly increasing the computational burden on the backbone network, thus supporting high-frequency generation of continuous actions by motion experts while maintaining extremely low computational and parameter overhead. This effectively resolves the contradiction between high computational costs and the difficulty in balancing real-time requirements in existing technologies. Through the deep coupling of the above technical features, this application achieves systematic improvements in four aspects: logical reliability, system robustness, spatial accuracy, and computational efficiency, providing an innovative high-performance solution for long sequence complex robotic operations.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0078] Although the foregoing embodiments are described within the general context of computer systems running in conjunction with operating systems and applications, those skilled in the art will recognize that other implementations can also be performed by incorporating other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0079] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.< / obj> < / prim> < / obj> < / prim> < / inst> < / obj> < / prim> < / prim> < / obj> < / prim> < / obj> < / obj> < / inst> < / obj> < / inst> < / obj> < / prim> < / obj> < / remaining> < / prim> < / next> < / current> < / remaining> < / completed> < / prim> < / next> < / current> < / inst> < / obj> < / inst> < / obj> < / inst> < / obj> < / remaining> < / completed> < / prim> < / next> < / current>

Claims

1. A long sequence control method for robots based on graph thinking chains and dual channels, characterized in that, include: Based on a predefined task graph template, the graph executor maintains the explicit state of the current task and outputs the state context information. The state context information, the predefined task graph, the robot's observation information, and the overall task instructions are input into the visual-language model. The visual-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the visual-language model generating the object identifier to be operated through the state context information. Based on the state context information and the last hidden state of the visual-language model, a spatial channel intermediate product is generated through a heatmap decoding head. The spatial channel intermediate product includes a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and operation target region. Based on the first Gaussian heatmap and the second Gaussian heatmap, a spatial guidance feature is generated to represent the spatial relationship from the operation source region to the operation target region. The intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot are input into the motion expert module, which then generates a continuous sequence of actions to control the robot.

2. The method according to claim 1, characterized in that, The graph executor is configured to perform the following operations: Maintain the explicit state including the current node state, the identifier of the currently operated object, and the set of objects whose operations have been completed; The robot's single operation primitive executed based on the current text channel intermediate product and the spatial channel intermediate product is determined as the current subtask, wherein the operation primitive includes grasping and placing; Based on the success or failure result of the current subtask, the explicit state is updated according to a predefined deterministic state transition function T, i.e. ,in, For a moment The explicit state. For a moment The explicit state; The output is a sequence of tokens as state context information, wherein the token sequence includes <current> 、 <next> 、 <prim> 、 <completed> 、 <remaining> At least one of them, and used to inject into the vision-language model.< / remaining> < / completed> < / prim> < / next> < / current> 3. The method according to claim 2, characterized in that: The graph executor transforms the set of objects that have completed operations into a logical mask decoding constraint, which is applied to the decoding process of the object identifier to be operated on in the vision-language model, so that when generating the object identifier to be operated on, selection can only be made from outside the set of objects that have completed operations.

4. The method according to claim 2, characterized in that, The method further includes: When the current subtask is determined to have failed, the graph executor maintains its current state and triggers the vision-language model to regenerate scene-adaptive semantic instructions based on the latest observation information. When the current subtask is determined to be successful, the graph executor completes the update and transition of the explicit state according to the deterministic state transition function.

5. The method according to claim 4, characterized in that: The determination is based on at least one of the following conditions: abnormal force after the gripper closes, abnormal displacement after the gripper closes, operation timeout, or target not achieved.

6. The method according to claim 1 or 4, characterized in that: The intermediate product of the text channel is structured prompt words.<OBJ,INST> The object to be operated is identified <obj>The scene-adaptive semantic instructions are abstract object identifiers. <inst> Includes at least one of the following: contact surface, force, path constraint, or correction strategy.< / inst> < / obj> 7. The method according to claim 2, characterized in that: The heatmap decoding head is a new lightweight module added to the visual-language model, and the number of parameters is less than the preset threshold of the visual-language model backbone network. The semantics of the spatial channel intermediate product generated by the heatmap decoding head are jointly determined by the primitive type and object identifier in the state context information. When the primitive type is grasping, the first Gaussian heatmap represents the current position of the end effector and the second Gaussian heatmap represents the grasping point area of ​​the target object. When the primitive type is placement, the first Gaussian heatmap represents the position of the grasped object and the second Gaussian heatmap represents the position of the target placement area. The dimensions of the first Gaussian heatmap and the second Gaussian heatmap are both 56×56. The process by which the heatmap decoding head generates the aforementioned Gaussian heatmap is as follows: using two sets of learnable query vectors, cross-attention calculation is performed on the last layer of the hidden state of the visual-language model, which contains the state context information, to obtain the low-resolution hidden representation in the source space and the low-resolution hidden representation in the target space. Then, the low-resolution hidden representation in the source space and the low-resolution hidden representation in the target space are adjusted to a low-resolution two-dimensional matrix format, and then decoded to 56×56 through transposed convolution or upsampling, finally generating the first Gaussian heatmap and the second Gaussian heatmap. The text channel and the spatial channel maintain orthogonal division of labor by adopting an attention isolation strategy. That is, the query vector of the heatmap decoder only reads the hidden state of the visual-language model and does not write back the token sequence of the visual-language model. In addition, the action expert module only reads the prefix token and does not directly perform attention calculation on the tokens of fast actions.

8. The method according to claim 7, characterized in that, The step of fusing the first Gaussian heatmap and the second Gaussian heatmap to generate a spatial guidance feature representing the spatial relationship from the operation source region to the operation target region includes: Using the first Gaussian heatmap respectively and the second Gaussian heatmap As weights, the visual feature maps aligned with the Gaussian heatmap space are... Perform weighted pooling to obtain the source visual vector. and target visual vector The formula is: ， ， in, These are the pixel coordinates of the feature map. Indicates that it is located at The feature vector of the location; For the source visual vector The target visual vector and the difference between the two The concatenated combined vector is encoded to obtain the spatial guidance feature. The formula is: ， in, This represents a multilayer perceptron. This indicates a vector concatenation operation.

9. The method according to claim 1, characterized in that, The step of inputting the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot into the motion expert module, and having the motion expert module generate a continuous sequence of actions for controlling the robot, includes: The motion expert module uses the robot's current observation information, the intermediate products of the text channel, and the spatial guidance features. As a condition, the distribution of conditional actions is modeled by flow matching, and a continuous sequence of actions for controlling the robot is generated through multi-step denoising sampling.

10. A robot long sequence control device based on graph thinking chain and dual-channel, characterized in that, include: The maintenance module is used to maintain the explicit state of the current task based on a predefined task graph template through the graph executor, and output the state context information. The first generation module is used to input the state context information, the predefined task graph, the robot's observation information, and the overall task instructions into the vision-language model. The vision-language model generates a text channel intermediate product based on this. The text channel intermediate product includes an object identifier to be operated and a scene-adaptive semantic instruction generated based on the observation information. The explicit state imposes constraints on the process of the vision-language model generating the object identifier to be operated through the state context information. The second generation module is used to generate spatial channel intermediate products through a heatmap decoding head based on the state context information and the last hidden state of the visual-language model. The spatial channel intermediate products include a first Gaussian heatmap and a second Gaussian heatmap that respectively represent the current operation source region and the operation target region. The extraction module is used to fuse the first Gaussian heatmap and the second Gaussian heatmap to generate a spatial guidance feature that represents the spatial relationship from the operation source region to the operation target region. The third generation module is used to input the intermediate product of the text channel, the spatial guidance features, and the current observation information of the robot into the motion expert module, which then generates a continuous motion sequence for controlling the robot.

Images