A long-range mobile operation control method integrating skill-state map and closed-loop execution

Abstract

The application discloses a long-range mobile operation control method integrating skill-state graph and closed-loop execution, constructs a layered skill library decoupling bottom electromechanical control and high-level semantics, and a topological perception skill-state graph; through strictly limiting semantic-level planning of a visual language model in a topologically feasible subgraph, and combining multi-step forward feasibility verification of cumulative changes of a robot ontology state, high-level task logic disassembly and accurate alignment of bottom operation generation under non-explicit scene mapping are realized; in the robot execution phase, a closed-loop execution mechanism decoupling physical safety boundaries and semantic reasoning logic is introduced, and light self-state monitoring is performed by using high-frequency ontology sensation feedback; when detecting unexpected deviation of a physical trajectory and planning logic, a dynamic semantic re-planning process limited to the skill-state graph is triggered. The application significantly improves the planning success rate and physical execution robustness of the robot in long-range tasks.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary technical fields of embodied intelligence, artificial intelligence, and robot multimodal perception and autonomous control, specifically relating to a long-range mobile operation control method that integrates skill-state diagrams and closed-loop execution. Background Technology

[0002] With the continuous integration of embodied intelligence technology and large-scale model technology, modern robots are gradually moving from structured, fixed assembly line operations to highly unstructured, complex, and dynamic indoor living and working environments. In these scenarios, long-range mobility operations have become the core standard for evaluating the autonomous capabilities of intelligent agents. Long-range mobility operations require robots to not only possess precise low-level electromechanical control capabilities, but also to be able to parse highly abstract human natural language instructions into long sequences of physical interaction actions, lasting dozens of steps and involving multiple scene roaming navigations, target multimodal recognition, spatial obstacle avoidance, and fine and compliant grasping and placement. This process highly and tightly couples the robot's multimodal visual perception, high-order semantic logic decision tree generation, and low-order complex electromechanical dynamics control, and the robot must maintain operational stability and consistency over long periods of time under continuously changing physical environmental constraints.

[0003] Traditional paradigms for solving such long-running operational problems have gone through several key stages of development, but they have all revealed irreconcilable systemic flaws and bottlenecks when dealing with complex tasks:

[0004] (1) Symbolic methods based on classic task and motion planning and explicit 3D mapping. Representative systems in this stage include SayPlan and classic robot planning frameworks that rely heavily on explicit 3D scene graphs. Such systems rely heavily on pre-extracting the subtle features of the physical environment and transforming them into rigid explicit topological maps, or using PDDL (Planning Domain Definition Language) expressions for symbolic representation. This method based on strict discrete logic reasoning performs reasonably well in terms of theoretical coherence and logical rigor, but in real-world indoor environments that are unknown or frequently dynamically changing, maintaining a large and constantly demanding explicit 3D semantic map faces an extremely high data throughput disaster. At the same time, explicit symbolic modeling lacks effective representation and generalization capabilities for fine-grained visual contexts, making it unable to flexibly and dynamically respond to undefined objects outside the training library or obstacles with temporarily changed layouts, resulting in its extreme fragility in real-world applications.

[0005] (2) Imitation learning and end-to-end vision-language-action model methods based on large-scale real-world data pre-training. In recent years, researchers have developed a large number of basic policy models with huge parameter scales, such as RT-1, RT-2, VIMA, RoboAgent, Octo, OpenVLA, and various diffusion-based methods, such as DP-VLA and DP3. These networks directly map visual images to joint torques or end-effector pose increments without the need for intermediate representations by fine-tuning or full-scale training on massive amounts of cross-morphological robot operation data. This implicit behavior cloning technology has shown extremely strong single instruction understanding ability and extremely flexible short-range fine visual motion control level. However, when faced with long-range operations that often involve splicing together dozens of sub-tasks, these end-to-end black-box policy networks are prone to serious problems of "cascade error". Due to the lack of explicit step memory and logical module division, once a small displacement error occurs in the preceding action (such as navigating to the wrong coordinate system or a deviation of a few millimeters in the grasping posture), this multi-step cumulative error will be gradually amplified by the huge neural network. More seriously, a single network has difficulty performing logical self-correction within the latent space, which ultimately leads to a significant decrease in the success rate of long-term tasks in the latter half of the execution.

[0006] (3) Zero-shot decomposition methods for open-loop tasks driven by large-scale language models (LLM) and visual language models (VLM). To address the problem of excessively long task chains, recent research, such as DeCo, Inner Monologue, VoxPoser, OWMM-Agent, and ReAct architectures, attempts to introduce the powerful prior knowledge of large models into embodied control, endowing the system with zero-shot semantic reasoning capabilities. However, existing VLM-driven control schemes are often in a state of "open-loop execution" or severely lacking physical constraints. When VLM generates planning sequences, although it can write action chains that seem to conform to human semantic common sense (e.g., go to the sink first, then get the plate), it often generates illusions out of thin air due to the lack of memory tracking of the robot's dynamic ontology physical state and understanding of the underlying kinematic constraints. For example, VLM may instruct the robot to perform another grasping action when its gripper is already occupied, violating physical common sense, i.e., lacking consideration of the compatibility of accumulated states. Such semantically reasonable but physically difficult or unexecutable transitional behaviors often lead to deadlock in the middle of execution.

[0007] (4) Attempts at closed-loop error correction techniques with high latency and significant resource waste. A small number of cutting-edge papers have recognized the importance of closed-loop monitoring and have attempted to introduce closed-loop visual language monitoring, such as Code-as-Monitor, to identify operational errors and attempt to correct them. However, such frameworks typically require frequent requests to the VLM or large model for visual confirmation or environmental reassessment at every minor control frequency in the action sequence. As is well known, calling a large language model with hundreds of billions of parameters for multimodal reasoning will consume a great deal of computing power and lexical resources, while also facing high network requests or computational latency. Such solutions are difficult to meet the high-frequency real-time response requirements of robot edge computing systems with limited resources in reality. At the moment of physical slippage, resistance overload, or collision, the slow semantic recovery of the large model often misses the best window for safety intervention and error correction, ultimately leading to task termination or failure.

[0008] In summary, there is an urgent need for an innovative architecture in the field of embodied intelligence and long-range robotics operations. This architecture must meet the following requirements: it should not rely on heavy and complex global explicit geometry construction, it should effectively curb the illusion problem of large models from the underlying mechanism, and it should achieve the organic unity of ultra-fast error correction of underlying physical execution and high-level dynamic semantic replanning with extremely low computational overhead. Summary of the Invention

[0009] Purpose of the invention: This invention aims to solve the problem of control failure of robots in long-term, cross-scene, multi-step home or industrial tasks caused by model illusion or error accumulation, and proposes a long-range mobile operation control method that integrates skill-state diagram and closed-loop execution.

[0010] Technical solution: The long-range mobile operation control method integrating skill-state diagram and closed-loop execution described in this invention includes the following steps:

[0011] (1) Construction of multi-level control interface and embodied state abstraction: In the robot control system, a hierarchical skill library and a topology-aware skill-state graph are constructed; the hierarchical skill library includes behavioral skills that directly interact with the underlying electromechanical hardware and semantic skills with contextual semantic representation; the skill-state graph uses the semantic skills as directed graph nodes and the topologically and physically feasible transfer paths between skills as directed edges, providing the embodied intelligent system with physical common sense abstraction and execution logic prior constraints in a non-explicit graphing environment;

[0012] (2) Semantic planning of visual language model with topology awareness: Obtain the multimodal visual observation state of the robot at the current timestamp. With long-range natural language task instructions Based on stateless topological subgraphs, semantic-level multi-step task decomposition is performed using a visual language model. This allows for the search and generation of preliminary candidate skill sequences that conform to the topological connectivity of the environment within the legally connected domains of the graph network. ;

[0013] (3) Feasibility verification of state-driven execution chain: Using the preceding states and state change functions in the skill-state diagram, the global state consistency of the candidate skill sequence in the latent space is verified based on physical common sense; by recursively applying the state change function along the candidate action chain to predict the future execution state of the robot body, if the verification is successful, the final executable semantic skill sequence is output. Otherwise, the error will be reported to the visual language model and the process will be returned to step (2) for correction and replanning;

[0014] (4) Dual-track closed-loop monitoring based on multimodal heterogeneous feedback: The robot's underlying execution controller executes executable semantic skill sequences. Throughout the entire lifecycle, the robot chassis and robotic arm's proprioceptive feedback data and visual sensor observation data are collected in real time to perform dual-layer state verification in both physical feedback and visual semantic dimensions, so as to continuously evaluate whether the actual physical result of each skill execution strictly matches the expected output boundary of the state change function.

[0015] (5) Dynamic semantic replanning under graph constraints: If the multimodal dual-track monitoring in step (4) detects a physical operation error or semantic result logic deviation that exceeds the target threshold, the current predetermined open-loop execution chain is immediately terminated when either of the two errors is detected, and the actual physical attributes observed by the robot control system at the moment before the execution error occurs are extracted as the latest known physical state. It sets this as the initial anchor state, and triggers the replanning mechanism of the visual language model in the directed search space of the skill-state graph to generate a corrected skill sequence that seamlessly recovers from the current error state to the original task objective path. Then, it returns control to the underlying layer to continue executing the task.

[0016] Furthermore, the behavioral-level skills mentioned in step (1) are low-level basic control primitives that do not depend on specific semantics of the environment, including joint-level impedance control, compliant force control, and end-effector Cartesian kinematic inverse programming based on reinforcement learning; the semantic-level skills are high-level abstraction capabilities with explicit environmental interaction attributes and temporal pre-dependent conditions, and each semantic-level skill is formed by the sequential combination of several behavioral-level skills or by instantiation and mapping in a specific parameter space through an end-to-end policy network.

[0017] Furthermore, the skill-state graph described in step (1) is a directed graph topology with highly structured properties. , where nodes Directed edges represent specific semantic-level skills. Represents the reasonable transfer logic of local execution within physical space; and each node A set of preceding states must be attached. and state change function Among them, the set of previous states Defined as a tuple of multidimensional physical conditions that a robot system must satisfy before safely executing the skill at this node; the tuple includes the relative spatial position of the robot chassis. The end effector of the left robotic arm is occupied. and the right robotic arm end effector is occupied. State change function This refers to the deterministic mathematical mapping and state value changes that occur to the multidimensional physical state of the robot body after the node skill has been executed in the physical world.

[0018] Furthermore, the state change function The operators for each dimension are defined as follows: For chassis movement skills, the spatial mapping function... Execution space transformation operator This transforms the robot's basic physical coordinate system from environmental anchor point A to environmental anchor point B; and for the robotic arm's object manipulation skills, a gripper mapping function is used. and Applying incremental combination operators To transfer the end effector from an idle state to hold a specific target object. The state, or the application release operator. To remove the gripper from holding a specific target object The state is reset to idle.

[0019] Furthermore, the stateless topological subgraph described in step (2) It uses graph theory dimensionality reduction algorithms to extract the complete skill-state diagram. Explicitly extract the set of preceding states With state change function After attributes, only the abstract skeleton network structure of the graph nodes' connectivity and adjacency relationships is retained.

[0020] Furthermore, the visual language model described in step (2) generates an initial candidate skill sequence. At that time, its internal multi-step reasoning and the growth path of the search tree are constrained by... Within the legally directed connected domain.

[0021] Furthermore, the specific logical operation process of the global state consistency verification based on physical common sense in step (3) is as follows: the system reads the robot sensor calibration initial physical state as... Along the candidate skill sequence generated by the visual language model The state transition equation is calculated iteratively. Predict the physical cumulative effect at each step in the sequence; for any step in the sequence... A progressive skill, the system determines the Boolean validation function. The return value is whether it matches the absolute prerequisite state set of the next skill to be executed in the sequence. Logical exclusion or physical conflict occurs.

[0022] Furthermore, the dual-track closed-loop monitoring of multimodal heterogeneous feedback described in step (4) includes two parallel monitoring processes decoupled on the physical time axis, including high-frequency body state monitoring and low-frequency semantic verification monitoring. The high-frequency body state monitoring reads the data streams of the robot's multi-axis joint encoder and end-feed gripper tactile sensor by directly mounting it on the industrial communication bus, verifying the physical execution success rate of the motion primitives at a frequency much higher than the visual control cycle. At the low-frequency time node when each semantic-level skill node claims to have been completed, the system asynchronously calls the visual language big model to observe the multi-view images and state change functions of the current environment from multiple perspectives. The theoretically expected environmental evolution parameters are compared and evaluated to determine whether abnormal object slippage, navigation position deviation, or long-term operational semantic errors have occurred.

[0023] Furthermore, after the dynamic semantic replanning under the graph constraint environment described in step (5) is initiated, the system memory stack will not be cleared and the entire long-range task chain will not be reset from the beginning. Instead, the abnormal physical states currently observed will be processed through the perception layer. The skill-state graph network input into memory performs local relocation of graph nodes. Utilizing a large visual language model, it searches for a path within a strictly constrained topological graph space that can reconstruct damaged states. Reconnect to the shortest corrected recovery path to the foremost node of the remaining unexecuted target subtasks.

[0024] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. Suppressing the illusion phenomenon of VLM and improving the success rate of long-term tasks: This invention effectively restricts transitional behaviors that do not conform to physical logic by introducing closed-loop iterative verification of skill-state diagram constraints; in long-term tasks in both simulated and real environments, this method can significantly improve the success rate of the final task compared with existing baseline methods.

[0026] 2. Optimize computational reasoning efficiency and save lexical resources: This invention constrains the reasoning search range of VLM to the pre-extracted topological subgraph, reducing the model's invalid search and cognitive burden in the large state space; quantitative evaluation shows that compared with the complete skill-state graph planning method without pre-extraction, this invention can effectively shorten the planning reasoning delay and significantly reduce the consumption of total lexical units and thinking part lexical units, which better meets the robot's requirements for deployment response latency;

[0027] 3. No explicit semantic mapping deployment and strong adaptability to dynamic scenes: This invention does not require the construction and maintenance of a high-precision global 3D map or a heavy symbolic world model. Instead, it relies on a layered skill library and skill-state graph to achieve lightweight embodied state alignment. This strategy reduces the system's dependence on static environmental structures, enabling the control system to adapt more flexibly to indoor operation scenarios with layout changes or local dynamic elements.

[0028] 4. Enhanced execution robustness and automated fault recovery capability: This invention constructs a closed-loop execution mechanism through high-frequency proprioceptive state monitoring combined with dynamic replanning based on state diagrams. When the system encounters physical anomalies or slippage during stages such as grasping, it can detect execution errors in a timely manner without human intervention and autonomously generate error correction strategies. This mechanism helps the system to promptly discover and accurately locate local execution errors and autonomously invoke recovery strategies online, thereby significantly enhancing the system's anti-interference capability and operational continuity in long-term tasks. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention;

[0030] Figure 2 This diagram shows a comparison of the system integration architecture of the long-range operation method of this invention with the logic of existing conventional VLM technology.

[0031] Figure 3 The diagrams above illustrate the configurations and hardware matrix layouts of various real multimodal physical experiments involved in this invention; wherein, (a) is a top view of the scene layout; (b) is a diagram of the real scene setup; (c) is a diagram of the robot hardware configuration platform; and (d) is a diagram of the objects and furniture props involved in the experiment.

[0032] Figure 4 The following are experimental result analysis diagrams related to the present invention; wherein, (a) is a comparison diagram of the success rate of the method of the present invention and existing methods; (b) is a detailed execution status diagram of 10 real machine experimental trajectories; and (c) is a qualitative visualization diagram of the successful execution of the trajectory of the present invention. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings.

[0034] like Figure 1 As shown, this invention proposes a long-range mobile operation control method integrating skill-state diagrams and closed-loop execution. After acquiring heterogeneous data input from multi-source, multi-modal sensors, it first undergoes high-level macroscopic plan decomposition based on the rigid mathematical constraints of the skill-state diagram. Then, a proprioceptor monitor sensitively identifies minute physical deviations. Finally, the error parameters are passed to the engine for rigorous hardware and software interaction algorithm path panoramic mapping with constrained semantic replanning of graph nodes. The specific implementation process is as follows:

[0035] Step 1: Construction of Multi-Level Control Interfaces and Embodied State Abstraction: A hierarchical skill base and skill-state diagram are constructed within the robot control system to provide physical common-sense constraints for high-level reasoning. The specific implementation process is as follows:

[0036] Skill layering and decoupling: Decoupling complex operational domains into "behavioral skills" that do not depend on specific semantics and "semantic skills" that have clear environmental context.

[0037] Skill-State Graph Modeling: Constructing a Lightweight Directed Topology Graph Structure Among them, nodes A single semantic-level skill is uniquely associated with a directed edge. This represents the logical transfer of local physical execution between skills. To prevent illusions, for each node... Forcefully bind two core constraint properties:

[0038] a) Pre-state dependency Defined as the multidimensional safe state that the system must be in before executing the skill at this node. For a dual-arm mobile platform, it is simplified as a tensor triple: Among them, variables Represents the coordinates of the macroscopic topological anchor point in the environment where the robot chassis is currently docked; variables and Record and constrain the current physical load state of the end grippers of the robot's left and right robotic arms (e.g., whether they are "idle" or "holding a target object").

[0039] b) State Variation Function Defined as a deterministic change in the system state after the skill is executed. In the specific operator implementation, the spatial displacement function Corresponding trigger Calculate operators to refresh the robot's environmental anchor points; gripper state function and Corresponding trigger increment operator (Convert the gripper mark to lock onto the target object) or release operator (Clear the holding status to zero).

[0040] Step 2, topology-aware semantic planning for visual language models: This step suppresses unconstrained open-ended generation of visual language models, restricting their task decomposition process to within legally connected boundaries. The specific implementation is as follows:

[0041] Dimensionality Reduction Topology Extraction: From the Full Skill-State Graph Dynamically stripping complex constraint attribute matrices and A stateless topological subgraph containing only the skeleton of valid connectivity relationships is extracted. .

[0042] Graph-guided target decomposition: extracting multimodal visual observation vectors With parsed human long-range text instructions The input is synchronously fed into the Visual Language Model (VLM). By designing the system interface, the VLM is constrained to only traverse the topology graph. Perform an adjacency step search on the valid directed edges, and output the initial candidate skill execution sequence that only meets the graph connectivity requirements. .

[0043] Step 3: Verification of the feasibility of the state-driven execution chain: In order to avoid contradictions caused by accumulated errors in multi-step operations, global physical coherence must be confirmed through a sandbox before the underlying driver is activated.

[0044] Recursive state evolution simulation: intercepting the candidate sequences generated in the above steps Read the current real sensor feedback to obtain the initial physical background state of the system. Then, the state evolution mapping is invoked in the system's hidden space. Perform node-by-node mathematical recursion calculations for the sequence Predict the entity's occupancy status after each hypothetical timestamp.

[0045] Contradictory logic blocking determination: For any step advancing in the simulated sequence... Step-by-step action: The system performs a hard check and calculates the return value of the Boolean probe function.

[0046] The mechanism continuously monitors the hand occupancy status and physical spatial pose resulting from the previous simulation. Does it logically conflict with the absolute activation prerequisites required for the next skill to be issued immediately (e.g., "forcibly issuing a grabbing command again before the left claw is cleared")? Any sequence that triggers FALSE will be automatically rejected and feedback will be sent to the VLM for bias correction until a physically safe final task queue is output. .

[0047] Step 4: Dual-track closed-loop monitoring based on multimodal heterogeneous feedback: Once the safety sequence... The system begins driving the underlying electromechanical servo network, employing an asynchronous, decoupled mode to capture unexpected runtime disturbances.

[0048] High-frequency proprioception closed-loop: During the polling cycle, the output torque of each joint reducer, the difference between the end-effector Gaussian micro-motion friction data and the drive encoder value are directly read from the industrial bus. This link exists independently of visual recognition and is specifically used to determine things like "execution". "Unexpected slippage of objects due to sudden loss of friction during command" is a basic type of hard physical accident.

[0049] Low-frequency visual macroscopic verification: Only during the low-frequency intervals of sequence node iterations, VLM is scheduled to capture high-resolution environmental snapshots of the hand-eye collaborative panoramic matrix, and macroscopic semantic alignment verification is performed between the expected state of the theoretical model and the multi-pose state of the real scene to determine whether long-term motion derailment has occurred.

[0050] Step 5, Dynamic Semantic Replanning in a Graph-Constrained Environment: If an operation error is detected in Step 4, this step will repair the broken chain at minimal cost instead of resetting the global progress.

[0051] Capture and update the incident state anchor point: Immediately acquire the messy physical state data of the underlying bus at the moment the exception was triggered, take over it, and set it as a new independent node in the current system. .

[0052] Local recovery path planning within a limited graph domain: Instead of reorganizing the entire original task objective sequence, a targeted VLM is invoked, and a heuristic algorithm is used to plan a locally corrected subsequence within the original skill-state graph space containing the recovery skill. This subsequence aims to recover the currently deviated from the damaged state as quickly as possible. The status is restored and docked to the next available safe startup node that has not yet been executed in the original plan, re-merging and seamlessly closing the loop to continue the original interrupted work sequence task flow.

[0053] The differences between this invention and previous work are as follows: Figure 2 As shown, this invention highlights the design logic and information flow of building a robust and safe foundation by using three unique defense modules: closed-loop verification, state comparison, and graph-guided dynamic replanning, compared to the serious "VLM illusion" (i.e., missing sub-steps or logical misalignments at the semantic level, and control failure and derailment at the action level due to open-loop execution) that exist in existing open-loop technologies.

[0054] The experimental setup of the real robot system of this invention is as follows: Figure 3As shown, (a) is a top view of the scene layout; (b) is a diagram of the actual scene setup; (c) is a diagram of the robot hardware configuration platform; and (d) is a diagram of the objects and furniture props involved in the experiment. The diagrams visually demonstrate the comprehensive configuration topology of the multi-degree-of-freedom dual-arm collaborative mobile electromechanical hardware platform used to verify the present invention, as well as the spatial suspension and binding arrangement scheme of the surround multimodal audiovisual sensors. The diagrams also cover typical test interaction objects such as storage spaces and operating tables in home and industrial miniature environments, as well as scene replication layout examples of complex physical boundary obstacles.

[0055] Experimental results are as follows Figure 4 As shown, the quantitative and qualitative evaluation results of the method of this invention on a real-world robotic platform are comprehensively demonstrated. Overall, the system effectively mitigates the error accumulation problem in long-term tasks through a skill-state diagram-based planning mechanism and a closed-loop execution strategy. Figure 4 As shown in (a), in the longest-running test comprising 17 subtasks, the method of this invention maintained a significantly higher success rate in long-running subtasks. Baseline methods showed limitations: the end-to-end ACT failed momentarily due to covariate shift caused by a lack of modular decomposition; DeCo*, lacking state verification and recovery strategies, saw its success rate drop to 0% at subtask 7; even the manually defined task order (GT) with perfect initial planning degraded to 0% at step 16 due to the inability to recover from physical anomalies during open-loop execution. In contrast, the method of this invention not only achieved a high success rate in initial planning but also ultimately achieved a cumulative success rate of 60%. Figure 4 Figure (b) illustrates the execution details at the trajectory level, revealing that the high survival rate of the method of this invention is mainly due to proactive closed-loop error correction, rather than simply relying on a flawless execution process. When encountering physical anomalies, the system's replanning module can accurately detect errors and synthesize local recovery actions, only leading to final failure when encountering extreme unrecoverable states such as objects falling out of the reachable workspace. Figure 4 (c) provides a step-by-step demonstration of a long-duration task, highlighting the system's adaptive replanning capability in response to unexpected events. When unforeseen errors such as grasping failure occur in subtask 7, the system can seamlessly trigger replanning (such as the robotic arm returning to its original position, readjusting the chassis, and re-grabbing), successfully recovering from the failure and smoothly completing the subsequent long-duration task.

[0056] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A long-range mobile operation control method integrating skill-state diagrams and closed-loop execution, characterized in that, Includes the following steps: (1) Construction of multi-level control interface and embodied state abstraction: In the robot control system, a hierarchical skill library and a topology-aware skill-state graph are constructed; the hierarchical skill library includes behavioral skills that directly interact with the underlying electromechanical hardware and semantic skills with contextual semantic representation; the skill-state graph uses the semantic skills as directed graph nodes and the topologically and physically feasible transfer paths between skills as directed edges, providing the embodied intelligent system with physical common sense abstraction and execution logic prior constraints in a non-explicit graphing environment; (2) Semantic planning of visual language model with topology awareness: Obtain the multimodal visual observation state of the robot at the current timestamp. With long-range natural language task instructions ; Based on stateless topological subgraphs, semantic-level multi-step task decomposition is performed using a visual language model. This allows for the search and generation of preliminary candidate skill sequences that conform to the topological connectivity of the environment within the legal connected domains of the graph network. ; (3) Feasibility verification of state-driven execution chain: Using the preceding states and state change functions in the skill-state diagram, the global state consistency of the candidate skill sequence in the latent space is verified based on physical common sense; by recursively applying the state change function along the candidate action chain to predict the future execution state of the robot body, if the verification is successful, the final executable semantic skill sequence is output. Otherwise, the error will be reported to the visual language model and the process will be returned to step (2) for correction and replanning; (4) Dual-track closed-loop monitoring based on multimodal heterogeneous feedback: The robot's underlying execution controller executes executable semantic skill sequences. Throughout the entire lifecycle, real-time data collection of proprioceptive feedback from the robot chassis and robotic arm, along with observation data from the visual sensors, is used to perform dual-layer state verification in both the proprioceptive attribute feedback dimension and the visual semantic dimension. This is to continuously evaluate whether the actual physical results of each skill execution strictly match the expected output boundary of the state change function. (5) Dynamic semantic replanning under graph constraints: If the multimodal dual-track monitoring in step (4) detects a physical operation error or semantic result logic deviation that exceeds the target threshold, the current predetermined open-loop execution chain is immediately terminated when either of the two errors is detected, and the actual physical attributes observed by the robot control system at the moment before the execution error occurs are extracted as the latest known physical state. It sets this as the initial anchor state, and triggers the replanning mechanism of the visual language model in the directed search space of the skill-state graph to generate a corrected skill sequence that seamlessly recovers from the current error state to the original task objective path. Then, it returns control to the underlying layer to continue executing the task.

2. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, The behavioral-level skills mentioned in step (1) are low-level basic control primitives that do not depend on specific semantics of the environment, including joint-level impedance control, compliant force control, and end-effector Cartesian kinematic inverse programming based on reinforcement learning; the semantic-level skills are high-level abstraction capabilities with explicit environmental interaction attributes and temporal pre-dependent conditions, and each semantic-level skill is formed by the sequential combination of several behavioral-level skills or by instantiation and mapping in a specific parameter space through an end-to-end policy network.

3. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, The skill-state graph described in step (1) is a directed graph topology with highly structured properties. , where nodes Directed edges represent specific semantic-level skills. Represents the reasonable transfer logic of local execution within physical space; and each node Configured with a set of preceding states and state change function Among them, the set of previous states Defined as a tuple of multidimensional physical conditions that a robot system must satisfy before safely executing the skill at this node; the tuple includes the relative spatial position of the robot chassis. The end effector of the left robotic arm is occupied. and the right robotic arm end effector is occupied. State change function This refers to the deterministic mathematical mapping and state value changes that occur to the multidimensional physical state of the robot body after the node skill has been executed in the physical world.

4. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 3, characterized in that, The state change function The operators for each dimension are defined as follows: For chassis movement skills, the spatial mapping function... Execution space transformation operator This transforms the robot's basic physical coordinate system from environmental anchor point A to environmental anchor point B; and for the robotic arm's object manipulation skills, a gripper mapping function is used. and Applying incremental combination operators To transfer the end effector from an idle state to hold a specific target object. The state, or the application release operator. To remove the gripper from holding a specific target object The state is reset to idle.

5. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, The stateless topological subgraph in step (2) It uses graph theory dimensionality reduction algorithms to extract the complete skill-state diagram. Explicitly extract the set of preceding states With state change function After attributes, only the abstract skeleton network structure of the graph nodes' connectivity and adjacency relationships is retained.

6. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, Step (2) involves the visual language model generating an initial candidate skill sequence. At that time, its internal multi-step reasoning and the growth path of the search tree are constrained by... Within the legally directed connected domain.

7. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, The specific logical operation process of the global state consistency verification based on physical common sense in step (3) is as follows: The system reads the robot sensor calibration initial physical state as... Along the candidate skill sequence generated by the visual language model The state transition equation is calculated iteratively. Predict the physical cumulative effect at each step in the sequence; for any step in the sequence... A progressive skill, the system determines the Boolean validation function. Does the return value match the absolute set of prerequisite states for the next skill to be executed in the sequence? Logical exclusion or physical conflict occurs.

8. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, The dual-track closed-loop monitoring of multimodal heterogeneous feedback described in step (4) includes two parallel monitoring processes that are decoupled and run on the physical time axis, including high-frequency body state monitoring and low-frequency semantic verification monitoring; high-frequency body state monitoring reads the data stream of the robot's multi-axis joint encoder and end gripper tactile sensor by directly mounting it on the industrial communication bus, and verifies the physical execution success rate of the motion primitive at a frequency higher than the visual control cycle. Low-frequency semantic verification monitoring involves the system asynchronously invoking the visual language big model to analyze the multi-view observation images and state change functions of the current environment at each low-frequency time node where a semantic-level skill node is declared to have completed execution. The theoretically expected environmental evolution parameters are compared and evaluated to determine whether abnormal object slippage, navigation position deviation, or long-term operational semantic errors have occurred.

9. The long-range mobile operation control method integrating skill-state diagram and closed-loop execution according to claim 1, characterized in that, After the dynamic semantic replanning under the graph constraint environment described in step (5) is initiated, the system memory stack will not be cleared and the entire long-range task chain will not be reset from the beginning. Instead, the abnormal physical state currently observed will be processed through the perception layer. The skill-state graph network input into memory performs local relocation of graph nodes. Utilizing a large visual language model, it searches for a path within a strictly constrained topological graph space that can reconstruct damaged states. Reconnect to the shortest corrected recovery path to the foremost node of the remaining unexecuted target subtasks.

Images