Task Execution Control Method Based on Semantic Agent and Its Application

Abstract

This invention discloses a task execution control method based on an execution semantic intelligent agent and its application, comprising: receiving a unified execution semantic frame, wherein the unified execution semantic frame is structured data generated based on multimodal input information and representing the task execution context of the current round; based on the unified execution semantic frame, performing compilation processing through an execution semantic compilation chain to generate an action contract, wherein the execution semantic compilation chain includes multiple links in semantic frame verification, task semantic compression, parameter slot binding, skill graph routing, candidate plan adjudication, and action contract assembly, wherein the action contract contains skill sequences and execution constraints used to drive the downstream action strategy layer; and sending the action contract to the downstream action strategy layer so that the downstream action strategy layer generates continuous action control signals according to the action contract. This invention achieves decoupling of interaction semantics and action control, thereby improving the adaptability, reliability, and auditability of the system in dynamic interactions.

Description

Technical Field

[0001] This invention relates to the fields of robot control and artificial intelligence technology, and in particular to a task execution control method based on an execution semantic intelligent agent and its application. Background Technology

[0002] As Large Language Models (LLMs) have evolved from text-generation-oriented conversational systems to agent runtimes, and further to agent systems oriented towards real-world tasks, the logic for generating agent capabilities has continuously shifted from "single-round generation" to "runtime execution." The core evolutionary lines can be summarized as tool contractualization, runtimeization, context protocolization, and program knowledge skillization. The former means that the model no longer merely outputs text, but interacts with external systems under constraints through structured tool interfaces. The latter elevates model invocation to a runtime that continuously maintains session state, tool sets, context windows, and execution loops. The access to external data sources, services, and workflows has shifted from temporary patchwork to standardized protocols.

[0003] In the current internationally available technological landscape, the core positioning of semantic intelligent agents remains a self-managed agency runtime for unified orchestration of messages, sessions, tools, and nodes, rather than an execution framework specifically designed for the low-level control of humanoid robots. Official documentation defines it as a unified gateway connecting various instant messaging applications and intelligent content creation and sharing platforms, emphasizing that its core consists of the gateway, session control, typed WS API (a type-safe, full-duplex communication protocol application programming interface), node role connections, and long-lifecycle session control.

[0004] Based on verifiable publicly available information, there is insufficient evidence to suggest that semantic intelligent agents have achieved mature and significant direct implementation on mainstream humanoid robot platforms abroad. In contrast, the current technological focus of publicly available cutting-edge humanoid robot technologies abroad remains primarily on the Vision-Language-Action (VLA) model and its hierarchical variants. Figure (a humanoid robot company) emphasizes a unified perception, language understanding, and high-frequency continuous control VLA path in its Helix (an embodied intelligence platform). Its System 2 is responsible for scene understanding and language semantics, while System 1 outputs continuous upper-body movements at 200 Hz. Physical Intelligence (a robotics company) employs a two-tiered structure of "high-level semantic policy + low-level VLA execution" in π0 (a vision-language-action flow model for general robot control) and Hi Robot (a hierarchical interactive robot system). The high-level VLM (visual-language large model) decomposes complex instructions into intermediate steps, which are then passed to the low-level policy to generate specific actions.

[0005] However, VLA-based humanoid robot control technology still has fatal flaws at the principle level. Since the basic modeling assumption is to compress task semantics, environmental state, action constraints and execution decisions into the same parameterized policy, and then directly generate continuous actions or action blocks from this policy, the control process that should have been explicitly unfolded, "understanding-decomposition-constraint-execution-verification", is implicitly folded into a conditional action mapping. This paradigm works effectively in short-term, closed, and distributed operational scenarios. However, its inherent limitations become apparent when it enters humanoid robot scenarios characterized by continuous interaction, dynamic semantic updates, temporary modification of task objectives, and real-time constraints from safety rules and external feedback. First, high-level semantics cannot be stably externalized into an executable structure independent of action strategies, making it difficult to form an action semantic layer that is approvable, traceable, and rewritable. Second, after task decomposition, process verification, and anomaly recovery are absorbed into the strategy, the system lacks an explicit reorganization interface for new semantics, constraints, and states inserted midway, relying solely on strategy reconditioning or additional layer patches. Third, to alleviate the temporal tension between high-frequency continuous control and multi-step inference, existing VLA systems often have to introduce compensation mechanisms such as two-layer structures, action block generation, or real-time chunking (RTC, an inference stage optimization technique used to improve the real-time performance and smoothness of visual-language-action (VLA) models in robot control). This precisely demonstrates that its underlying logic has not truly solved the problem of "how to stably transform interactive semantics into a governable execution process."

[0006] On the other hand, while general-purpose agentic runtime systems such as OpenClaw (an open-source AI agent) exist, which excel at handling message access, session scheduling, tool invocation, and node execution, their execution objectives are primarily geared towards software tools, remote commands, and service orchestration in general digital environments. They have not yet developed mature solutions for the action semantics, control interfaces, and safety loops necessary for embodied systems like humanoid robots. Lacking a dedicated execution semantic organization mechanism for robotic scenarios, they struggle to directly translate the continuously changing task intentions, environmental constraints, and safety conditions in natural interactions into a unified closed loop of action contracts, skill interfaces, execution mode switching, and feedback replanning within the existing application forms of OpenClaw.

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The purpose of this invention is to solve the technical problem of how to achieve a stable and auditable transformation of high-level task semantics into continuous actions in dynamic interactive scenarios, and to propose a task execution control method based on execution semantic intelligent agents and its application.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A task execution control method based on an execution semantic agent includes the following steps: S1, the execution semantic agent receives a unified execution semantic frame, wherein the unified execution semantic frame is structured data representing the current round's task execution context, generated based on multimodal input information; S2, based on the unified execution semantic frame, an execution semantic compilation chain is used to compile and process the data to generate an action contract that serves as an independent intermediate interface between high-level interactive semantics and downstream continuous action control, wherein the execution semantic compilation chain includes multiple stages such as semantic frame verification, clarification judgment, constraint projection, task semantic compression, parameter slot binding, skill graph routing, candidate plan adjudication, and action contract assembly, and the action contract contains skill sequences and execution constraints for driving the downstream action strategy layer; S3, the action contract is sent to the downstream action strategy layer so that the downstream action strategy layer generates continuous action control signals based on the action contract.

[0010] In some embodiments, in step S2, based on the unified execution semantic frame, compilation processing is performed through the execution semantic compilation chain, including: verifying and clarifying the unified execution semantic frame; projecting security boundaries and resource constraints to subsequent compilation stages; semantically compressing the task objective and binding it to parameter slots; the execution semantic agent does not directly process the original multimodal input information, nor does it directly generate continuous action control signals.

[0011] In some embodiments, in step S2, the execution of the semantic compilation chain is based on a secondary development of a general agent runtime base. The secondary development includes refactoring the objective function of the general agent runtime base from a general tool call to an action contract generation oriented towards robot control.

[0012] The present invention also provides a robot control method, comprising: processing voice, vision and memory input through a multimodal interactive agent to generate a unified execution semantic frame; executing a task execution control method based on an execution semantic agent as described in any of the preceding claims, generating an action contract based on the unified execution semantic frame; and executing the action contract through an action policy layer to generate continuous actions for controlling the robot.

[0013] The present invention also provides a method for storing action contract data, the method comprising: generating an action contract by executing a task execution control method based on an execution semantic agent as described in any of the preceding claims; and storing the action contract in a non-volatile storage medium.

[0014] The present invention also provides a method for transmitting action contract data, comprising: generating an action contract by executing a task execution control method based on an execution semantic agent as described in any of the preceding claims; and transmitting the action contract to a receiving end via a network.

[0015] The present invention also provides an execution semantic agent device, the device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the task execution control method based on the execution semantic agent as described in any of the preceding claims.

[0016] The present invention also provides a robot control system, the system comprising: a multimodal interactive agent configured to generate a unified execution semantic frame; an execution semantic agent device as described above, configured to receive the unified execution semantic frame and generate an action contract; and an action strategy layer configured to receive and execute the action contract to output continuous control instructions for the robot.

[0017] The present invention also provides an electronic device including the semantic agent device described above.

[0018] The present invention also provides a computer-readable storage medium storing a computer program / instruction and / or action contract data thereon, wherein the computer program / instruction, when executed by a processor, implements the task execution control method based on execution semantic intelligent agent as described in any of the preceding claims to generate the action contract data.

[0019] The beneficial effects of this invention compared to the prior art include: This invention establishes an independent execution semantic compilation layer between upstream multimodal input information and the downstream action strategy layer, stably compiling high-level task semantics into structured action contracts, thus effectively decoupling interaction semantics from action control. Specifically, through compilation stages such as semantic frame verification, task semantic compression, and parameter slot binding, the task intent described in natural language is transformed into standardized action contracts with clear skill sequences and execution constraints. This enables the downstream action strategy layer to generate continuous action control signals based on a unified interface specification. This invention externalizes continuously changing interaction semantics into auditable and recompilable intermediate execution objects. When facing complex situations such as task correction, constraint insertion, and anomaly recovery, the system does not need to re-perform a complete semantic understanding; instead, it achieves rapid adjustments through the recompilation of action contracts, effectively avoiding semantic drift and control ambiguity. This invention constructs an independent and governable execution semantic compilation layer, achieving decoupling between interaction semantics and action control, thereby improving the system's adaptability, reliability, and auditability in dynamic interactions.

[0020] Furthermore, the secondary development approach based on the general agent runtime platform allows this invention to fully utilize existing mature technologies while performing specific optimizations for robot control scenarios, reducing implementation complexity while ensuring technological advancement. The network transmission capability of action contracts also provides technical support for the construction of distributed robot control systems.

[0021] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0022] Figure 1 This is a flowchart of the task execution control method in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the core link of the semantic intelligent agent in an embodiment of the present invention; Figure 3 This is a schematic diagram of the interaction-execution architecture driven by a unified round-based execution semantic frame in an embodiment of the present invention; Figure 4 This is a diagram of the execution semantic agent-VLA collaborative execution mechanism for continuous control strategy access in this embodiment of the invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0024] To address the two core technical challenges in existing embodied robot control systems—the difficulty in stably externalizing high-level interactive semantics into an independent execution structure and the lack of specialized execution semantic mechanisms for robot scenarios during general agent runtime—and to further achieve stable transformation of high-level task semantics into continuous action control under conditions of continuous interaction, dynamic constraint insertion, anomaly recovery, and state write-back, this invention proposes a layered technical solution centered on an execution semantic agent. The aim is to establish an independent, auditable, and recompilable execution semantic compilation layer between the multimodal interactive agent and the VLA / action policy layer, allowing the interaction side to focus solely on fact convergence and unification. The Execution Semantic Frame (TESF) generation process involves the execution semantic agent being solely responsible for organizing and generating skills, action plans, and action contracts within a defined target state, environmental state, safety state, capability requirements, action intentions, and parameter slot constraints. The VLA / Action Policy layer is only responsible for implementing continuous actions according to the action contracts. This clearly establishes a stable boundary between "fact convergence—execution compilation—action implementation" in engineering, preventing high-level semantic reorganization, execution organization and orchestration from interfering with low-level action control. Ultimately, this achieves a manageable, traceable, rewritable, and scalable execution control chain for complex robot interaction tasks. Specifically:

[0025] Technical Problem 1: Addressing the shortcoming of VLA (Variable Access Layer) which compresses task semantics, environmental constraints, and execution decisions into a parameterized strategy, making it difficult to externalize high-level semantics in continuous interaction into a governable execution process, this invention does not attempt to add more conditional inputs or compensation mechanisms within the existing VLA. Instead, it re-unfolds the originally folded process from the principle layer of the control chain. The "understanding-decomposition-constraint-execution-verification" process is explicitly split into three continuous and clearly defined modules. Specifically, the multimodal interactive agent is first responsible for the factual convergence of speech, vision, memory, and governance constraints, generating a unified execution semantic frame. The execution semantic agent then uses this semantic frame as the sole input, and under given target state, environmental state, safety state, capability requirements, action intentions, and parameter slot constraints, completes task semantic compression, parameter binding, capability parsing, skill graph routing, candidate plan adjudication, and action contract assembly. The VLA / action strategy layer then generates continuous actions only around the skill sequence, object constraints, spatial constraints, execution preconditions, termination conditions, and failure fallback conditions in the action contract. Thus, for the first time, high-level semantics are stably externalized into an execution organizational structure independent of action strategies. Task decomposition, process verification, and anomaly recovery are no longer implicitly absorbed into a single strategy, but are explicitly reorganized through a unified execution semantic frame write-back, action contract recompilation, and execution receipt closed loop. In other words, this solution does not add a specification layer outside of VLA, but inserts an intermediate execution semantic layer that is approvable, traceable, write-backable, and recompilable before it. Therefore, it can directly solve the fundamental defect of existing VLAs in continuously interacting, target correcting, constraint insertion, and anomaly recovery scenarios, which makes it difficult to stably transform dynamic semantics into a governable execution process.

[0026] Technical Issue 2: Given that the current application of execution semantic agents mainly focuses on tool invocation and node execution in general digital environments, and lacks the development of robot action semantics, control interfaces, and a security closed loop, this invention does not directly transplant the native OpenClaw model to the robot side. Instead, it focuses on its existing strengths in session operation, skill loading, tool invocation, node access, and code proxy, aiming to develop a dedicated execution semantic organization mechanism for robot scenarios, thereby filling the most critical layer beyond its original application boundaries. Specifically, it retains OpenClaw's runtime advantages in message access, session scheduling, capability registration, and execution approval, but reconstructs the original tool execution chain for general digital environments into an execution semantic compilation chain for robot control. This involves a unified execution semantic frame leading to semantic frame verification, clarification judgment, constraint projection, task semantic compression, parameter slot binding, capability parsing, skill graph routing, candidate plan adjudication, execution mode selection, plan normalization, and action contract assembly, before connecting with downstream VLAs via an action contract bridge. In this way, OpenClaw is no longer used as a general-purpose agent executor, but is instead confined to the runtime foundation of the execution semantic intelligent agent. Its output object changes from "software tool call results" to "action contracts," and its feedback object changes from "general execution state" to "execution receipts and environmental increments that can be written back to a unified execution semantic frame." Therefore, this solution does not exaggerate its ability to directly solve all of OpenClaw's embodied control problems, but rather precisely addresses its most core and missing aspect: establishing a layer of execution semantic organization mechanism specifically for robotic scenarios on top of the general runtime. This allows continuously changing task intentions, environmental constraints, and safety conditions to be stably converted into skill interfaces, execution mode switching, action contract outputs, and feedback replanning loops, thereby achieving a truly practical engineering connection between interaction and control.

[0027] like Figure 1 As shown, this embodiment of the invention provides a task execution control method based on an execution semantic agent, including the following steps: S1. The execution semantic agent receives a unified execution semantic frame, which is structured data representing the current round's task execution context, generated based on multimodal input information. Specifically, the unified execution semantic frame is structured data generated by the upstream multimodal interactive agent after fact convergence, constraint fusion, and state organization based on speech information, visual information, ontology state information, and / or memory information, etc., and includes at least several of the following: current task objective, scene state, object constraints, safety boundaries, capability requirements, action intentions, and parameter slots.

[0028] S2. Based on a unified execution semantic frame (which can be used as the sole direct processing input), the execution semantic agent performs compilation processing through the execution semantic compilation chain to generate an action contract that serves as an independent intermediate interface between high-level interactive semantics and downstream continuous action control. The execution semantic compilation chain includes multiple stages such as semantic frame verification, clarification judgment, constraint projection, task semantic compression, parameter slot binding, skill graph routing, candidate plan adjudication, and action contract assembly (the stages of the "execution semantic compilation chain" can be added, deleted, merged, or their order adjusted according to actual needs). The action contract contains skill sequences and execution constraints used to drive the downstream action strategy layer (e.g., vision-language-action model), including at least several of the following: skill sequence, skill call parameters, object constraints, spatial constraints, execution order, execution preconditions, termination conditions, failure rollback conditions, and execution receipt fields.

[0029] In step S2, based on the unified execution semantic frame, compilation processing is performed through the execution semantic compilation chain, including: verifying and clarifying the unified execution semantic frame; projecting security boundaries and resource constraints to subsequent compilation stages; semantically compressing the task objective and binding it to parameter slots. Specifically, the execution semantic compilation chain includes pre-compilation processing: verifying the field integrity, semantic consistency, and security legality of the unified execution semantic frame; generating a clarification request or aborting compilation when missing parameters, ambiguities, or constraint conflicts are detected; writing security boundaries, resource limits, termination conditions, and failure criteria into compilation constraints; and compressing the task objective into skill-level task units and binding them to target objects, spatial locations, tool capabilities, execution order, and action parameter slots.

[0030] In step S2, the execution of the semantic compilation chain is based on secondary development of the general agent runtime base. This secondary development includes refactoring the objective function of the general agent runtime base from general tool calls to action contract generation oriented towards robot control. Specifically, the execution of the semantic compilation chain is obtained by refactoring the general agent runtime base. This refactoring includes configuring the session execution, skill loading, tool calls, and node access capabilities of the general agent runtime base into skill routing, candidate plan adjudication, and action contract assembly capabilities oriented towards robot control.

[0031] S3. The action contract (which can be the sole coupling object) is sent to the downstream action strategy layer, which then generates continuous action control signals based on the skills, parameters, objects, space, and safety constraints defined by the action contract. The execution semantic agent does not directly process the raw multimodal input information (hierarchical encoding), nor does it directly generate joint-level, trajectory-level, torque-level, or other continuous action control signals.

[0032] S4. Receive the execution receipt and / or environment increment returned by the downstream action strategy layer, and write the execution receipt and / or environment increment back to the unified execution semantic frame; when the task objective, scene state, security boundary, object constraint or execution result changes, trigger the execution semantic compilation chain again based on the written-back unified execution semantic frame to generate the updated action contract.

[0033] This invention also provides a robot control method, comprising: performing fact convergence and state organization on voice, vision, ontology state and / or memory input through a multimodal interactive agent to generate a unified execution semantic frame; executing the task execution control method based on the execution semantic agent as described in any of the preceding claims to generate an action contract based on the unified execution semantic frame; and generating continuous actions to control the robot through an action strategy layer under the skills, parameters and constraints defined by the action contract.

[0034] The present invention also provides a method for storing action contract data, comprising: generating an action contract by executing a task execution control method based on an execution semantic intelligent agent as described in any of the preceding claims; and storing the action contract, which includes a skill sequence, parameter mapping, execution constraints, and acknowledgment fields, as an intermediate execution interface data in a non-volatile storage medium.

[0035] This invention also provides a method for transmitting action contract data, comprising: generating an action contract by executing a task execution control method based on an execution semantic agent as described in any of the preceding claims; and transmitting the action contract as an intermediate execution interface between a high-level interactive semantic layer and a downstream action strategy layer to a receiving end (e.g., a robot end, an edge control end, or an action strategy execution end) via a network.

[0036] This invention also provides an execution semantic agent device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the task execution control method based on the execution semantic agent as described in any of the preceding embodiments. The device further includes: a semantic frame verification module for performing integrity verification on received unified execution semantic frames; a task compilation module for performing task semantic compression and parameter slot binding on the verified semantic frames; a plan adjudication module for performing risk assessment and routing selection on candidate execution plans based on a capability graph; and a contract assembly module for normalizing the adjudicated execution plans into standard action contracts.

[0037] This invention also provides a robot control system, comprising: a multimodal interactive agent configured to generate a unified execution semantic frame; an execution semantic agent device as described above configured to receive the unified execution semantic frame and generate an action contract; and an action strategy layer configured to receive and execute the action contract to output continuous control instructions for the robot.

[0038] This invention also provides an electronic device, including the semantic agent execution device described above.

[0039] This invention also provides a computer-readable storage medium storing computer programs / instructions and / or action contract data, wherein the computer programs / instructions, when executed by a processor, implement the task execution control method based on execution semantic intelligent agents as described in any of the preceding claims to generate action contract data.

[0040] The following describes specific embodiments of the present invention.

[0041] This embodiment provides a layered technical solution for the control link of embodied intelligent agents. Its core lies in proposing and implementing an independent technical layer called the execution semantic intelligent agent. The execution semantic intelligent agent refers to an intelligent agent type located between the multimodal interactive intelligent agent and the downstream action strategy layer, taking a unified execution semantic frame as input and an action contract as output, specifically responsible for compiling the high-level task semantics into a governable execution organization result. Its essential function is not to re-understand the interaction, nor to directly generate joint-level, trajectory-level, or torque-level control, but rather, under given target states, environmental states, safety states, capability requirements, action intentions, and parameter slot constraints, to stably compile the high-level task semantics into an action contract consisting of skills, action plans, parameter mappings, execution preconditions, termination conditions, and failure fallback conditions. Based on this definition, the complete technical solution of this embodiment consists of three interconnected modules: First, an execution semantic agent ontology is proposed and developed, establishing an execution semantic compilation main chain that includes semantic frame verification, clarification judgment, constraint projection, task semantic compression, parameter slot binding, capability parsing, skill graph routing, candidate plan adjudication, execution mode selection, plan normalization, and action contract assembly; Second, an "interaction-execution" collaborative architecture driven by a unified execution semantic frame is constructed, stably connecting the fact convergence completed by the multimodal interaction agent and the high-level execution compilation completed by the execution semantic agent at the backend orchestration layer to form a new interaction-execution agent link; Third, a three-system layered docking scheme is constructed for the semantic interaction agent, the execution semantic agent, and the VLA / action strategy layer, using the action contract as the only legal coupling object, so that the downstream action strategy layer only revolves around the contract to generate continuous actions, and forms a closed loop through execution receipt writing and semantic frame recompilation. In terms of specific implementation, OpenClaw is only one implementable example of an execution semantic agent. That is, this embodiment can use its session execution, skill loading, tool invocation, node access and code proxy capabilities as a secondary development foundation to realize the execution semantic agent. However, the embodiments of the present invention are not limited to OpenClaw. Any agent that meets the above-mentioned input boundaries, output boundaries and functional definitions and can complete the stable compilation of unified execution semantic frames to action contracts is within the scope of the implementation of the execution semantic agent of the embodiments of the present invention.

[0042] The role of the semantic agent in robotics is reflected in its collaborative mechanism with multimodal interactive agents around a unified round of semantic frames. OpenClaw is just one example. When the semantic agent is applied to robotics, it is not an auxiliary execution end outside the multimodal interactive agent, but rather it is connected to the backend orchestration layer. It directly receives the unified round of semantic frames that have completed automatic speech recognition (ASR), visual trigger determination and necessary visual injection, memory gating and context assembly, and the semantic frames simultaneously drive the dialogue branch and the control branch to operate collaboratively. Specifically, Automatic Speech Recognition (ASR) / Vision / Memory first generates round-execution semantic frames through unified semantic orchestration. These frames include not only the user's task objective but also scene state, object constraints, safety conditions, stage progress, and interaction intent. Therefore, the Dialogue Path can generate confirmations, explanations, follow-up questions, progress reports, and result descriptions in real time based on the same semantic frame. The Control Path, on the other hand, synchronously completes action contract configuration, skill interface selection, execution mode switching, Robot Adapter mapping, and restricted execution based on the same semantic frame. Its main operational chain is: Automatic Speech Recognition (ASR) / Vision / Memory → Unified Semantic Orchestration and Round-Execution Semantic Frame Generation → Branching into Dialogue Path (LLM (Large Language Model) → TTS (Text-to-Speech) Streaming Expression) and Control Path (Execution-Semantic Agent → Robot Adapter → Restricted Execution) running in parallel → Execution feedback is written back to the round state center → Dialogue supplementation, exception explanations, or control replanning are triggered as needed. Thus, robots can simultaneously complete understanding, expression, execution, and correction within the same semantic state, enabling interactive expressions and physical actions to be synchronized in time, semantically cognate, and mutually corrected in state, thereby highly simulating the complex behaviors of humans in real-world situations.

[0043] The principles behind the proposal and development of semantic intelligent agents are as follows: In the robot execution system, the execution semantic agent is positioned between the unified execution semantic frame and the downstream action strategy layer, responsible for compiling high-level task semantics into executable organizational results. Its processing object is not raw speech, vision, or free text, but rather the unified execution semantic frame after semantic convergence, constraint fusion, and state organization. Therefore, the agent's output should not be joint-level, trajectory-level, or torque-level control quantities, but rather an action contract composed of skills, action plans, parameter constraints, execution preconditions, termination conditions, and failure fallback conditions. In other words, the engineering task of this layer is not "directly controlling the robot," but rather transforming understandable task states into an organizeable, inspectable, and deliverable execution semantic structure, providing strictly constrained upstream input to the downstream action strategy layer.

[0044] The decisive role of the semantic agent in the robot control system lies in its intermediate compilation responsibility of unifying the execution semantic frame into executable control organization results. This allows the high-level task semantics, which have already achieved semantic convergence upstream, to enter the downstream action strategy layer in a structured, constrained, and deployable form. The unified execution semantic frame still provides high-level semantic information such as target state, environmental conditions, safety boundaries, capability requirements, action intentions, and parameter constraints. This information itself cannot be directly expanded into continuous actions. The task of the semantic agent is to further organize this information into skills, action plans, parameter mappings, sequence relationships, execution preconditions, termination conditions, and failure fallback conditions, without changing the original task objectives and safety boundaries. This forms a downstream-oriented action contract, enabling the VLA to translate the plan into an executable continuous action strategy under a clear object, constraint, and organizational structure. From the perspective of control layering, this intelligent agent essentially assumes the responsibility of the execution organization layer in the robot control chain: it is neither responsible for re-understanding multimodal interactions nor for generating joint-level, trajectory-level, or torque-level control signals. Instead, it is specifically responsible for determining, based on the unified execution semantic frame, which skills to invoke, which execution path to adopt, how to combine available capabilities, and under what constraints to start, terminate, or roll back the corresponding execution process. Therefore, its functional boundaries must be strictly limited to the level of "compiling high-level task semantics into action contracts." Tasks involving fact convergence, semantic disambiguation, and contextual understanding should be completed by the multimodal interactive agent, while tasks involving continuous action generation, ontological dynamics adaptation, and real-time control should be completed by the VLA and its downstream control layers. Only after this boundary is clearly defined can the execution semantic agent play its proper pivotal role in the robot control system, that is, transforming abstract task requirements into execution organization results that can be stably accepted by the action strategy layer, while avoiding overlapping responsibilities, semantic drift, and control overstepping between the interaction understanding layer and the action control layer.

[0045] From an implementation perspective, the execution semantic agent is not constructed out of thin air. Instead, it is built upon the existing session execution, skill loading, tool invocation, plugin extension, node access, and external code proxy capabilities in the OpenClaw open-source code. Further development involves reconstructing its internal main chain around the robot scenario. In this embodiment, what OpenClaw can inherit is not its general assistant product form, but its runtime skeleton, namely the session execution kernel, skill execution, tool mode registration, plugin loading, node command plane, and external proxy bridging capabilities. What needs to be rewritten as the execution semantic agent is its objective function, main forward chain, and delivery object. Current platforms mostly emphasize general session closure and tool invocation closure, while this emphasizes a task compilation closure under unified execution semantic frame constraints. This means that OpenClaw is no longer considered a terminal intelligent agent product in this context, but is absorbed as the runtime foundation and capability-bearing layer of the execution semantic agent.

[0046] like Figure 2 As shown, the internal main chain of the execution semantic agent is explicitly unfolded as follows: streaming round entry, semantic frame validator, clarification decision gate, constraint projector, task semantic compressor, parameter slot binder, capability parser, skill graph router, candidate plan generator, risk cost adjudicator, local skill executor, plug-in workflow orchestrator, restricted code planner, plan normalizer, action contract assembler, downstream delivery adapter, and action contract. This chain is not a parallel stacking of several functional modules, but a sequential unfolding of a complete execution semantic compilation process at the engineering level. Among them, the streaming round entry is responsible for receiving the current round semantic input from the unified execution semantic frame and establishing the round context required for subsequent processing; the semantic frame validator is responsible for checking field integrity, semantic consistency, and security legality, and sending the verification results to the clarification decision gate and constraint projector respectively; the clarification decision gate is responsible for identifying missing parameters, ambiguity, conflict, or low-confidence inputs, and suspending subsequent orchestration and sending back clarification requests when necessary; the constraint projector is responsible for explicitly projecting safety boundaries, resource limits, termination conditions, and failure criteria to the task semantic compressor and its subsequent chains. Only after the current round of inputs passes validity verification and constraint pre-injection is completed can the task semantic compressor begin to converge the high-level task semantics. Therefore, the entire main chain has clear prior constraints, sequential causality, and engineering endpoints, rather than remaining at the level of conceptual diagram description.

[0047] Task semantic compression and parameter slot binding constitute the first set of key technologies for execution semantic intelligent agents. While the unified execution semantic frame already contains information such as target state, environment state, safety state, capability requirements, action intent, and parameter slots, it still contains higher-level task expressions, insufficient for direct consumption by the downstream action strategy layer. Task semantic compression aims to consolidate high-level task objectives into skill-level task units and sub-task relationships, freeing subsequent orchestration from reliance on open-ended natural language interpretation. Parameter slot binding, on the other hand, writes the object, spatial location, action sequence, time requirements, intensity range, termination conditions, and failure fallback conditions into a structured template, giving each candidate plan explicit constraints. After this processing, execution prerequisites no longer float in descriptive text but are solidified into a set of inspectable, routable, and rewritable fields. The direct result is that downstream processes will not freely guess the action object or execution conditions due to "different understandings," thus significantly reducing execution ambiguity.

[0048] Building upon this foundation, the clarification decision gate, constraint projector, task semantic compressor, and parameter slot binder collectively constitute the first set of key technologies in the front-end compilation chain of the execution semantic agent. Although the unified execution semantic frame already carries high-level information such as target state, environment state, safety state, capability requirements, action intent, and parameter slots, this information still needs explicit standardization in the front-end compilation chain before entering the downstream skill organization stage. The clarification decision gate intercepts low-confidence, missing field, or undefined inputs before the execution chain begins, preventing the system from entering skill orchestration with incomplete key information. The constraint projector pre-writes safety, resource, sequence, termination, and failure conditions into the subsequent task compilation process, making these conditions no longer secondary descriptions but internal constraints that every candidate plan must inherit. The task semantic compressor then converges high-level task objectives into skill-level task units and their sub-task relationships, while the parameter slot binder further writes objects, spatial locations, action sequences, time requirements, force ranges, termination conditions, and failure fallback conditions into a structured template. After this front-end processing, the execution premises no longer float in descriptive text, but are solidified into a set of fields that can be checked, routed, verified, and written back, thus significantly reducing the risk of downstream misjudgment of action objects, environmental conditions, and execution boundaries.

[0049] The candidate plan generator, risk and cost adjudicator, local skill executor, plug-in workflow orchestrator, and restricted code planner together constitute the third set of key technologies in the back-end compilation link of the execution semantic agent. The candidate plan generator does not predetermine a single execution path, but rather generates multiple feasible candidate solutions based on task structure, parameter slots, capability graph, and historical execution records. The risk and cost adjudicator then comprehensively evaluates the candidate solutions from five dimensions: semantic consistency, capability accessibility, contextual continuity, execution risk, and resource cost, and determines whether to prioritize them for the local skill execution path or the plug-in workflow orchestration path. It is important to note that the execution mode in the diagram does not result in three completely parallel and independent terminal paths after adjudication. Instead, the risk and cost adjudicator first allocates the main path between the local skill executor and the plug-in workflow orchestrator, and then the restricted code planner conditionally bridges, completes, and restricts the execution organization results produced by the aforementioned paths. Therefore, the restricted code planner is no longer defined as the default main chain, but is instead limited to a restricted compensation layer that is only activated when there are skill gaps, complex structures, or temporary absences of bridging logic. Its role is to complete the preceding execution path with code-level organization, rather than bypassing the established main path and directly replacing the entire execution orchestration process. After this processing, the relationship between local skill execution, plugin orchestration, and restricted code planning is consistent with the sequence and convergence logic in the diagram, and the execution semantic agent thus possesses a stable back-end decision structure.

[0050] To prevent the aforementioned adjudication process from degenerating into a mere patchwork of empirical engineering and to establish an independent, auditable runtime decision-making kernel, a single core algorithm can be introduced between risk cost adjudication and execution mode selection, denoted as the "Execution Semantic Candidate Plan Optimization and Mode Routing Algorithm," whose expression is as follows:

[0051]

[0052] in, α 1- α 5 represents the weight coefficients for each evaluation item, corresponding to the semantic consistency item, capability matching item, context continuity item, risk penalty item, and cost penalty item, respectively. π Indicates candidate implementation plans, π t Denotes the set of candidate execution plans for round t; m t Indicates the first t The optimal execution mode determined by the loop-mode routing operator. τ t It is the first t The semantic frames are executed uniformly in rounds;G t This is the current capability map; h t It is the context and the verified execution history. s t This is the current safe state; Ψ sem Ψ cap Ψ ctx Ψ risk Ψ cost These represent semantic consistency, capability matching, contextual continuity, and risk and cost assessment items, respectively; Ω ( π,s t ) represents a safety hard-gated function; R (·) denotes the pattern routing operator. (·) represents the pattern routing operator. sem Taking the (semantic consistency) evaluation function as an example, one feasible implementation is to use a pre-trained semantic similarity model to calculate the task description and unified execution semantic frame of the candidate plan π. τ t The cosine similarity between the task objectives is used as the score for this evaluation item. The safety hard-gating function Ω( π , s t One implementation of this is to check if the skill or resource required by plan π exists in a safe state. s t If an item is within the defined permitted list but not within the prohibited list, the check passes and outputs 1; otherwise, it outputs 0 to reject the plan. The pattern routing operator R selects a pattern based on the overall evaluation score and a preset pattern priority strategy. For example, it prioritizes the local skill execution mode; if the overall score is below a threshold or the capability is not met, it then tries the plugin workflow mode and the restricted code planning mode in sequence. The essence of this algorithm is not to train a new end-to-end model, but to select the optimal organizational path from multiple execution semantic candidate schemes under hard safety constraints and determine the execution mode it should enter. With this setting, the execution semantic agent has an independent, stable, and auditable runtime decision-making core.

[0053] In the final convergence phase, the plan normalizer, action contract assembler, and downstream delivery adapter jointly compress heterogeneous execution paths into deliverable standard objects, and fix the endpoint boundary of the execution semantic agent at this point. The plan normalizer's task is to unify the heterogeneous results from local skill execution paths, plug-in workflow orchestration paths, and constrained code planning compensation paths into a homogeneous execution organization representation. The action contract assembler then generates action contracts based on this, the content of which includes at least skill sequences, parameter mappings, object constraints, spatial constraints, execution preconditions, termination conditions, failure fallback conditions, and verifiable execution goals. The downstream delivery adapter is further responsible for converting the action contract into an interface form that can be stably received by the downstream action policy layer. Therefore, the action contract is not equivalent to the action policy, much less a continuous control sequence; it is essentially still a standardized delivery object of high-level execution semantics. The execution of semantic agents by stopping at the action contract and not continuing to generate continuous actions is not due to a lack of capability, but rather a deliberate design choice based on engineering layers: its responsibility is to ensure that the semantic constraints of higher-level tasks have been organized, the structure converged, and the path unified before entering the downstream layer, while the unfolding of continuous actions, the generation of control sequences, and the adaptation of ontological dynamics fall within the responsibilities of the downstream action strategy layer. It is precisely because the endpoint is stably fixed at the "downstream delivery adapter → action contract" segment that the interface boundaries of the entire chain are truly clear, and the correspondence between the architecture diagram and the main text becomes feasible.

[0054] To ensure the execution semantic agent forms a closed loop rather than being a one-off converter, execution receipts, telemetry data, and anomalies returned by the downstream action policy layer must re-enter the unified execution semantic frame update process. The execution receipt interpreter first performs result attribution and structured interpretation of the downstream feedback. The verified result extractor then extracts confirmed completed skill results, environmental state increments, and failure reasons. The semantic frame write-back engine then updates the target state, environmental state, safety state, and capability knowledge accordingly. With this processing, the execution semantic agent no longer simply "hands over the task," but continuously maintains an auditable, replayable, and correctable execution semantic organization closed loop. Its engineering significance lies in the fact that subsequent rounds do not need to start from scratch to understand the execution status, but can continuously adjust the task organization and execution path based on verified results.

[0055] The unified round-execution semantic frame-driven interaction-execution coordination architecture is as follows: The reason why the execution semantic agent is defined separately is fundamentally because interaction understanding, execution orchestration, and continuous control belong to different semantic granularities and control levels. The multimodal interaction agent is responsible for forming factual states, answering questions such as "What does the user want to achieve? What is the current environment? Which constraints are valid?" The execution semantic agent is responsible for forming the execution organization results, answering questions such as "Which skills should be arranged? How should parameters be filled in? What execution path should be adopted?" The downstream action strategy layer is responsible for expanding the above organization results into a continuous sequence of actions. Without an intermediate execution semantic compilation layer, the system either degenerates into the interaction side directly assigning action descriptions or the downstream control side repeatedly interpreting user intentions. Both of these will lead to drift in object reference, environmental conditions, temporal logic, and security boundaries. Therefore, the boundary of the execution semantic agent must stop at the action contract and must not overstep its bounds to encroach on the responsibilities of interaction understanding or continuous control.

[0056] In the "interaction-execution" collaborative system, the key to stable system deployment lies not in whether voice, vision, memory, tool calls, and control interfaces appear simultaneously in the same architecture, but in the form, level, and constraints of the user's multimodal interaction information to be stably transcribed into downstream executable control semantics. If this intermediate layer is not established separately, the system typically degenerates into two types of mismatched paths: first, the interaction side generates a natural language response and then appends an action description to the output, resulting in language expression and control execution not sharing the same factual basis; second, the execution side rereads the original voice or text and performs task understanding again, leading to parallel existence of interaction understanding and execution understanding without unified constraints. The former problem is that object reference, temporal conditions, environmental evidence, and security boundaries are prone to drift; the latter problem is that two competing semantic interpretation processes form within the system, ultimately weakening consistency and verifiability. Therefore, the interaction agent and the execution semantic agent cannot adopt a loosely coupled connection method of "output followed by action" or "repeated understanding at the execution end". Instead, a unified round-level semantic hub must be set up in the back-end orchestration layer so that the expression branch and the execution branch can cooperate around the same round, the same set of facts and the same set of constraints.

[0057] Based on the above constraints, the connection mechanism discussed in this section is not a module splicing relationship in the general sense, but a hierarchical collaborative mechanism centered on a unified round execution semantic state (see [link to relevant documentation]). Figure 3The basic principle of this mechanism is that multimodal interactive agents do not directly output control commands to the downstream action layer, and semantic agents do not directly consume the surface form of language. The only legitimate connection between the two is the unified round-execution semantic frame formed by the fusion of speech understanding, visual evidence injection, memory gating, and policy constraints. Thus, the interaction side assumes the responsibility of "semantic convergence," and the execution side assumes the responsibility of "semantic compilation." The system no longer relies on the output end's interpretation or repeated understanding to maintain collaboration, but instead uses structured states as a common basis at the backend orchestration layer. The fundamental significance of this approach lies in transforming the relationship between "understanding" and "execution" from serial text transmission to a state-driven process, ensuring consistency between language feedback and control planning in terms of objects, environment, constraints, priorities, and round identities, thereby providing a unique and auditable semantic entry point for subsequent execution and feedback.

[0058] The responsibilities of a multimodal interactive agent are strictly defined from the outset. Its function is not to generate continuous robot actions, nor to replace the execution end in skill planning, but rather to compress heterogeneous inputs from voice, vision, memory, and governance into a round-level semantic state that can be subsequently compiled, audited, and rewritten. Specifically, the voice channel serves as the main entry point for the user's explicit intent, providing task requests, object designations, conditional constraints, sequential relationships, and interaction context; the vision channel handles environmental evidence and object disambiguation, confirming real objects in the scene, their spatial locations, and whether they meet the current execution prerequisites; the memory channel maintains cross-round continuity, handling confirmed objects from previous rounds, contextual inheritance, previous execution results, and unfinished task remnants; and the policy and governance channel injects system-level boundaries, writing permission levels, action whitelists, prohibition conditions, resource quotas, and risk thresholds into the current round's semantic state. Therefore, the product of the multimodal interactive agent is not "action," but rather a unified semantic fact after evidence convergence. All information that can affect the object of action, execution order, spatial relationships, triggering conditions, or security restrictions must be transformed from discrete input into explicit state variables at this stage, and must not remain at the level of vague natural language descriptions. For this reason, the role of a multimodal interactive agent in the overall framework should be defined as a round-based fact builder, rather than a control command generator.

[0059] The responsibilities of the execution semantic agent differ from this. Its task is not to reinterpret the user's original expression, as the semantic convergence process has already been completed on the interaction side; its core function lies in compiling the unified round execution semantic frames under field constraints, transforming state-level semantics into skill-level task arrangement and execution constraints. More specifically, the execution semantic agent is not concerned with "what the user said," but rather with "which skills should be selected, how parameters should be filled in, in what order execution should be organized, and under what conditions should execution, rollback, or replanning be carried out, given the constraints of goals, environment, safety, and capabilities." Therefore, the execution semantic agent is neither a post-processor of the dialogue system nor a low-level action controller that directly outputs joint instructions or control variables, but a high-level execution orchestrator located between the unified semantic state and the downstream action policy layer. Its output is not a natural language sentence, but a set of skills / action plans with object constraints, temporal relationships, interface mappings, termination criteria, and failure rollback conditions. Thus, a clear upstream and downstream relationship is formed between the multimodal interactive intelligent agent and the execution semantic intelligent agent: the former is responsible for compressing user intent, environmental evidence and system boundary into a unified fact state, while the latter is responsible for generating a high-level task arrangement with execution significance on top of this fact state; the two complement each other without overlapping in terms of semantic granularity and scope of responsibility.

[0060] To ensure the feasibility of the aforementioned upstream and downstream relationships, they must be connected through explicit data objects, rather than remaining at the level of abstract representation. This data object is the unified round execution semantic frame. This semantic frame is neither a general context cache nor an arbitrarily expandable free text container, but rather a structured state unit with fixed core fields. Its core fields include at least: task objective, representing the target state to be converged to in this round; environment state, representing objects, scenarios, spatial relationships, and observable conditions; security state, representing permission levels, action whitelists, restricted areas, and runtime interlocking conditions; capability requirements, representing the skill categories, tool interfaces, and execution resources required in this round; action intent, representing the skill-level task type; parameter slots, representing objects, positions, sequences, intensity, duration, termination conditions, and failure criteria; memory summary, used to maintain continuity across rounds; visual evidence, used to carry the results of target confirmation and environment verification; policy boundaries, used to record platform governance rules and system-level restrictions; and round identifiers, used to ensure that state write-back, anomaly alignment, and subsequent replanning all revolve around the same round. Once a unified round execution semantic frame is established, the collaborative relationship between the interactive agent and the execution semantic agent changes from "module connection" to "taking on different responsibilities around the same state object": the interactive side is responsible for filling in the state, and the execution side is responsible for compiling the state.

[0061] After the unified round execution semantic frame is clearly defined, the system requires a single, implementable core algorithm to perform constrained selection of candidate semantic states and simultaneously derive the dialogue plan and execution plan from them. This core algorithm can be described as a unified round execution semantic compilation and dual-track splitting algorithm:

[0062] in, t Indicates the current round number. Zt Indicates the first t The set of all candidate semantic frames to be executed in a round. z The candidate unified execution semantic frame is defined as follows: z = ( q , e , s , n , i , r , μ )∈ Z t ; q The target state is used to characterize the target that this round of tasks expects to converge to; e Environmental states are used to characterize external conditions such as objects, scenes, and spatial relationships. s It is a security state, used to characterize security boundaries such as permissions, restricted areas, interlocks, and whitelists; n These are capability requirements, used to characterize the skills, tools, or execution resources needed for this round; i It is the high-level control semantics used to represent "what to do" in terms of action intent; r It is a set of parameter slots and constraints used to represent necessary execution information such as object, position, order, force, termination condition and failure criteria; μ It is a memory summary used to characterize the continuous information extracted from historical dialogues and states in the current round. a t It is the first t Round ASR semantic decoding results; v t It is the first t Visual evidence or visual injection results; M t It is the first t Wheel memory input; p t It is the first t Round strategy / constraint injection, Gt It is the current system's capability map or skill map; function Φ sem ( z | a t , v t , Mt , p t () represents the semantic consistency term, used to measure candidate frames. z The degree of consistency between speech, vision, memory, and constraints; Φ exe ( z | G t ) represents the executability item, used to measure whether the candidate frame can be actually implemented by existing skills, toolchains, execution semantic intelligent agent runtimes, and downstream interfaces; Φ ctx ( z | M t ) represents the context continuity term, used to measure the consistency between candidate frames and historical rounds, object references, and dialogue continuity; Φ risk ( z | s t ) represents a risk item, used to measure the risk of a candidate frame violating regulations, exceeding authority, or going out of control in the current security state; Φ cost ( z The ) represents the cost item, which measures the execution cost of the candidate frame, including latency, resource consumption, tool call complexity, and code generation burden. λ 1. λ 2. λ 3. λ 4 and λ 5 represents the weight coefficients for semantic consistency, executability, context continuity, risk penalty, and cost penalty, respectively; constraint function Γ( z , s t ) represents a hard-gated function, when Γ( z , s t When Γ( )=1, it indicates that the candidate frame has passed the security and rule checks; when Γ( z , s t When )=0, it means that the candidate frame is directly rejected and will not proceed to subsequent execution; operator B (·) indicates the dual-track splitting operator, used to synchronously derive two types of outputs from the optimal unified execution semantic frame; δ * t This represents the optimal dialogue plan, used to generate explanations, feedback, follow-up questions, confirmations, or exception descriptions; K This represents the optimal execution plan, used to generate skills, action plans, and constraints, and to distribute them to the execution semantic agent runtime.

[0063] Based on the aforementioned states and algorithms, the underlying principle of the connection mechanism can be further clarified into three consecutive stages. The first stage is the semantic convergence stage, whose goal is to compress speech, vision, memory, and policy constraints from discrete inputs into a unified round-execution semantic frame, so that all factors that may affect the execution result are explicitly represented. The second stage is the semantic compilation stage, whose goal is for the execution semantic agent to read fields such as target, environment, safety, capability requirements, action intentions, and parameter slots in the unified semantic frame and translate them into skill-level task arrangements. The third stage is the dual-track splitting stage, whose goal is to derive dialogue expression branches and execution semantic branches in parallel on the same semantic frame, so that the system can output explanations, feedback, follow-up questions, confirmations, and exception descriptions, and can also form skills / action plans and constraints that can be inherited by subsequent action layers. The decisive feature of this three-stage mechanism is that the dialogue branch and the execution branch do not re-infer the task from the original input separately, but jointly derive the result from the same semantic frame, thus naturally maintaining consistency in object reference, environmental conditions, temporal logic, and safety boundaries. This mechanism is the fundamental reason why the "interaction-execution" collaboration has been engineered and established.

[0064] The compilation process within the semantic agent is not freely generated, but rather consists of a set of decomposable and auditable orchestration operations. First, action intent compression confines open-ended natural language tasks into skill-level task types identifiable by the skill graph. Second, parameter slot filling writes necessary execution information, such as object, position, order, intensity, termination conditions, and failure criteria, into a structured template, avoiding subsequent reliance on ambiguous text interpretation. Subsequently, capability matching and skill candidate retrieval combine the robot's current embodiment, skill library, tool definitions, and interface resources to screen execution paths with practical applicability. Based on this, cost budget gating provides constrained assessments of latency, resource consumption, risk level, and invocation complexity. Finally, execution mode selection chooses the minimum sufficient execution path among skill direct invocation mode, MCP (Model Context Protocol) orchestration mode, and restricted code mode. It is particularly important to clarify that the secondary-developed OpenClaw should be retained as the proxy core of the semantic agent, but the role of the large code model must be strictly limited to a conditional higher-order synthesizer, not the sole channel through which all executions must pass. Regular, high-frequency, low-risk, and well-structured tasks should be completed primarily using existing skills and deterministic workflows. Only when existing skills are insufficient, task combinations are complex, or temporary generation of invocation logic is required should restricted code mode be used. According to this definition, OpenClaw's role in this system is to receive unified semantic frames, complete high-level skill compilation and workflow organization, rather than replace the downstream action layer in generating continuous control.

[0065] To ensure the aforementioned connection mechanism forms a truly operational closed loop, rather than a one-off static orchestration, the telemetry data, environmental state increments, abnormal results, and verified execution results returned by the robot itself should all be rewritten back to the unified state center, triggering semantic frame updates, capability knowledge accumulation, and subsequent planning corrections. In this way, the relationship between the multimodal interactive agent and the execution semantic agent is no longer a one-way call, but a dynamic collaborative process continuously running around the unified round execution semantic frame: the interactive side is responsible for organizing user intent and environmental evidence into executable states, the execution side is responsible for compiling these states into skill plans, the downstream action layer is responsible for implementing the skill plans into continuous actions, and the actual execution results then correct the next round of state construction and execution compilation. Thus, the core innovation of this part can be accurately summarized as: establishing a unique and legitimate intermediate semantic layer between the multimodal interactive agent and the execution semantic agent through the unified round execution semantic frame, and on this basis, achieving constrained semantic compilation, dual-track diversion, and a closed loop of state feedback.

[0066] The interface solution for the three systems—semantic interaction agent, execution semantic agent, and VLA / action policy layer—is as follows: In humanoid robot scenarios characterized by continuous interaction, dynamic semantic updates, and the ability to insert safety constraints at any time, the single VLA (Vision-Language-Action) model is prone to revealing insufficient organizational capabilities not because it cannot generate actions, but because it typically compresses task semantics, environmental states, action constraints, and execution decisions into the same action generation process. This results in a lack of an independent, reconfigurable, and rewritable intermediate layer in the "understanding-decomposition-constraint-execution-verification" process. If the user adds restrictions, modifies the target, or changes the object midway through execution, the system, without an explicit execution organizational layer, often has to rely on policy reconditioning or external patches to complete the correction. This directly weakens the auditability, pluggability, and anomaly recovery capabilities of the execution semantics. Therefore, this embodiment of the invention constructs a three-system architecture: "semantic interaction agent - execution semantic agent - VLA / action policy layer". It is not a formal extension of the existing structure, but a necessary layering system established to address the gaps in the VLA's dynamic task reorganization interface: the upper layer is responsible for converging change semantics, the middle layer is responsible for compiling them into manageable execution organization results, and the lower layer is responsible for turning the organization results into continuous actions.

[0067] Based on the aforementioned necessity, the engineering role of the three systems should not be broadly described as "improving robot intelligence," but rather precisely defined as: adding an externalizable execution organization layer before VLA to address the core issue that dynamic semantic updates cannot be directly transformed into a stable execution process. Specifically, the semantic interaction agent is responsible for converging speech, vision, memory, and governance constraints into a unified execution semantic frame; the execution semantic agent is responsible for completing task semantic compression, parameter slot binding, capability parsing, skill graph routing, candidate plan adjudication, and action contract generation under the constraints of this semantic frame; the VLA / action policy layer no longer undertakes high-level semantic reorganization of the entire task chain, but instead focuses on generating continuous action policies based on action contracts. The key point directly addressed by this integration method is the interface for inserting new semantics and constraints midway through execution: when a user adds new information such as "put down the cup before opening the door," "don't touch the hot surface," or "grab the box on the left," the system does not require VLA to re-understand the complete long-range semantic reasoning. Instead, the semantic interaction agent first updates the unified execution semantic frame, then the execution semantic agent recalculates skills, sequence relationships, preconditions, termination conditions, and failure fallback conditions, and finally only assigns the new action contract to VLA for re-implementation. Thus, VLA is decoupled from the burden of "understanding, decomposing, and controlling," allowing it to focus on its core competency of action implementation, while the execution semantic agent assumes the responsibilities of explicit organization, approval, write-back, and replanning. This is precisely the direct solution provided by the three systems when integrating with VLA in engineering.

[0068] Figure 4 The demonstration shows that the three systems are not stacked sequentially, but rather operate in a hierarchical and collaborative structure around "semantic externalization - execution compilation - action implementation". The semantic interaction agent first completes multimodal fact convergence, compressing the task objective, environmental state, safety constraints, capability requirements, action intentions, and parameter slots into a unified execution semantic frame. The execution semantic agent then reads this semantic frame and generates an action contract that can be checked, approved, rolled back, and written back. The VLA / action policy layer then outputs continuous actions or action blocks based on the skill sequence, object constraints, spatial constraints, execution preconditions, termination conditions, and failure rollback conditions in the action contract. In this way, the high-level task decomposition, anomaly recovery, and process verification that were originally implicitly handled by VLA are moved up to an independent execution semantic layer that can be externalized, governed, and inserted. The acknowledgments, telemetry, and anomalies after execution can be written back to the unified execution semantic frame, driving the next round of interactive semantic correction and execution semantic recompilation. The result is not to weaken VLA, but to provide VLA with a stable, explicit and dynamically reconfigurable upstream interface, so that it no longer has to implicitly bear the semantic fluctuations and governance constraints in continuous interaction scenarios.

[0069] In the engineering integration of the three systems, the actual coupling point between the "interaction-execution" architecture and VLA is not at the natural language output end, nor at the unified execution semantic frame itself, but rather fixed at the interface pair of the action contract bridge and the contract interpreter. The engineering implications are: the semantic interaction agent forms a round-level factual state; the execution semantic agent generates an action contract that has been organized, constrained, and adjudicated; and what the VLA / action policy layer consumes must be a high-level execution object that has already been encapsulated with object constraints, spatial constraints, sequence relationships, execution preconditions, termination conditions, and failure fallback conditions, rather than the raw semantic input or an unorganized task description. By fixing the interface at the position of "action contract bridge → contract interpreter", VLA is clearly defined as the action implementation layer, only responsible for unfolding the established contract into continuous actions, and no longer undertaking the responsibilities of high-level semantic reorganization, dynamic constraint absorption and execution organization interpretation. Correspondingly, the core technical problem solved by the interaction-execution architecture is to stably externalize the continuously changing interaction semantics into a governable, rewritable and reorganizable execution organization structure, and then deliver it to VLA with a unified interface, thereby cutting off the semantic drift and responsibility overlap caused by "natural language directly driving action strategy" in engineering.

[0070] Based on the above interface positioning, the main forward chain of the three systems can be relied upon. Figure 4 The process is clearly defined as follows: the user's current input first enters the streaming round entry point, then flows sequentially through the speech semantic decoder, visual evidence injector, memory gating unit, and governance constraint injector. After fact convergence in the semantic fact assembler, a unified execution semantic frame is formed. This unified execution semantic frame then enters the execution semantic agent, first inputting into the semantic frame verifier, and then the verification results are distributed in parallel to the clarification decision gate and the constraint projector. Under the condition of "normal input," the clarification decision gate continues to send the input forward to the semantic frame compressor, while under the condition of "clarification feedback," it sends it back to the upstream interaction side along the dotted line in the diagram. The constraint projector directly injects security, resource, sequence, termination, and failure conditions into the semantic frame compressor, thereby enabling the semantic frame compressor to simultaneously receive two paths of information: "normal input continues forward" and "constraint projection results." The compressed semantics then sequentially enter the parameter slot binder, capability resolver, skill graph router, candidate plan arbiter, and pattern router. The pattern router then distributes the main execution path to the local skill executor or plugin workflow orchestrator. Finally, it is uniformly merged into the restricted code planner to complete conditional bridging and completion. Subsequently, it is output to the VLA / action strategy layer via the plan normalizer, action contract assembler, and action contract bridger.

[0071] It should be further clarified that the restricted code planner in the diagram is not an open code generation stage, but rather a conditional compensation layer in the back-end compilation chain of the semantic agent. Its input is simultaneously constrained by the action contract precursor, capability graph, parameter slot binding results, safety state, and execution mode selection results. This planner can only generate bridging code within predefined and verified code templates, function libraries, and robot control API sets, based on candidate plans that have passed risk cost adjudication. It must not directly construct execution logic that exceeds the action contract boundary, such as unregistered skills, unauthorized interfaces, or constraints on object constraints, spatial constraints, force range, sequence relationships, termination conditions, and failure fallback conditions. Before the generated results are officially released, they must also enter a lightweight sandbox verification and contract consistency check stage. Through simulated calls, field checks, or restricted playback, the action sequence, resource consumption, call order, and safety boundary compliance are pre-checked and compared item by item with the current action contract. Only when the verification results simultaneously meet the requirements of security hard gating, capability reachability, and contract consistency are the code results allowed to be incorporated into the downstream delivery chain via the plan normalizer and action contract assembler. Otherwise, code regeneration, pattern rollback, or execution failure rollback processes are triggered. Therefore, in this embodiment of the invention, the restricted code planner is not a freely programmable agent, but a controlled high-order synthesizer located before action contract assembly, jointly locked by the unified execution semantic frame, capability registry, policy constraints, and verification mechanism. Its role is to provide auditable, blockable, and rollback-capable code-level bridging capabilities when existing skill and plugin workflows are insufficient to cover complex composite tasks, without disrupting the main chain boundary of "unified execution semantic frame - execution semantic compilation - action contract - downstream action landing - execution receipt writeback". On the VLA side, the action contract further sequentially enters the contract interpreter, skill-action decomposer, constraint projector, policy scheduler, action block generator, and ontology adapter, ultimately forming continuous action output.

[0072] After the main forward chain is established, the key to the three systems forming a workable rather than one-time mapping engineering solution lies in the execution receipt closed loop at the bottom of the diagram and its upward-triggered recompilation chain. Specifically, the continuous action output does not stop at the VLA side as the endpoint, but continues to be input into the execution receipt interpreter. After the interpreter summarizes and generates structured execution results, these results are sent to the verified result extractor. Then, the semantic frame write-back engine writes the verified results, environment state increments, and exception information back to the upstream semantic organization layer. On this basis, the semantic frame recompilation trigger further injects changes such as "new semantics, new constraints, new objects, and new exceptions" back into the main chain of the three systems. This allows the system to re-trigger the execution semantic compilation based on the updated unified execution semantic frame, without the VLA directly absorbing the pressure of the entire high-level semantic reorganization. To ensure the real-time effectiveness of the execution receipt closed loop, the written execution receipts, environment state increments, and exception information are all timestamped and undergo timeliness checks before triggering semantic frame recompilation. If the written-back state is lagging behind the current round, the latest environment state obtained in real time is used first. Meanwhile, the write-back loop distinguishes between incremental updates and full recompilation based on the level of state change. Only when a change in task objective, a major anomaly, or a violation of security constraints is detected is the execution semantic agent triggered to fully recalculate skills, execution paths, and action contracts. Therefore, the three-system integration solution truly solves the core difficulties of a single VLA in continuous interaction scenarios: dynamically changing semantics, constraints, and anomalies are no longer handled by implicit reconditioning of action policies, but are first moved up to explicit semantic states, then the execution semantic agent recalculates skills, execution paths, and action contracts, and finally the VLA re-implements continuous actions based on the new contracts.

[0073] In engineering implementation, the key to the three systems lies not in adding another layer of general semantic interpretation to VLA, but in strictly confining the coupling between the interaction-execution architecture and VLA to a single, stable, and writable interface object. To this end, the execution semantic agent does not directly pass the raw semantic state or natural language description to VLA. Instead, it first compiles the unified execution semantic frame into an action contract, and then VLA performs continuous actions only around this contract. The results of execution do not intrude back into VLA; instead, they are written back to the unified execution semantic frame, triggering subsequent recompilation. This approach provides an explicit engineering entry point for dynamic semantic updates, constraint reorganization, and anomaly recovery without compromising VLA's action implementation responsibilities. Its core implementation can be illustrated by the following code example:

[0074] from dataclasses import dataclass from typing import Any, Dict, List @dataclass(frozen=True) class ActionContract: tesf_version: str skill_sequence: List[Dict[str, Any]] constraints: Dict[str, Any] execution_goal: str def bridge_execute_and_writeback( tesf: Dict[str, Any], contract: ActionContract, vla_interpreter, vla_executor, ) ->Dict[str, Any]: # The only legitimate coupling object: the action contract, not raw semantics or natural language vla_input = vla_interpreter({ "tesf_version": contract.tesf_version, "goal": contract.execution_goal, "skills": contract.skill_sequence, "constraints": contract.constraints, }) # VLA is only responsible for implementing the contract into a series of actions. receipt = vla_executor(vla_input) # The execution result does not directly modify the internal VLA, but instead writes back to TESF and triggers a recompilation. tesf["validated_results"] = receipt["validated_results"] tesf["environment_delta"] = receipt["environment_delta"] tesf["recompile_required"] = receipt.get("recompile_required", False) return tesf In the engineering implementation of the three systems, the boundaries must be fixed at two insurmountable interface points: one is the unified execution semantic frame, and the other is the action contract bridge → contract interpreter. The former constitutes the only legal interface between the semantic interaction agent and the execution semantic agent. All content related to speech understanding, visual evidence injection, memory integration, governance constraint writing, and round fact convergence stops at the unified execution semantic frame. The execution semantic agent must not retrospectively rewrite the original interaction facts. It can only start from the semantic frame verifier and, under the field constraints of the given target state, environment state, security state, capability requirements, action intention, and parameter slots, complete semantic compression, parameter binding, capability parsing, skill routing, plan adjudication, mode selection, and action contract assembly. The latter constitutes the sole engineering interface between the semantic agent and the VLA / action policy layer. The semantic agent terminates at the action contract bridge, and its responsibility is limited to outputting a high-level execution object consisting of skill sequences, parameter mappings, object constraints, spatial constraints, execution preconditions, termination conditions, and failure fallback conditions. It must not generate action blocks, trajectories, joint targets, or torque controls beyond its boundaries. Correspondingly, the VLA only receives action contracts from the contract interpreter and is responsible for expanding the contract into a continuous action policy and ontology adaptation result. It must not assume the responsibility of high-level semantic reorganization, interaction understanding, or execution organization and arrangement. Thus, the boundaries of the three systems can be strictly defined as follows: the semantic interaction agent terminates at fact convergence, the semantic agent terminates at the action contract, and the VLA terminates at continuous action output. Any new semantics, new constraints, new objects, or new anomalies must not be directly inserted into the VLA, but must return to the unified semantic frame through the acknowledgment write-back and semantic frame recompilation chain before re-entering the semantic compilation process. With this configuration, the inputs, outputs, permissions, and responsibilities of each layer in the system remain unidirectionally clear, thus enabling the engineering interface to obtain stable boundaries that are approvable, replaceable, replayable, and auditable.

[0075] The specific project implementation plan is as follows: At the implementation level, the execution semantic agent is implemented as an independent execution semantic compilation service located between the unified execution semantic frame and the action contract. Its input is strictly limited to TESF, and its output is strictly limited to the action contract. The intermediate main chain is responsible for compiling high-level task semantics into skills, action plans, parameter constraints, execution preconditions, termination conditions, and failure fallback conditions. It does not receive raw natural language, generate continuous actions, or directly undertake the internal policy unfolding of VLA. Thus, this module is fixed as an "execution semantic compiler" in engineering, with its responsibility limited to the generation of action contracts. The subsequent generation of continuous actions and ontology control are explicitly left to the downstream action policy layer.

[0076] From a system implementation perspective, the execution semantic agent should adopt a four-layer structure of "gateway service - execution compilation service - model service - capability registry," rather than stacking all logic within a single script. The gateway service is responsible for session entry, state persistence, and upstream / downstream scheduling; the execution compilation service, which is the execution semantic agent ontology in this invention, is responsible for TESF access, main chain compilation, and action contract output; the model service currently calls GPT-5.4 through the OpenAI Responses API (unified response interface) to complete task semantic compression and candidate plan generation, and will subsequently be replaced with a self-developed model (VeriLoop) using the same provider (model provider) interface; the capability registry is responsible for registering skills, tool modes, plugin capabilities, node capabilities, and restricted code proxy capabilities for unified parsing and routing by the main chain. OpenAI has officially defined the Responses interface as a unified high-capability response interface, and OpenClaw also supports accessing custom model providers through provider configuration. Therefore, "using GPT-5.4 first, and then replacing it with the VeriLoop model" is continuous in engineering and does not require rewriting the main chain.

[0077] In the main chain implementation, the execution semantic agent should no longer be organized according to the general agent loop, but should be strictly organized according to the execution semantic compilation chain as follows: Streaming round entry → Semantic frame validator → Clarification decision gate → Constraint projector → Task semantic compressor → Parameter slot binder → Capability parser → Skill graph router → Candidate plan generator → Risk cost adjudicator → Pattern router → Local skill executor / plugin workflow orchestrator / restricted code planner → Plan normalizer → Action contract assembler → Downstream delivery adapter. The front end is responsible for field integrity verification, clarification blocking, and constraint pre-injection; the middle end is responsible for compressing high-level task semantics into skill-level tasks and completing parameter slot binding, capability parsing, and graph routing; the back end is responsible for multi-candidate plan generation, risk cost adjudication, execution mode selection, and unified normalization assembly, ultimately outputting only the action contract, not the natural language description. With this definition, the main goal of the entire chain is "compilation" rather than "answering," thus maintaining a single and clear engineering focus.

[0078] From a secondary development perspective, what OpenClaw truly needs to inherit is not its user-side assistant shell, but rather its session runtime, skills loading mechanism, capability registration structure, and ACP bridging capabilities. The Pi integration documentation indicates that OpenClaw uses an embedded AgentSession access method, thus enabling direct control over session lifecycle, tool injection, system prompts, persistence, and model switching; skills are loaded in the form of a SKILL.md directory and support multi-level priority loading; the ACP bridging is suitable as a host for restricted code planning paths, rather than the default execution main chain. Therefore, in practical implementation, the executing semantic agent should absorb OpenClaw's runtime skeleton and capability loading mechanism, but should actively strip away its multi-channel messaging, terminal assistant, and general dialogue surface layer functions unrelated to this invention to avoid further confusion regarding engineering boundaries.

[0079] In terms of the arrangement of the client, server, and output links, the client is only responsible for task initiation, state observation, and result reception, and does not participate in the execution semantic compilation; the gateway is only responsible for session management, TESF injection, and call scheduling; the execution semantic agent ontology only outputs the action contract; and the downstream VLA side consumes the contract and unfolds continuous actions. Its most important engineering result is not that "the robot is immediately more intelligent," but that the system obtains a high-level execution organization hub that can be independently tested, approved, and replaced for the first time. DeepMind publicly adopts a "high-level orchestration-action execution" layering, and this solution further solidifies this high-level orchestration into a dedicated execution semantic compilation service with TESF as input and action contracts as output, thereby externalizing the high-level organization process that was originally hidden within the action strategy into a governable, replaceable, and auditable engineering main chain.

[0080] After the semantic agent has completed independent development and possesses stable compilation capabilities with TESF as input and action contracts as output, the next problem to solve is no longer how to construct its internal main chain, but how to achieve low-latency, controllable, and clearly defined engineering integration with the existing multimodal interactive agent "Wanzi" at the backend orchestration layer. Therefore, the implementation focus of this part shifts to the overall orchestration of the interaction-execution agent: that is, the front-end "Wanzi" is responsible for multimodal fact convergence and interaction response, the back-end orchestration layer completes unified semantic selection and control branch activation, and then the execution semantic agent receives the selected TESF and continues to complete the compilation of high-level task semantics into action contracts.

[0081] Regarding the access method, the key to low latency and robustness lies not in adding extra caching or intermediate message layers, but in strictly compressing cross-module transmitted objects. The current script implements this by having the backend only transcribe the selected single semantic frame into a core TESF (Unified Execution Semantic Frame) via the TESF Adapter, and then have the Execution Semantic Branch Adapter call Execution SemanticAgent.compile() to enter the execution semantic main chain, instead of forwarding the entire batch of raw multimodal data to the execution semantic agent for re-understanding. Thus, on the one hand, semantic merging, risk assessment, and candidate selection already completed on the interaction side are no longer repeatedly calculated downstream; on the other hand, if the Dialogue Control Bifurcator determines that the current round only requires clarification or does not meet the execution threshold, the control branch skips directly to avoid invalid compilation calls, thereby concentrating latency on rounds that truly need to enter the execution organization. Meanwhile, the Execution Semantic Branch Adapter performs layered returns for ClarificationRequired, Boundary Violation, and general exceptions. The execution semantic main chain pre-writes the risk level, target object, space constraints, and resource limits into the TESF (Unified Execution Semantic Frame), so that the backend orchestration layer will not directly lose control when facing high-risk, missing parameter, or boundary conflict scenarios, but will instead send back to the upper layer in an interpretable state.

[0082] The resulting engineering outcome is a clearly defined and stable interactive-executive agent: the client-side "Wanzi" is responsible for receiving and organizing speech, vision, memory, and governance constraints; the backend orchestration layer is responsible for candidate TESF (Unified Execution Semantic Frame) selection and branch activation; and the execution semantic agent only receives TESF and outputs action contracts, without swallowing back interaction logic or overstepping boundaries into VLA (Vision-Language-Action Model) or continuous action control. The current model service boundary is fixed in the code as the `model="gpt-5.4"` configuration of the OpenAI Responses Client, so at this stage, task semantic compression and candidate plan generation can be completed using GPT-5.4. Once VeriLoop (developed by the applicant) is completed, only the interface of this model provider needs to be replaced; the interactive-executive main chain itself does not need to be overturned. In other words, the real value of this implementation scheme is not simply connecting the "pill" and the execution semantic agent, but forming a sustainable and scalable low-latency semantic compilation chain in the backend orchestration layer: the frontend is responsible for fact convergence, the backend is responsible for candidate selection, and the execution semantic agent is responsible for action contract output. The entire system then has clear input boundaries, output objects, and subsequent model replacement paths.

[0083] The engineering value of the three systems lies in the single action contract interface established between the multimodal interaction-executing semantic agent and VLA (Vision-Language-Action). Current publicly available cutting-edge approaches to embodied systems have shown that high-level orchestration and action execution must be layered: Google DeepMind places Gemini Robotics-ER (high-level embodied reasoning layer) before the action execution layer, while Figure explicitly separates high-level task organization from low-level action implementation in the manner of S2 (semantic reasoning) - S1 (action target generation) - S0 (low-level whole-body control). Based on this logic, the three systems should not allow VLA to continue to absorb the pressure of original semantic reorganization. Instead, the semantic interaction agent should first complete fact convergence, and the executing semantic agent should compile the TESF (Unified Execution Semantic Frame) into an inspectable, approveable, and rewriteable action contract. Then, VLA should complete the continuous action implementation only around this contract.

[0084] Therefore, the specific development approach should be fixed as a low-latency, strong-boundary main chain: the front-end "Maruko" is only responsible for multimodal fact convergence, the back-end first selects a single TESF by the Interaction Execution Bridge Engine, and then the Execution Semantic Agent Core compiles it into a Coupled Action Contract. Subsequently, it enters the VLA side only through the Action Contract BoundaryGuard → Default Contract Interpreter → Demo VLA Executor. The Execution Receipt after execution is then returned to the TESF by the TESF Writeback Engine and triggers the necessary recompilation. Its low latency doesn't stem from additional cache stacking, but rather from transmitting only a single semantic frame and a single action contract across layers, avoiding the repeated transmission of original speech, visual, and natural language descriptions. Its stability doesn't come from VLA providing a fallback explanation for everything, but from the action contract boundary guard pre-checking contract integrity, followed by the write-back chain reintegrating anomalies and environmental increments into TESF. This moves the processing of "new semantics, new constraints, and new states" forward to the execution semantic layer, rather than forcibly pushing them into the action policy. This implementation essentially transforms VLA from a "high-level semantic reorganizer" into an "action contract executor," making the three systems a truly implementable, replaceable, and auditable engineering solution.

[0085] The development equipment, toolchain, and computing power costs are configured as follows: To clearly present the software implementation path of this implementation plan, the relevant code scripts are divided into three core modules according to functional boundaries: the development of the semantic agent, the construction of the interaction-execution architecture and agent development, and the development of the three-system interface between the semantic interaction agent, the execution semantic agent, and the VLA / action strategy layer. At the same time, the directories of the shared contract layer, the model provider layer, the runtime service layer, and the test layer are also provided to ensure that the organizational structure of each script in terms of input, output, calling relationships, and engineering responsibilities is complete, the boundaries are clear, and it is feasible.

[0086] In terms of development equipment, the entire project is divided into three categories: "local development host - lightweight gateway node - robot-side execution terminal". Firstly, the local development host handles script development, unit testing, log analysis, and integration testing. Ideally, this should be an x86_64 architecture laptop or workstation with at least 32 GB of main memory and high-speed NVMe storage to simultaneously run the execution semantic main chain, bridging engine, contract examples, and test suites. Secondly, if a long-term resident control plane is required, the OpenClaw Gateway can be deployed on a lightweight node. Official documentation and common deployment instructions indicate that the gateway itself is lightweight, and model inference can be completed via cloud APIs; therefore, a Raspberry Pi or a small VPS (Virtual Private Server) is sufficient to support the control plane. Thirdly, the robot-side terminal is only responsible for sensor, actuator, and feedback acquisition, and does not handle high-level execution semantic compilation, thus concentrating the hot path pressure on the backend orchestration layer.

[0087] Once the aforementioned hardware foundation is established, the software toolchain should be organized around a four-layer structure: "Gateway Service - Semantic Execution Compilation Service - Model Provider - Capability Registry," rather than simply a loose collection of independent scripts. In practical implementation, the service entry point should utilize FastAPI (a lightweight interface framework) and Uvicorn (an asynchronous execution server) to handle interface publishing and scheduling logic. The structured data layer should employ Pydantic (a data validation framework) and PyYAML (a configuration parsing library) to manage TESF (Unified Execution Semantic Frames), action contracts, and the capability registry. The network call layer should use HTTPX (an asynchronous HTTP client) to complete model requests and external service access. The testing layer should use Pytest (a testing framework) for unit testing and regression verification. If the secondary development approach of OpenClaw is adopted, its Agent Session, skills, provider, and capability loading mechanisms should be primarily inherited, rather than including its general assistant surface layer in this solution.

[0088] Once the hardware and software boundaries are clearly defined, the configuration of computing power and cost can be arranged according to a two-stage approach: "API-driven during development and model mounting during deployment." In the stage before the self-developed large model VeriLoop is mounted, the most economical and reliable approach is to have the local development host handle code execution, log analysis, and link verification, while high-intensity inference is handled by calling GPT-5.4 via the OpenAI Responses API (Unified Response Interface). At this time, the local device does not need to bear the additional GPU memory pressure of the large model; the cost is mainly concentrated on interface calls and service integration. When subsequently converted to private or semi-private deployment, dedicated inference nodes can be configured according to the actual GPU memory requirements of a 30B-level model. Taking the Qwen 30B-level model as a reference, a single A100 80GB or H100 80GB card is recommended as the standard computing power card during the development and verification stages. If entering a stage of ultra-long context or high-concurrency deployment, a multi-card parallel solution should be adopted. The official reference total GPU memory for a 1 million-token (word-level) context is approximately 240GB. Therefore, the entire computing power configuration path is not a one-time heavy investment. Instead, the main chain of the project is first finalized by "local development + cloud inference", and then the subsequent upgrades are completed by "dedicated inference nodes + self-developed model replacement". This ensures R&D efficiency while controlling early computing power costs and leaves a clear and stable engineering entry point for model replacement.

[0089] The embodiments of the present invention have the following beneficial effects: 1. Compared to existing solutions that simply push interactive semantics directly into the action generation chain or merely add action descriptions at the output, the beneficial effect of the execution semantic agent proposed in this invention lies in establishing an independently developable, testable, and replaceable execution semantic compilation layer between high-level task semantics and continuous action control. This layer takes a unified execution semantic frame as input and action contracts as output, externalizing object constraints, spatial conditions, sequential relationships, termination conditions, and failure rollback conditions—originally floating in natural language—into structured execution organization results. This enables the robot control process to, for the first time, possess intermediate execution objects that are approvable, traceable, rewritable, and recompilable. Therefore, the system no longer implicitly assumes the responsibilities of task decomposition, constraint absorption, and anomaly recovery through a single action strategy. Instead, it completes skills orchestration, pattern selection, and contract assembly through an explicit execution organization layer, significantly reducing the drift risk of high-level semantics during execution and ensuring stable boundaries for subsequent model replacement, capability expansion, and safety governance.

[0090] 2. The beneficial effect of the unified execution semantic frame-driven interaction-execution collaborative architecture lies in the stable integration of multimodal interactive agents and execution semantic agents at the backend orchestration layer, forming a low-latency, low-repetitive computation, and clearly defined interaction-execution main chain. This scheme does not send raw speech, vision, and memory data downstream for repeated understanding. Instead, the "Wanzi" (a component) first completes fact convergence, candidate semantic frame generation, and unified semantic selection. Only the selected single TESF (Translation-Semantic Frame) is sent to the execution semantic compilation chain, and the control branch is directly blocked when the execution threshold is not met or clarification is required. The direct result is that the boundaries of responsibility between the interaction side, the execution side, and the backend orchestration layer remain clear in one direction, meeting the response speed requirements of real-time interaction scenarios while ensuring interpretable feedback in high-risk, parameter-missing, or conflicting scenarios.

[0091] 3. The beneficial effect of the three-system docking scheme of semantic interaction agent—execution semantic agent—VLA / action policy layer lies in using the action contract as the only legal coupling object. It externalizes the continuously changing interaction semantics into a manageable execution organization structure, and then entrusts the VLA to realize continuous action policies. This provides a practical engineering solution to the current problem of VLA lacking an independent execution organization layer for dynamic semantic updates, temporary constraint insertion, and anomaly recovery. By strictly layering high-level interaction semantic convergence, mid-level execution semantic compilation, and low-level action policy generation, new semantics, new objects, new constraints, and new anomalies in the system are no longer directly pushed into the VLA for implicit reconditioning. Instead, they are first written back to the unified execution semantic frame, then the execution semantic agent is triggered to recompile skills, sequence relationships, and action contracts. Finally, the VLA regenerates actions only around the new contract.

[0092] The key protected object of this invention is not a single script, a single model, or a single platform access method, but rather the three-layer core technical solution built around the execution semantic agent: the definition and development mechanism of the execution semantic agent, the unified execution semantic frame-driven "interaction-execution" collaborative architecture, and the three-system layered docking scheme of semantic interaction agent, execution semantic agent, and VLA / action policy layer. OpenClaw is merely one implementable instance of the execution semantic agent in this invention, not the only implementation form. Any agent that satisfies the definitions of input boundaries, output boundaries, main chain functions, and organizational responsibilities of the execution semantic agent in this invention, and can stably compile the unified execution semantic frame into an action contract, and its secondary development forms, should fall within the scope of protection. Correspondingly, the focus of protection in this invention is not limited to a specific model name, tool platform, or code implementation, but rather to the underlying execution semantic organization mechanism, interface boundaries, main chain arrangement order, branch coordination method, contract coupling method, and write-back closed-loop logic.

[0093] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A task execution control method based on an execution semantic agent, characterized by, The method comprises the following steps: S1, receiving a unified execution semantic frame by the execution semantic agent, the unified execution semantic frame being structured data representing a current round task execution context and being generated based on multi-modal input information; S2, performing compilation processing through an execution semantic compilation chain based on the unified execution semantic frame to generate an action contract as an independent intermediate interface between high-level interaction semantics and downstream continuous action control, wherein the execution semantic compilation chain comprises multiple links of semantic frame verification, clarification determination, constraint projection, task semantic compression, parameter slot binding, skill graph routing, candidate plan arbitration, and action contract assembly, and the action contract contains a skill sequence and an execution constraint condition for driving a downstream action policy layer; S3, sending the action contract to the downstream action policy layer to generate a continuous action control signal according to the action contract by the downstream action policy layer.

2. The method of claim 1, wherein, In step S2, the compilation processing through the execution semantic compilation chain based on the unified execution semantic frame comprises verifying and clarifying the unified execution semantic frame, projecting a safety boundary and a resource constraint to a subsequent compilation link, and performing semantic compression on a task target and binding it with a parameter slot.

3. The method of claim 1, wherein, In step S2, the execution of the execution semantic compilation chain is based on secondary development of a general agent runtime base, and the secondary development comprises reconstructing a target function of the general agent runtime base from a general tool call to action contract generation oriented to robot control.

4. A robot control method characterized by, The method comprises processing voice, visual, and memory input through a multi-modal interaction agent to generate a unified execution semantic frame, performing the task execution control method based on the execution semantic agent according to any one of claims 1-3 to generate an action contract based on the unified execution semantic frame, and performing the action contract through an action policy layer to generate a continuous action for controlling a robot.

5. A method of storing action contract data, characterized by, The method comprises performing the task execution control method based on the execution semantic agent according to any one of claims 1-3 to generate an action contract, and storing the action contract in a non-volatile storage medium.

6. A method of transmitting action contract data, characterized by, The method comprises performing the task execution control method based on the execution semantic agent according to any one of claims 1-3 to generate an action contract, and transmitting the action contract to a receiving end through a network.

7. An executing semantic agent apparatus, characterized by The device comprises a memory and a processor, and the memory stores a computer program which, when executed by the processor, implements the task execution control method based on the execution semantic agent according to any one of claims 1-3.

8. A robot control system, characterized in that The system comprises a multi-modal interaction agent configured to generate a unified execution semantic frame, an execution semantic agent device according to claim 7 configured to receive the unified execution semantic frame and generate an action contract, and an action policy layer configured to receive and execute the action contract to output a continuous control instruction for a robot.

9. An electronic device, comprising: Includes the semantic agent execution device as described in claim 7.

10. A computer readable storage medium having stored thereon computer programs / instructions and / or action contract data, characterized in that, When the computer program / instruction is executed by the processor, it implements the task execution control method based on execution semantic intelligent agent as described in any one of claims 1-3 to generate the action contract data.

Images